WMultisite is oveloaded

hey guys we have a multisite setup on a high end server (we host the servers on our datacenter) curently with 20 sites and we have seen that the wp install show as a exhausted memory – the usual cant allocate xx bytes (even the memory on php.ini is on 512M)

any advice on how we can optimize it?

WordPress memory usage exceeded 128 MB

WordPress peak memory usage: 450.94 MB

Number of database queries: 3153