New to the community? Start here

Optimal performance with the WordPress Multisite cron

Our dev team made a pretty clean WPMU cron setup, but we’re not convinced it’s optimal and wondered if anyone think they can do better! :slight_smile:

There is about 100 sites.

#1 The cron job (every 30 minutes):

wget -q -O - https://www.magikweb.net/wp-cron-multisite.php >/dev/null 2>&1

#2 That PHP file loops in all active sites to trigger their individual WordPress crons:

<?php
if ( ! defined('ABSPATH') ) {
	require_once( './wp-load.php' );
}
global $wpdb;
$sites = $wpdb->get_results( "SELECT domain, path FROM " . $wpdb->blogs . " WHERE archived=0 AND deleted=0;" );
foreach( $sites as $site ) {
    $url = "https://" . $site->domain . ( ( ! empty( $site->path ) ) ? $site->path : '/' ) . 'wp-cron.php';
    wp_remote_get( $url, array( 'blocking' => false ) );
}

#3 There are currently 3277 cron jobs according to Advanced DB Cleaner (a lot from WooCommerce and WPML):

[attachments are only viewable by logged-in members]

#4 We are having issues since MySQL 5.7 where “MySQL has gone away” during off-work hours and we can’t figure out what queries (or crons) are causing it. It’s why we’re looking into this.

Is there a considerably better (performance) way to do this?

Mon, 24 Aug 2020 14:14:41
  • Aaron
    • Ex Staff

    Looping and doing them all at once probably is a big hit on your server. Most multisites we only do a cron for the main site and let traditional cron handle the subsites (what we do in our hosting) as those are usually less important schedules.

    The only solution I know of out there is https://humanmade.com/2017/05/29/cavalcade-scaling-wordpress-jobs/ there are som e tutorials out there for running it.

    On this site we have a crazy setup using AWS Batch and an autoscaling ECS cluster.

  • Vince
    • Site Builder, Child of Zeus

    Thank you Aaron,
    That helps!

    What jobs are important in regards to the main site? (any example?) We might try reverting back to normal crons (except for the root site).

    For now we added a usleep( 500000 ); in the loop to give it some air to breath. We’ll also make a script that spreads all the cron jobs evenly during the day, hopefully that’ll buy us some time to grow a little more.

    We’re not ready to go on a Cloud solution like AWS just yet since we only have about 100 sites on there (mostly low-traffic ones). We’re using a 6 vCPU and 10 GB RAM VPS at the moment.

    Let us know if you have any other hints, otherwise we’ll look into AWS or another managed hosting solution for more scalability.

  • Vince
    • Site Builder, Child of Zeus

    Here is our newer optimal way to run Multisite crons using WP CLI to avoid extra HTTP requests.

    We created this bash file (wp-cron-multisite.sh).

    #!/bin/bash

ulimit -d unlimited
ulimit -n 1024
ulimit -t unlimited
ulimit -u 120

PATH_TO_WORDPRESS="/home/path/public_html"
DEBUG=false
DEBUG_LOG=/home/path/public_html/wp-content/debug.log

if [ "$DEBUG" = true ]; then
        echo $(date -u) "Running WP Cron for all sites." >> $DEBUG_LOG
fi

for URL in = $(wp site list --field=url --path="$PATH_TO_WORDPRESS" --deleted=0 --archived=0)
do
        if [[ $URL == "http"* ]]; then
                if [ "$DEBUG" = true ]; then
                        echo $(date -u) "Running WP Cron for $URL:" >> $DEBUG_LOG
                        wp cron event run --due-now --url="$URL" --path="$PATH_TO_WORDPRESS" >> $DEBUG_LOG
                else
                        wp cron event run --quiet --due-now --url="$URL" --path="$PATH_TO_WORDPRESS"
                fi
        fi
done

    The cron calls it with this command:
    /bin/bash /home/path/public_html/wp-cron-multisite.sh

    This works on a SiteGround VPS (had to increase the process limits through support).
    Again, if anyone sees a way to optimize that further, let us know! (maybe a way to kill processes so they don’t cumulate)

    Kind regards,
    Vince