New to the community? Start here

[Broken Link Checker] Broken Link Checker is slow

I’ve been using Broken Link Checker off and on for awhile now, but I didn’t know how much of a resource drain it is until I noticed it showing up in my php_slow.log file.

I found more info on the plugin page here (https://wordpress.org/support/topic/will-make-your-blog-and-server-unresponsive/) where Biplav suggests adjusting some settings.

But, perhaps instead of providing default settings that are set so high they slow our servers down, can you change the default settings be more reasonable and much lower, please?

In the meantime I’ve changed the settings and will have to watch it and see if it continues to show up in php_slow.log.

It sounds like you recently bought this plugin and are in the process of improving it. How’s that coming along? Because I’m still seeing it in my php_slow.log on a server that gets zero traffic right now.

Please help!
Thanks.

Logs are here:

[25-Jun-2020 23:25:20]  [pool staging] pid 25918
script_filename = /var/web/staging/public_html/wp-admin/admin-ajax.php
[0x00007fd55a818180] usleep() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:61
[0x00007fd55a817f20] waitForToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:38
[0x00007fd55a817e20] takeToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/modules/checkers/http.php:69
[0x00007fd55a817cf0] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/links.php:281
[0x00007fd55a817910] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:2876
[0x00007fd55a816d30] work() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:3194
[0x00007fd55a816cb0] ajax_work() /var/web/staging/public_html/wp-includes/class-wp-hook.php:287
[0x00007fd55a816920] apply_filters() /var/web/staging/public_html/wp-includes/class-wp-hook.php:311
[0x00007fd55a816870] do_action() /var/web/staging/public_html/wp-includes/plugin.php:478
[0x00007fd55a816590] do_action() /var/web/staging/public_html/wp-admin/admin-ajax.php:175

[01-Jul-2020 23:27:28]  [pool staging] pid 13479
script_filename = /var/web/staging/public_html/wp-admin/admin-ajax.php
[0x00007f1436a19180] usleep() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:61
[0x00007f1436a18f20] waitForToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:38
[0x00007f1436a18e20] takeToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/modules/checkers/http.php:69
[0x00007f1436a18cf0] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/links.php:281
[0x00007f1436a18910] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:2876
[0x00007f1436a17d30] work() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:3194
[0x00007f1436a17cb0] ajax_work() /var/web/staging/public_html/wp-includes/class-wp-hook.php:287
[0x00007f1436a17920] apply_filters() /var/web/staging/public_html/wp-includes/class-wp-hook.php:311
[0x00007f1436a17870] do_action() /var/web/staging/public_html/wp-includes/plugin.php:478
[0x00007f1436a17590] do_action() /var/web/staging/public_html/wp-admin/admin-ajax.php:175

[04-Jul-2020 23:28:54]  [pool staging] pid 32627
script_filename = /var/web/staging/public_html/wp-admin/admin-ajax.php
[0x00007f7ef2219180] usleep() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:61
[0x00007f7ef2218f20] waitForToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:38
[0x00007f7ef2218e20] takeToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/modules/checkers/http.php:69
[0x00007f7ef2218cf0] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/links.php:281
[0x00007f7ef2218910] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:2876
[0x00007f7ef2217d30] work() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:3194
[0x00007f7ef2217cb0] ajax_work() /var/web/staging/public_html/wp-includes/class-wp-hook.php:287
[0x00007f7ef2217920] apply_filters() /var/web/staging/public_html/wp-includes/class-wp-hook.php:311
[0x00007f7ef2217870] do_action() /var/web/staging/public_html/wp-includes/plugin.php:478
[0x00007f7ef2217590] do_action() /var/web/staging/public_html/wp-admin/admin-ajax.php:175

[06-Jul-2020 21:53:14]  [pool staging] pid 8334
script_filename = /var/web/staging/public_html/wp-admin/admin-ajax.php
[0x00007f92fdc19180] usleep() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:61
[0x00007f92fdc18f20] waitForToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:38
[0x00007f92fdc18e20] takeToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/modules/checkers/http.php:69
[0x00007f92fdc18cf0] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/links.php:281
[0x00007f92fdc18910] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:2876
[0x00007f92fdc17d30] work() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:3194
[0x00007f92fdc17cb0] ajax_work() /var/web/staging/public_html/wp-includes/class-wp-hook.php:287
[0x00007f92fdc17920] apply_filters() /var/web/staging/public_html/wp-includes/class-wp-hook.php:311
[0x00007f92fdc17870] do_action() /var/web/staging/public_html/wp-includes/plugin.php:478
[0x00007f92fdc17590] do_action() /var/web/staging/public_html/wp-admin/admin-ajax.php:175

[07-Jul-2020 23:30:03]  [pool staging] pid 31070
script_filename = /var/web/staging/public_html/wp-admin/admin-ajax.php
[0x00007f9712a19180] usleep() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:61
[0x00007f9712a18f20] waitForToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/token-bucket.php:38
[0x00007f9712a18e20] takeToken() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/modules/checkers/http.php:69
[0x00007f9712a18cf0] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/includes/links.php:281
[0x00007f9712a18910] check() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:2876
[0x00007f9712a17d30] work() /var/web/staging/public_html/wp-content/plugins/broken-link-checker/core/core.php:3194
[0x00007f9712a17cb0] ajax_work() /var/web/staging/public_html/wp-includes/class-wp-hook.php:287
[0x00007f9712a17920] apply_filters() /var/web/staging/public_html/wp-includes/class-wp-hook.php:311
[0x00007f9712a17870] do_action() /var/web/staging/public_html/wp-includes/plugin.php:478
[0x00007f9712a17590] do_action() /var/web/staging/public_html/wp-admin/admin-ajax.php:175
Wed, 15 Jul 2020 1:27:51
  • Adam
    • Support Gorilla

    Hi J

    I hope you’re well today and thank you for your question!

    We’ve acquired Broken Link Checker not so long ago, indeed, after the plugin was not really under active development for quite some time so there was a lot to be done and there still is – we’ve got quite a long list of further improvements and fixes. They’ll be coming over to the plugin gradually over time with upcoming updates.

    As for PHP slow_log and the impact on server resources. The slow log entry (regardless of whether it’s related to this plugin or any other script) doesn’t necessarily mean it’s a cause of resource usage – it may or may not be. It might as well be… a result of resource issues which, in turn, may be related to “just too limited resources” or e.g. to some conflict.

    I’m not saying that it is in this case but in general, an entry in a slow log isn’t always and indication that a given, logged there, script is the culprit – it quite often might actually be result of other issue.

    As for providing “lower default settings” – yes, I think that could actually be good mostly if it comes to “server load limit” and “target resource usage” and probably also “max execution time” while “timeout” isn’t really that high by default. I’ve already asked our developers what we could do about that.

    However, there’s still no way to find a “universal setting” that will help everywhere as there’s way too many other factors that affect that – such as e.g. actual server resources and the way server is configured and the site configuration and overall complexity. That’s why these settings are available for you to be changed rather than “hard-coded”.

    But yes, we are still working on a lot of various improvements and fixes, including performance related so for now it would be best to

    – experiment a bit with those settings to find out those that on your specific setup are not causing too excessive resource usage but still allow plugin to function
    – keep the plugin up to date to be sure that all the fixes and improvements that will be released, will be applied “on the go”.

    Kind regards
    Adam

  • J
    • Site Builder, Child of Zeus

    Hi Adam Czajczyk

    As for PHP slow_log and the impact on server resources. The slow log entry (regardless of whether it’s related to this plugin or any other script) doesn’t necessarily mean it’s a cause of resource usage – it may or may not be. It might as well be… a result of resource issues which, in turn, may be related to “just too limited resources” or e.g. to some conflict.

    I’m not saying that it is in this case but in general, an entry in a slow log isn’t always and indication that a given, logged there, script is the culprit – it quite often might actually be result of other issue.

    Typically php slow log entries are due to a script taking more than 5 seconds to execute unless another timeout has been set.

    My point is that on a server with zero traffic and not much else going on in terms of server resources (no highly intensive WP plugins, API calls, or DB interaction, for example), it is unusual for a PHP script to take more than 5 seconds to execute, which is what appears to be going on here.

    My guess as to what is actually happening, based on the timestamps of when these are logged (a few were at around the same time of day), is that this might be happening during a backup or other resource-intensive activity occurring on the server. If BLC were to run during this time, it could be battling for CPU or memory, resulting in longer execution times. My personal opinion is this is what is happening. The solution, in my opinion, is to restrict it to running once an hour and reduce all the Advanced settings to much lower levels (1.00 for Load Limit, 10% target resource usage, unchecking “Run when dashboard is open”, etc).

    As for providing “lower default settings” – yes, I think that could actually be good mostly if it comes to “server load limit” and “target resource usage” and probably also “max execution time” while “timeout” isn’t really that high by default. I’ve already asked our developers what we could do about that.

    Yes, those are the settings I’m referring to which could be lowered, at least in my opinion. A lower default setting for the plugin. At least, that’s what I would do.

    However, there’s still no way to find a “universal setting” that will help everywhere as there’s way too many other factors that affect that – such as e.g. actual server resources and the way server is configured and the site configuration and overall complexity. That’s why these settings are available for you to be changed rather than “hard-coded”.

    I wasn’t really meaning a “universal setting”, I was only suggesting a lower setting by default (lowering the settings you mentioned, along with turning off the “run while dashboard is open” setting by default). Let users raise these settings if they want to, rather than having to manually turn them down after it causes problems on their server (as reported by the many 1 and 2 star ratings on the WP plugin page – https://wordpress.org/support/plugin/broken-link-checker/reviews/). Consider it good PR and marketing if nothing else, because you’ll get less negative reviews about it.

    But yes, we are still working on a lot of various improvements and fixes, including performance related so for now it would be best to

    – experiment a bit with those settings to find out those that on your specific setup are not causing too excessive resource usage but still allow plugin to function
    – keep the plugin up to date to be sure that all the fixes and improvements that will be released, will be applied “on the go”.

    Glad to hear you guys are still trying to improve it.

    But I don’t believe that it should be on the user to have to tweak the settings in order to ensure it doesn’t cause potential issues for the server. Having the lower default settings would only help you guys in the long run, too, as I mentioned regarding the reviews and feedback (and entries in php_slow.log!).

    Hopefully this is helpful for you guys, and I share this in the interest of improving this useful plugin.

    Thanks again.

  • Adam
    • Support Gorilla

    Hi J

    Thanks for insightful response!

    Typically php slow log entries are due to a script taking more than 5 seconds to execute unless another timeout has been set.

    Yes, that’s right. It might be more or less than 5 seconds sometimes, depending on server, but in general – that’s exactly what it is. But my point was that the fact that the script is taking more than e.g. 5 seconds to execute doesn’t always mean that this script is a problem.

    Very often a given script actually takes more than this to execute because of other issues – because it depends on other script/resource that has to be executed/completed first and it doesn’t so it blocks it. It’s sometimes slow response of webserver/database caused by other script/request “overloading” it and so on… So it’s not always that simple.

    But yes, it is definitely an indication that “something’s going on”. I wasn’t really referring specifically to this specific case too, it was more of a “general remark” on slow log itself :slight_smile:

    My point is that on a server with zero traffic and not much else going on in terms of server resources (no highly intensive WP plugins, API calls, or DB interaction, for example), it is unusual for a PHP script to take more than 5 seconds to execute, which is what appears to be going on here.

    Agreed, it sounds like it shouldn’t be taking that much. In case of Broken Link Checker there’s one more thing to take into account though – it’s performance will also be affected to some degree by… the performance of the target servers of URLs that are being checked as each link that’d detected is also being checked. But yes, it is rather unexpected to outcome and does suggest that the script itself suffers some performance issues (and, as mentioned previously, our developers are still looking into it to get it improved over and over with future releases).

    As for the resource settings/configuration – I agree it shouldn’t be “up to you”. I believe it’s actually good that those settings can be changed but the main idea behind would be to treat it as a sort of “advanced settings” that you may want to “experiment with” mainly in two cases only:

    – if all is fine but you want to see if you can speed things up – like make plugin discover new links and check all links faster
    – and in case of troubleshooting if plugin unexpectedly fails or uses unusually (!) lot of resources

    while other than this, out of the box, there shouldn’t be a need to tweak those settings and even think of them :slight_smile:

    Like I mentioned previously, our developers are already looking into improvements related to performance and stability so with future releases that should be getting better over time but I’ve also passed over your feedback to our Broken Link Checker lead developers too for further consideration.

    Thanks again for the feedback and I made sure that some points are already flying your way :slight_smile:

    Best regards,
    Adam

  • J
    • Site Builder, Child of Zeus

    Thanks, Adam Czajczyk , I appreciate the points!

    No problem, hopeful my feedback is helpful. Makes sense that the links being tested could cause the delays, that is a good point.

    Perhaps have the PHP script that’s responsible for checking the links factor in the plugin’s “timeout” setting (from the “Advanced” tab)? This would allow it to calculate the estimated execution time for checking those links so that every time the script runs to check URLs, it is never taking on more URLs than it could handle (in the interest of keeping the script off the php_slow log, at least) – for example, if the Timeout setting is 30 secs, then never take on more than about 8 links to check per script execution. Many smaller executions of that PHP script rather than fewer, larger ones.

    Anyway, just an idea, my 2c… I’m sure you guys have great ideas about how to solve this. I know with programming and development there are many ways to solve a problem! It’s great that your devs are continuing to work on improvements.

    I look forward the the updates.

    Thanks again.