New to the community? Start here

[SmartCrawl Pro] SmartCrawl triggering Wordfence

I keep running into this issue with other plugins as well, but more so with SmartCrawl and whitelisting the action never helps.

While in SmartCrawl I receive the message.

_______________

Background Request Blocked

Wordfence Firewall blocked a background request to WordPress for the URL /wp-admin/admin-ajax.php. If this occurred as a result of an intentional action, you may consider whitelisting the request to allow it in the future.

I do not want to keep putting Wordfence into learning mode as a workaround since this will put my sites at risk (is on a multisite w/ mission critical sites). Can this get properly fixed once and for all to figure out why this happens with your plugins?

We have developed closed-souirce plugins too that leverage admin-ajax.php and never run into this issue with those. Thanks for looking into this!

Fri, 7 Sep 2018 3:15:08
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Adam
    • Support Gorilla

    Hello Cave Consulting

    I hope you’re well today and thank you for your question!

    This happens when you run a SEO Check or URL crawler in SmartCrawl, right?

    These crawls are actually performed from “outside” of the site: SmartCrawl only “triggers” the process that’s crawling the site from our end. I’m a bit confused about the “admin-ajax.php” being blocked here because that should be a “local call”.

    When you’re white-listing, what exactly did you try to whitelist? These admin-ajax.php calls?

    Are there any IP locks put on that?

    I’ll forward the case to our developers but I’d like to get some more information first, to get a better picture, hence my questions above :wink:

    Best regards,

    Adam

  • Cave Consulting
    • Google Cloud Partner

    Hi Adam Czajczyk

    Not quite, we typically don't use the crawler. It just keeps appearing on every settings page in SmartCrawl while navigating to different pages in the dashboard. No IP lockouts occur, just prompts my Super Admin account to whitelist the background actions.

    Example is attached:

    [attachments are only viewable by logged-in members]

    I will turn on support access for the multisite where this is happening.

    Hope that helps, thanks.

  • Adam
    • Support Gorilla

    Hello Cave Consulting

    Thanks for your response and additional explanation.

    I wasn’t able to test that on your site because even though support access gives me a “super-admin” level access, when I tried to access dashboard of some sub-sites, I was redirected to a Google login (to login with Google account). I suppose it’s some sort of login/security process on your site.

    I’m mentioning it only for you to be aware that it actually “breaks” support access so it might make it more difficult to diagnose issues in future (hopefully there want be anything to diagnose though:stuck_out_tongue:).

    As for the issue in question. Thanks to the additional information from you and the screenshot I was able to actually replicate it on my own setup and find a cause for this. It happens because of one of the rules included by default with Wordfence Firewall, namely the “xss – XSS: Cross Site Scripting” rule.

    Disabling it immediately helps but on the other hand, it would also open the site for other, not that “friendly”, activities.

    I’m not exactly sure why Wordfence detects this as an xss attack but while in some other cases it’s able to recognize that it comes from a specific plugin, in this case it puts it under a “general” XSS rule.

    I checked with our developers and it’s been already reported but, to quite, it’s “not a bug, that’s just how both plugins work. I am totally not comfortable randomly disabling WF WAF rules. That defeats the purpose of using it”.

    That said, I suppose either whitelisting the action or letting WAF “learn” it, would be the safest solution, here.

    However, I have re-opened the issue with our developers as I think it needs to be addressed somehow anyway. I tend to agree with the quote that I posted above in terms of automatic whitelisting but I hope we’ll be able to find a better/other way to e.g. just detect WordFence WAF and notify site admin about necessary actions or an option to add some exception to Rules upon explicit consent from a site admin.

    Best regards,

    Adam