Need help fast?

Get instant 24/7 live chat support for any WP issue Get instant support

New to the community? Start here

Alnaimi Abdulrahman
WPMU DEV Initiate
72 pts: Hero points 12 pts: Rep. points LEVEL 1

SmartCrawl SEO analysis 404 error

I want to run SEO ANALYSIS on my site. I’m using Sucuri firewall proxy for my website.

I have these IP’s already whitelisted:

66.135.60.59

66.135.49.214

66.135.60.64

104.236.238.22

104.236.50.140

But still get the following error on the homepage:

1 URLs that result in a soft error (404 etc)

Mon, 13 Nov 2017 12:50:03

JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

I hope you are well today. I’ll be following up with you regarding your chat.

I’ve been looking into this issue on your site and it appears you’re SSL implementation isn’t 100% complete. I’m getting the following Mixed Content messages on your homepage:
```
Mixed Content: The page at 'https://{*******}.om/' was loaded over HTTPS, but requested an insecure image 'http://{*******}.om/wp-content/uploads/2017/10/muscat_century_ride.png'. This content should also be served over HTTPS.

(index):629



Mixed Content: The page at 'https://{*******}.om/' was loaded over HTTPS, but requested an insecure image 'http://{*******}.om/wp-content/uploads/2017/10/muscat_century_ride.png'. This content should also be served over HTTPS.
```
Can you please fix these URL references then re-check your site and see if the issue persists?

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Dear James

I have forced the SSL for all site Content now, but the issue still there.

can you please start the investigation again?

Best Regards
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

I’ve been digging into this issue deeper with a number of tools. I can see that there are no redirects or 404 errors for me in the Eastern US. From my end, everything looks great. However, I had some other staff check the site and not all were able to access you’re URL. This led me to check your DNS. Here’s a report on your nameserver and A Record propagation across the globe.

https://www.whatsmydns.net/#NS/omancycling.om

https://www.whatsmydns.net/#A/omancycling.om

What’s even more interesting is these are fluctuating. So, there seems to be an issue with your DNS configuration. How this relates to SmartCrawl is the reason you’re getting the 404 error is because our SmartCrawl API can’t consistently see your site. I’ve verified this with our Dev Team.

Can you please work with your hosting/DNS provider to determine why you are having these DNS issues?

The following DNS report should assist them in their troubleshooting. There seems to be issues with your nameserver and SOA PTR records. http://dnscheck.pingdom.com/?domain=omancycling.om&timestamp=1510649117&view=1

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Hi James

Actually, we are redirecting all of our website throw proxy firewall.

https://sucuri.net/website-firewall/

Any traffic which is not user-generated it might get blocked.

So if I cant get the list of IP that needs to be whitelisted it might help.

I have these IPs already whitelisted:

66.135.60.59

66.135.49.214

66.135.60.64

104.236.238.22

104.236.50.140

Best regards
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

Attached is a list of the IP ranges we use with our Amazon services. This should cover everything. Please let us know if this resolves the issue for you.

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Hi James

I have whitelisted the IPs but the issue still there.

I have enabled the real-time monitoring of the firewall to monitor all the traffic coming to the website while I’m running the SEO ANALYSIS but I can’t see any traffic coming to the website from wpmudev.

any suggestions?

Best Regards.
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Hi James

for now, I dont get any errors but after the analysis, the result is waired :

0 Total Discovered URLs

0 Newly Discovered URLs

0 URLs in the Sitemap

any suggestions?

Best Regards.
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

Sorry for the trouble this has caused. Could you try this IP address please?

45.55.78.242

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
HI James

I did, and still it didn’t work.

Best regards
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

I’m afraid we’ve offered up every IP address and range we use with no resolution. The general consensus is that this is related to the DNS issues I pointed out above. However, to properly diagnose this, we would need you to temporarily disable your firewall and test. Would you please do so and see if the issue persists. If it’s not the DNS, then something in your firewall rules is blocking communication.

Please let us know.

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Dear James

What is the HTTP User-Agent for the SMARTCRAWL ??

Best Regards
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

I hope you are well today.

The user-agent for the SEO Audit Scan would be Incsub_Crawler.

Hope this helps!

Best regards,

James Morris
Alnaimi Abdulrahman
- WPMU DEV Initiate
- 72 pts: Hero points 12 pts: Rep. points LEVEL 1
Dear James Morris

Sucuri is asking for the debug result log of the plugin.

how can I get this file?

Best regards
JMorris
- I do geek stuff
- 1,394 pts: Hero points 7,335 pts: Rep. points LEVEL 25
Hello Alnaimi Abdulrahman,

I’m afraid the plugin itself is not what performs the scan. The SmartCrawl SEO scan is actually performed by our API servers. How this is handled is a bit complicated as there are a few layers in between and there’s no “plugin debug” that we can provide, I’m afraid.

Best regards,

James Morris