New to the community? Start here

Robots txt is not showing the changes added in SmartCrawl

I already activated robots.txt and linked sitemap in SmartCrawl robots.txt settings. But the changes are not added in the robots.txt file.

Mon, 9 Mar 2020 12:51:27
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Adam
    • Support Gorilla

    Hi Ncy

    I hope you’re well today!

    I checked the site but I must say I didn’t find anything in configuration of the site itself that might be causing it. It would be good to test if the same happens with the theme different than the current one (preferably test with Twenty Twenty) but since it’s a live site I didn’t do such test as I didn’t have your permission.

    Having said that, I asked our SmartCrawl developers to check it again after me so they’ll look into it and I believe they’ll be able to find the culprit.

    Please keep an eye on this ticket and we’ll update you here as soon as we get to know more.

    Kind regards,
    Adam

  • Maciej Palmowski
    • Recruit

    Hi Ncy

    I just did a test on my localhost and everything looks fine. I also checked for a moment what will happen if I’ll upload a blank robots.txt file – than it’s served (the blank one).

    I’m starting to think that your hosting probably does something like this:
    – it checks if there is a robots.txt file – if it’s present then it shows it
    – if there is no file it serves the default one provided by hosting

    Smartcrawl doesn’t create a file but uses WordPress filters to simulate the behaviour. And this is probably the reason of the conflict – your hosting doesn’t get a file, so it serves the default robots.txt.

    Could you check or ask you hosting company is there a way to turn this default robots.txt off?