New to the community? Start here

[SmartCrawl Pro] how to exclude certain pages from indexing on search engines

how can I use <meta name=”robots” content=”noindex,nofollow”/> on select pages that I don’t particularly what people to find so easily, like thank you confirmation page for example.

Sat, 7 Nov 2020 1:38:27
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Williams Valerio
    • Staff

    Hello Tim LaPaglia ,

    Hope you’re doing well.

    This can be achieved by going to the page where you want to add the <meta name=”robots” content=”noindex,nofollow”/> tag by going to the Posts/Pages section, and click on “Edit”.

    [attachments are only viewable by logged-in members]

    In this section, go to the very bottom where you are going to be able to see the SmartCrawl section and go to Advanced, you will see here the option to enable/disable the Index, if you want to add the tag, just disable that and it will be automatically added.

    To add <meta name=”robots” content=”noindex,nofollow”/>, the option should looks like this:

    [attachments are only viewable by logged-in members]

    Now you can go to the page/post and check if the tag was successfully added with Dev Tools, for example from Chrome, Right-click > Inspect, then CTRL/COMMAND + F and type Noindex, you are going to be able the meta tag added.

    This example is from Chrome:

    [attachments are only viewable by logged-in members]

    You can also check the documentation about the in post/page editor here: https://wpmudev.com/docs/wpmu-dev-plugins/smartcrawl/#in-post-editor

    Best regards,
    Williams

  • Williams Valerio
    • Staff

    Hello there,

    You are very welcome, we are glad to help :slight_smile:

    To successfully remove pages from Search engines, you should add the noindex, nofollow meta tag as I explained before, then you can set the disallow rule in your robots.txt and remove these pages from Google Webmaster Tools account.

    You can set the Disallow rule in robots.txt, directly into SmartCrawl if you are using the robots.txt file generated by SmartCrawl, you can see more info about the tool here: https://wpmudev.com/docs/wpmu-dev-plugins/smartcrawl/#robots-txt-editor

    NOTE: Keep in mind that if you already have a robots.txt file and click on Activate, the file is going to be replaced with a new generated by SmartCrawl.

    In order to use the Disallow rule, you only have to add the slug of the page that you don’t want the robots from search engines to reach after the “Disallow:” rule as you can see in this image:
    [attachments are only viewable by logged-in members]

    On the image, I’ve added two rules to disallow the search with robots in the customization section of the SmartCrawl tool, you can find it by going to SmartCrawl > Advanced Tools > Robots.txt editor in the Customization section.

    I found a good article about robots.txt here: https://moz.com/learn/seo/robotstxt

    For Google Search Console, you can follow the steps here: https://www.sistrix.com/ask-sistrix/google-index-google-bot-crawler/how-can-i-remove-a-url-on-my-website-from-the-google-index/

    Also, you can check our tutorial to stop indexing pages from your site here: https://wpmudev.com/blog/stop-google-indexing/

    Best regards,
    Williams