New to the community? Start here

noindex URLs in Google Search Console that should not be

I have three URLs that are set to index through Yoast SEO including pages and posts. The website is set to index and I cannot find anything that should prevent crawling of these as they show up on the XML Sitemap. These are the three which are vital to my site and need to be indexed:

https://jncautoglass.com/service-areas/

https://jncautoglass.com/faq/

https://jncautoglass.com/gallery/

Tue, 4 Jun 2019 17:39:54
  • Nithin Ramdas
    • Support Wizard

    Hi Randy Robbins,

    That’s odd, just to be sure the results shared were from the Inspection tool, right?

    https://support.google.com/webmasters/answer/9012289?hl=en

    Do you notice any errors messages in the Inspection tool other than it’s not being crawled? I checked the sitemaps, and it does list the mentioned URLs in the sitemap which Google should have crawled.

    Could you please check the robots.txt file, and see whether there are any links being disallowed via it, and also check whether the sitemap.xml files are present in there, or not?

    robots.txt files are created in the root directory, that’s where the wp-config.php, .htaccess files reside.

    You can check the following article for the recommended content for robots.txt file:

    https://www.wpbeginner.com/wp-tutorials/how-to-optimize-your-wordpress-robots-txt-for-seo/

    That’s for example:

    User-Agent: *
    
Allow: /wp-content/uploads/
    
Disallow: /wp-content/plugins/
    
Disallow: /wp-admin/
    
Disallow: /readme.html
    
Disallow: /refer/
    

    
Sitemap: https://yourDomainURL.com/page-sitemap.xml

    Please let us know how that goes. Have a nice day ahead.

    Kind Regards,

    Nithin

  • Randy Robbins
    • New Recruit

    I just did a Live URL Test on those three pages and both the Service Areas and Gallery pages both were successful but for the FAQ page, I get the attached. For the other two, I requested that they be crawled. This site does not appear to have a Robots.txt file. Does this mean I should create one like the link you provided suggests?

    [attachments are only viewable by logged-in members]

  • Nithin Ramdas
    • Support Wizard

    Hi Randy Robbins,

    That’s odd, I don’t see any reference of “noindex” for the faq page when checked via the source code. Not sure whether there is any specific setting is Yoast side causing this too or not.

    Could you please enable support access to the dashboard, so that we could give a closer look at the settings in the dashboard side for the mentioned page too.

    Adding robots.txt file is a recommended action. It seems like when I check your website by going to the websiteURL.com/robots.txt, the file do load. I guess you have already made the changes.

    You can grant access from WPMU DEV > Support > Support Access > Grant Access, or check this manual: https://wpmudev.com/docs/getting-started/getting-support/#chapter-5

    Please let us know once you enable access so that we could get this sorted. Have a nice weekend. :slight_smile:

    Kind Regards,

    Nithin