New to the community? Start here

[SmartCrawl Pro] llms.txt in Smart Crawl pro

7

llms.txt is a proposed standard to help large language models (LLMs) like ChatGPT understand and interact with website content more effectively.

Sun, 4 May 2025 5:13:36
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Amin Nazemi
    • Staff

    Hi Michael

    I hope you are doing well today.

    We already have a couple of feature requests for AI-generated content and LLMS, and our development team is considering adding those features; however can’t give you an ETA about when its going to be implemented.

    We appreciate your suggestion and believe that it can be a useful feature. To stay informed about new updates, you can subscribe to our roadmap.
    https://wpmudev.com/roadmap/

    Best Regards
    Amin

  • Patrick Freitas
    • FLS

    Hi Reklamebanken

    We don’t have an estimated time yet, but note that although this option exist most of articles still point out that AI companies are evaluating it and some articles indicates that at this moment Google won’t be using it.

    Not that I am not saying we should not use it, just that we keep close eyes in the news about the llmstxt approach.

    Best Regards
    Patrick Freitas

  • Sajjad Rahat
    • Staff

    Hi Julian ,

    Thank you for the updates here. Our team is reviewing this module and planning for it, but we don’t have an exact ETA for this feature. Hopefully, we will also adapt this soon, based on the feature’s needs and requests from more of our Members.

    Best Regards,
    Sajjad Rahat

  • Tony G
    • Mr. LetsFixTheWorld

    Solution described here.

    Google probably won’t support LLMS.txt until they can switch their audience toward Gemini as a primary search tool.
    Then they may suggest some standard text or a different file that will favor Gemini and cause other AI some grief.

    Against the development of support for this feature, I refer readers to this article, which as of 10/2025 reports that virtually no one is publishing llms.txt files, and none of the related companies read it. https://www.semrush.com/blog/llms-txt/
    (A single article is subject to be wildly inaccurate but it agrees with others.)

    I have an alternate suggestion that is more future-proof. Rather than just allowing maintenance of a hardcoded file like robots.txt or llms.txt, open it up for any filename. This doesn’t even involve SmartCrawl so no one needs to wait for WPMUDEV to figure out their position on the topic.

    Create a CPT that has field FileName in addition to Title and Content. On save of this CPT, the file will be made available at the endpoint specified by siteroot/FileName, with correct permissions.
    Um, that’s it. There’s no need to actually write the file to siteroot. The fact that the file has a .txt extension is irrelevant. All they care about is domain.tld/FileName. We do this every day in WP – it’s already a Content Management System.

    That single CPT will allow for maintenance of any file, now, or whatever happens to be devised in the future. CPTs support permissions, revisions, notifications, and other features that people would undoubtedly ask about later. Why bother with a new hardcoded solution in a proprietary plugin (or waiting for a year for some company to approve the request) when you can have this with your own WordPress environment now using existing tools?

    Maybe I or someone else should publish a plugin for this after a few more seconds of thought. It’s not a big deal that takes a year for a company, managers, and developers to theorize, architect, design, or fine-tune.

    If I did this, I might watch the industry and generate standards-compliant templates which can be used with the CPT … cuz WordPress has templates. But all that does is to centralize the functionality that we already have in to yet another dedicated plugin – I would be creating the solution that I’m rallying against here, just to satisfy those who want plugins. And it would turn off people who don’t like specialty plugins because they still think from 2005 that plugins somehow clutter WordPress.

    TL;DR : Skip SmartCrawl on this one, and the years-long wait for WPMUDEV to maybe make you happy. Use the tools in front of you, get what you want now, and move on to the next challenge.