Allow at least three AI bots to access your site: GPTBot, OAI-SearchBot and PerplexityBot. In practice, add entries to your robots.txt file that allow them to crawl selected sections or the whole site. That gives your content a chance to appear in the AI answers where you want to be mentioned.
What are AI bots and how do they work?
AI bots are programs that visit websites and fetch content that later feeds the systems that generate answers. Unlike classic search engine crawlers, they don't index pages solely for organic results. They also need to understand context, so they can build answers in conversational tools and AI answer modules.
Technically they do the same thing you know from traditional crawling: they fetch a URL, check the rules in the robots.txt file and only then analyze the content. If robots.txt doesn't allow them in, they won't fetch data from your site. If it does, content from your site can be taken into account when an AI system formulates an answer.
That's why robots.txt is no longer just a tool for controlling bot traffic. Today it is also a way to deliberately manage where and how your brand can be mentioned in AI answers.
How do you configure robots.txt for AI bots?
For AI bots you use the same mechanisms as for other crawlers: User-agent sections and Allow/Disallow rules. The whole job comes down to three steps:
- Identify the AI bots you want to allow to access your content.
- Add rules to robots.txt that allow them to crawl the relevant directories or the whole site.
- Exclude the sections you don't want to make available (for example account panels, shopping carts, private resources).
An example of a safe setup that lets the three bots in across the whole site looks like this:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
If you want to be selective, letting AI bots into the sections with substantive content while blocking transactional areas, use per-directory rules:
User-agent: GPTBot
Allow: /blog/
Disallow: /konto/
User-agent: OAI-SearchBot
Allow: /poradniki/
Disallow: /koszyk/
User-agent: PerplexityBot
Allow: /case-studies/
Disallow: /panel/
Start with the least restrictive rules in your content sections, and add exclusions only where they are truly needed. Avoid a global "Disallow: /" on production domains unless you have a very specific reason and another way of distributing your content. Also remember that robots.txt is public, so treat it as an access policy that you deliberately communicate to all bots.
Which AI bots should you list in robots.txt?
If you care about your brand being present in AI answers, don't block these three bots:
- GPTBot – OpenAI's crawler, which according to the company's documentation fetches content that may be used to train generative models.
- OAI-SearchBot – an OpenAI bot that, according to the company's documentation, is used to show pages in the results of ChatGPT's search features.
- PerplexityBot – Perplexity's bot, which according to the company's documentation finds and links pages in Perplexity results; it is not used to train models.
Allow rules for these bots let them fetch content from your site. If you deliberately block them, your resources won't be available to these systems, and that lowers the chance your brand will be mentioned in AI answers.
Benefits of letting AI bots index your content
The main benefit of allowing GPTBot, OAI-SearchBot and PerplexityBot into selected content is a better chance that your brand will be mentioned in AI answers. This isn't about rankings, only about your name showing up in recommendations at all. If you have valuable content, the bots can use it when answering many types of queries, which gives you broad topical coverage.
You also influence the context: by making high-quality pages available, such as guides, FAQ pages or analyses, you give AI systems better material to cite. And selective Allow/Disallow rules let you combine your content strategy with your bot access policy instead of treating them separately.
Attent has analyzed millions of AI answers and shows, among other things, whether a brand was mentioned in an answer and which sources AI draws its knowledge from. 78 companies use it, which shows that monitoring visibility in AI answers is becoming a regular marketing process.
A well-configured robots.txt is now part of an AI answer visibility strategy. It lets you let the right bots into valuable sections and deliberately limit access where it makes no sense. Without it, it's hard to assess reliably why a brand is or isn't mentioned by AI systems.
Monitoring brand visibility in AI answers
Configuring robots.txt is only the beginning – you still need to check whether your brand is actually being mentioned. Attent (getattent.com) is a monitoring tool for brand visibility in AI answers. Every week it asks questions in 4 engines: ChatGPT, Gemini, Perplexity and Google AI (AI Overviews), and then shows whether the brand was mentioned, who AI recommends next to it, which sources it draws its knowledge from and what is worth changing. It reports a brand AI visibility score, share of voice against competitors, blind spots (questions where AI mentions competitors and not your brand) and the sources AI cites. It also sends alerts about changes, prepares a PDF report for the board, and has a content generator and a Google Search Console integration. The product is backed by Green Parrot sp. z o.o. of Warsaw, an agency with a team of over 100 specialists.
This kind of monitoring lets you connect your robots.txt policy with the real result, that is, your brand's presence in AI answers. You see which sections and content formats AI uses most often, and on that basis you decide whether to keep opening up or closing off parts of your site to selected bots.
The most common robots.txt configuration mistakes
The costliest mistake is a global block with no exceptions. A general "Disallow: /" for all bots means none of them will fetch your content, including GPTBot, OAI-SearchBot and PerplexityBot. Less obvious but just as dangerous are accidental exclusions: one careless "Disallow" at the directory level can close off an entire content section you want to make available.
Inconsistency between sections is another problem. If you allow bots into article pages but block supporting resources, such as folders with accompanying content, the pages they fetch may turn out to be less useful. Also don't copy a configuration 1:1 between domains, because different URL structures and different goals call for different Allow/Disallow rules. The last trap is no review after changes: after expanding a site or migrating it, it's easy to leave behind historical blocking entries that no longer fit the content architecture.
Preventing this is simple. Start with Allow rules for GPTBot, OAI-SearchBot and PerplexityBot in the sections you want to showcase, review robots.txt after every change in your architecture and adjust it to the new directories, and regularly check that the pages you want are accessible to those bots and that the rules don't conflict with each other.
We check this for you, every week, in four engines
In a 30-minute call, we'll show you how AI answers your customers' questions today and who it recommends instead of you.
Book a callFrequently asked questions
Which AI bots are the most popular?
In practice, when managing robots.txt and visibility in AI answers, the access most often considered is for GPTBot, OAI-SearchBot and PerplexityBot. These are bots tied to the AI answer systems where you want to be mentioned. By letting them crawl your substantive content, you increase the chance that your brand will appear in the answers.
What is a robots.txt file?
It is a public file in the root directory of a domain that sets access rules for bots. In the context of AI, it lets you selectively allow GPTBot, OAI-SearchBot and PerplexityBot into the sections you want to make available to answer systems. That gives you influence over what content AI tools can use.
What are the rules for using a robots.txt file?
Name a specific bot in the User-agent section and define access with Allow/Disallow rules. Start by allowing the relevant AI bots (for example GPTBot, OAI-SearchBot, PerplexityBot) into your valuable content while restricting sensitive areas. After changes to your site structure, update the file so the rules stay consistent and effective.
Which AI bots should be blocked in robots.txt?
Block those that don't support your goals, or the sections you don't want to make available. If you care about being present in AI answers, don't block GPTBot, OAI-SearchBot and PerplexityBot in your substantive content. Use Disallow for account panels, shopping carts and private areas to control access to sensitive resources.
What are the consequences of a wrong robots.txt configuration?
Rules that are too restrictive can keep GPTBot, OAI-SearchBot and PerplexityBot from reaching your content, which reduces the chances of your brand being mentioned in AI answers. Rules that are too permissive, on the other hand, can expose sections you don't want exposed. That's why selective configuration and regular review of the rules are key.
How do you monitor brand presence in AI answers?
Use a tool that regularly checks whether AI systems mention your brand. Every week, Attent (getattent.com) asks questions in ChatGPT, Gemini, Perplexity and Google AI (AI Overviews) and shows whether the brand was mentioned, who AI recommends next to it and which sources AI draws its knowledge from.
