What is robots.txt? Explained with a sign in an office lobby
Robots.txt is a plain text file that tells crawlers and bots which parts of a website they are asked not to visit. Explained with a sign in an office lobby: polite, public, and not a lock.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- Engineering
This week Google, OpenAI, Microsoft and Anthropic said they will support one shared way for websites to say whether their content may be used for AI training, and it lives in a small file most sites already have: robots.txt.
In one sentence: robots.txt is a plain text file at the top of a website that tells automated visitors, called crawlers or bots, which parts of the site they are asked not to visit; well-behaved crawlers follow it, but it is a polite request, not a lock.
The everyday picture
Picture an office building with a sign in the lobby. It says: "Visitors welcome. Please do not enter the archive room. Couriers from Company X, please stay on the ground floor."
Every day, visitors come in to take notes about the building. A careful visitor reads the lobby sign first and goes only where it allows.
But the sign is just a sign. The doors behind it are not locked. A rude visitor can ignore it and walk anywhere.
A crawler reads the lobby sign (robots.txt) before anything else. It visits the rooms the sign allows, skips the ones it asks it to avoid, and copies what it reads into its notes.
How it works, step by step
- The file (the lobby sign). The rules sit in a file called robots.txt at the very top of the site, for example example.com/robots.txt. The internet standard for robots.txt, RFC 9309, says it must be plain text at that exact spot.
- The crawler (the visitor making notes). Search engines and AI companies run programs that visit pages and copy what they find. A crawler that follows the rules fetches robots.txt before anything else.
- The groups (lines on the sign for each kind of visitor). A "User-agent" line names a crawler, such as Googlebot. The lines under it say which paths it may or may not visit, using "Allow" and "Disallow". A group named with a star (*) covers every crawler that is not named elsewhere. When two rules overlap, the more specific one wins.
- The visit (walking only into open rooms). The crawler visits the allowed pages and skips the rest. Crawlers may keep a saved copy of the file for up to about a day, so a change is not noticed instantly.
- The new AI line (a note about what visitors may do with their notes). Until now, a site that wanted to opt out of AI training had to name each AI company's crawler, such as GPTBot or ClaudeBot. The new proposal adds one shared group that names a use, AI training, instead of a visitor. It states a wish about how content is used. It does not stop anyone from reading the pages.
Where the picture stops
- The sign is public, and so is the archive room's name. Anyone can read robots.txt, including people you would rather not invite. The standard warns that the paths you list are easy to discover, so never list a secret page there to "hide" it.
- Staying out is not the same as being forgotten. Google explains that a blocked page can still appear in search results if other sites link to it, usually without a description. To keep a page out of Google, it points to a "noindex" tag or a password instead.
- There is no guard at the door. In a real building, someone might notice a visitor in the wrong room. On the web, nothing enforces robots.txt. It works only because reputable companies choose to follow it.
Why it matters for your business
- Customers finding you. One wrong line, such as "Disallow: /" in the star group, asks every crawler to skip your whole site. That can quietly remove you from search.
- A say in AI training. The shared signal may soon let you express an AI training choice in one line, instead of tracking a list of company crawlers. It is still a draft standard, so check how each company applies it.
- Real protection needs real locks. Robots.txt is not security. The IETF group working on AI preferences has said such preferences are not technical access controls. Private files need logins and passwords.
An example
Imagine a small furniture shop with a website and a blog of design guides. The owner wants shoppers to find the shop in search, but does not want the guides used to train AI models.
Their developer leaves the normal search rules open, so the site stays visible. When the shared AI training signal is supported, they add one group asking that the guides not be used for training. The customer account pages sit behind a login, because a sign alone would not keep them private.
Frequently asked questions
Where do I find my website's robots.txt?
Type your domain followed by /robots.txt into a browser, for example example.com/robots.txt. If you get a "not found" page, your site has none, and crawlers may visit every page.
Can robots.txt stop AI companies from using my content?
It can ask. The four companies in this week's news say they will support the new shared signal, which is meant to make that request simpler. It cannot technically stop a crawler that chooses to ignore it.
Will blocking AI crawlers hurt my Google ranking?
It depends on the rule. Blocking a search crawler such as Googlebot affects search. The new AI training signal is a separate line, kept apart from the rules about which pages crawlers may visit. Check each company's documentation before you change anything.
Does robots.txt hide a page from search results?
No. A blocked page can still be listed if other sites link to it. Use a noindex tag or password protection to keep it out.
Related explainers
- What is an LLM? Explained with a new hire who read the whole library
- What is caching? Explained with a restaurant kitchen
- Frontend vs backend, explained with a restaurant