Blocking the crawler that trains a model and blocking the one that fetches your page to answer a question are two decisions.
Most robots files make only one of them on purpose. Here is what the agents are, what a robots file can do about them, and how to see which you have blocked by accident.
Separate agents, separate rules
Each vendor runs more than one crawler. A rule aimed at one name leaves the others untouched, so a site can block the crawler it wanted and admit the one it did not.
The split that matters is between a crawler that gathers text for training and one that fetches a page when somebody asks a question. The second produces a citation with a link. Refusing training while allowing retrieval takes two rules.
Vendors add, rename and retire agents, so read your own file end to
end, list every group and every Disallow, and compare
the result with what you meant.
What robots.txt cannot do
- It is not access control. It is a request that operators may honour. Anything private needs authentication.
- It does not remove you from an answer. A model that learned something before the rule still knows it, and a page another site quotes is still quotable.
- Blocking the directories that hold stylesheets or scripts can leave a retrieval path with markup it cannot make sense of.
What we check
Two of the thirty-three checks land here, worth eight of a hundred points. Crawler treatment matters, but it does not decide whether you get quoted.
- CR-02, robots and crawler directives. Settled by machine. It reads the file, page-level directives and relevant headers, and fails on accidental blocking of public content, contradictory directives, and blocked assets that stop a page rendering meaningfully.
- AC-01, intentional crawler treatment. Read by a person, who asks whether somebody chose the treatment. It credits the choice, not the direction: refusing training crawlers and admitting retrieval ones scores as well as admitting everything. A file inherited from a template scores badly either way.
Finding out what yours says
The free scan reads your robots file, your sitemap and up to eight pages, settles thirteen of the thirty-three checks, and shows the captured response behind each finding. It takes about twenty seconds, needs no account or card, and produces no score.
The full rubric is published, including the AI and agent controls dimension this page draws on.