CiteUnseen

Rubric

AI and agent controls

Rubric v0.8 · 3 checks · 5 of 100 points

Is your crawler treatment a decision or a default?

Crawler treatment should be a decision you made rather than a default nobody chose. It is worth few points because it is usually already fine, and catastrophic on the rare occasion it is not.

Last updated · Written by Chris Brock, who wrote the rubric and signs the audits that use it.

Why this dimension exists

Five points across three checks, the smallest dimension in the rubric, which is about the weight the evidence supports.

The first check asks whether crawler treatment is a decision somebody made or a default nobody chose (<a href="../../ai-crawlers/">which AI crawlers your robots file actually addresses</a>). It credits the choice, not the direction: a site that blocks training crawlers and allows search ones has chosen. It does not credit a robots file inherited from a template, one that contradicts itself, or one that blocks the crawlers whose citations the business wants.

The second looks at machine-readable guidance: llms.txt and its neighbours (<a href="../../llms-txt/">what llms.txt is, and whether anything reads it</a>). It is not a ranking signal and no engine is known to require it. It is scored because a file for machines that asserts things the site does not is a defect.

The third concerns what an agent can do rather than read, and is scored lightly because the ground is still moving. WebMCP, the proposal that lets a page give an agent named functions, is not scored: it measures whether a site can be operated, not quoted. A paid audit reports what a site declares about it and states that we do not execute JavaScript, so we cannot confirm a tool works. <a href="../../webmcp/">What WebMCP is, and what it is not</a>, and <a href="../../webmcp/status/">which browsers support it today</a>.

The guides for this dimension.

  • llms.txt: what it is for, and why a wrong one is worse than none.
  • AI crawlers and robots.txt: which crawlers read which directive, and how to check what you have blocked.
  • WebMCP status: which browsers and agents implement it, read from primary sources and dated.

The 3 checks

1 of these 3 is deterministic, which means the free scan settles it without anybody reading the site.

AC-01 — Intentional crawler treatment

2 points · Proposed by machine, rated by a person

What is tested. Document relevant crawler directives and determine whether they align with the organization's stated goals.

What is recorded as evidence. Directives and owner-confirmed intent.

What fails it. Accidental denial of desired discovery; indiscriminate access inconsistent with policy.

How to fix it.

  1. Decide, per crawler, whether you want it: search crawlers, AI training crawlers, AI answer fetchers.
  2. Write that decision into robots.txt as named groups rather than leaving it to a wildcard.
  3. Match your CDN, WAF and bot-management rules to that decision. A rule at the edge overrides robots.txt in practice.
  4. Re-check after any CDN or hosting change.

How to confirm it is fixed. Request a page with each crawler's user agent and confirm you get what you intended.

Reference.

AC-02 — Machine-readable guidance quality

2 points · Settled by machine, from the evidence

What is tested. Inspect any llms.txt, AI policy, feed, API documentation, or equivalent machine-readable resource. Links inside a guidance file are checked for pointing off the audited domain only where the file is a link index, meaning llms.txt and ai.txt. A full-text file and a security policy are prose and are not held to that rule: no audit can describe itself without naming the site it audited.

What is recorded as evidence. Retrieved file, linked sources, consistency with canonical pages.

What fails it. Unsupported claims; stale generated summaries; secret material; treating an optional convention as a ranking guarantee.

How to fix it.

  1. Publish an llms.txt that points at the pages you most want quoted.
  2. Keep it to real, current, canonical URLs with a line each on what they cover.
  3. Generate it from the same source as your sitemap so the two cannot disagree.
  4. Serve it as text/plain at the root with a 200.

How to confirm it is fixed. Fetch /llms.txt and check every URL in it returns 200 and is canonical.

Reference.

AC-03 — Agent-facing transaction boundaries

1 point · Rated by a person

What is tested. Where the site exposes forms, APIs, purchases, or agent-oriented interactions, identify authentication and approval boundaries.

What is recorded as evidence. Public documentation, bounded interaction behaviour.

What fails it. Unclear commitments; unintended anonymous actions; instructions that bypass normal customer consent.

How to fix it.

  1. Decide what an automated agent is permitted to do on your site: read, enquire, book, buy.
  2. Say so in writing, somewhere an agent's operator can find it, and in your terms.
  3. Make the technical controls match the stated policy.
  4. Give a contact route for agent operators who need an exception.

How to confirm it is fixed. Read your own policy and check that the site enforces it.

Reference.