CiteUnseen

Rubric

Crawl and retrieval accessibility

Rubric v0.8 · 6 checks · 15 of 100 points

Can an answer engine actually fetch your pages?

If a retrieval system cannot fetch your pages reliably, nothing else on this list matters. Everything downstream depends on this working.

Last updated · Written by Chris Brock, who wrote the rubric and signs the audits that use it.

Why this dimension exists

Retrieval is often assumed rather than checked. A site that renders in a browser can still fail a fetch with no JavaScript engine, no session, and no patience for a four-hop redirect chain.

These checks come first because every other dimension assumes the page came back. If it did not, if the canonical host answers differently on the second request, if the body arrives as unrendered template output, or if the sitemap lists URLs that redirect, a strong score elsewhere describes a page no engine read.

This dimension is worth 15 of 100 points, which understates it. Treat it as a gate: a failure here caps everything downstream, and no structured data makes up for a page that does not reliably return.

The guide for this dimension.

The 6 checks

5 of these 6 are deterministic, which means the free scan settles them without anybody reading the site.

CR-01 — Stable successful retrieval

3 points · Settled by machine, from the evidence

What is tested. Retrieve the homepage and sampled priority pages using normal HTTP requests and a documented user agent.

What is recorded as evidence. Final URL, redirect chain, status code, content type, response time, response size.

What fails it. Repeated 4xx/5xx responses; redirect loops; inconsistent canonical host behaviour; empty successful responses.

How to fix it.

  1. Open every URL the report lists as failing and record the status code and the final URL after redirects.
  2. Fix the cause rather than the symptom: a 404 gets the page back or a 301 to its replacement, a 5xx gets the server error resolved, a timeout gets the slow query or upstream fixed.
  3. Collapse redirect chains longer than one hop so each old URL points straight at its final destination.
  4. Serve one scheme and one host. Redirect http to https and the non-canonical host to the canonical one, once.

How to confirm it is fixed. Request each listed URL again and confirm a 200 with at most one redirect.

Reference.

CR-02 — Robots and crawler directives

3 points · Settled by machine, from the evidence

What is tested. Inspect robots.txt, page-level robots directives, and relevant response headers.

What is recorded as evidence. Retrieved directives and the pages or agents affected.

What fails it. Accidental blocking of important public content; conflicting directives; blocked supporting assets that prevent meaningful rendering.

How to fix it.

  1. Read your robots.txt end to end and list every group and every Disallow.
  2. Confirm no rule blocks a page you want quoted. A Disallow on a directory blocks everything under it.
  3. Decide explicitly about AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Silence is an accident; a named group is a decision.
  4. Keep the file under 500 KiB, return it as text/plain with a 200, and name your sitemap in it.

How to confirm it is fixed. Fetch /robots.txt, confirm a 200 and text/plain, and check each crawler you care about against the groups.

Reference.

CR-03 — Sitemap quality

2 points · Settled by machine, from the evidence

What is tested. Discover declared and conventional sitemap locations, parse sitemap indexes, and sample listed URLs.

What is recorded as evidence. Sitemap URLs, parsing status, last-modified values when present, duplicate rate, invalid or redirected entries.

What fails it. Missing priority content; extensive broken URLs; inconsistent canonical hosts; obsolete entries.

How to fix it.

  1. Generate the sitemap from the same source that generates your routes so it cannot drift.
  2. Include only canonical, indexable, 200-returning URLs. Drop redirects, 404s and noindex pages.
  3. Set lastmod from the content's real modification date, not the build time.
  4. Reference the sitemap from robots.txt and submit it in Search Console.

How to confirm it is fixed. Fetch each URL in the sitemap and confirm every one returns 200 and is self-canonical.

Reference.

CR-04 — Canonical and duplicate control

2 points · Settled by machine, from the evidence

What is tested. Compare canonical declarations, redirects, alternate hosts, protocol variants, and representative parameterized URLs. Read the status of each redirect a sampled URL passes through, not only where it lands.

What is recorded as evidence. Canonical targets and conflicting examples.

What fails it. Canonical loops; cross-domain errors; multiple indexable versions; important pages canonicalized elsewhere without explanation; internal links that reach their destination through a temporary redirect, which is not read as a statement about which address is canonical.

How to fix it.

  1. Give every page a self-referencing absolute rel=canonical.
  2. Pick one form per page and make it the only one: trailing slash or not, www or not, http or https.
  3. Point parameterised, filtered and paginated variants at the canonical page rather than at each other.
  4. Make the canonical agree with the sitemap and with any og:url on the same page.
  5. Point internal links at the address that answers directly, and where a redirect has to stay, make it permanent (301) once the destination is settled: a temporary redirect is not read as a statement about which address is canonical.

How to confirm it is fixed. Load each duplicate variant and confirm the canonical it declares is the same single URL, and request each internal link that redirects and confirm the status is 301 rather than 302, 303 or 307.

Reference.

CR-05 — Rendered content availability

3 points · Rated by a person

What is tested. Compare raw HTML with rendered content for sampled pages.

What is recorded as evidence. Text and structured elements available before and after rendering.

What fails it. Primary claims, navigation, headings, or supporting evidence unavailable without brittle client-side execution.

How to fix it.

  1. Load each sampled page with JavaScript disabled and compare what remains against what a visitor sees.
  2. Move the primary claims, the H1, the navigation and the body copy into the server-rendered HTML.
  3. Where a framework is doing the rendering, turn on server-side rendering or static generation for these routes.
  4. Keep client-side rendering for what it is good at: interaction, personalisation, anything behind a sign-in.

How to confirm it is fixed. Run curl on the page and read the raw HTML. If the answer is not in there, no crawler has it either.

Reference.

CR-06 — Body text is machine-readable prose

2 points · Settled by machine, from the evidence

What is tested. Check the visible body text of sampled pages for template syntax that survived into what a reader and a retrieval system see: page-builder shortcodes, unresolved merge fields, raw template tags. Metadata is excluded. Syntax inside a meta attribute or a script block is not body text.

What is recorded as evidence. The visible text extracted from each sampled page, For each affected page, the syntax found and the share of visible text it occupies.

What fails it. Page-builder shortcodes printed as body text, so the passage a retrieval system reads is markup rather than prose; Unresolved merge fields such as {{first_name}} left in published copy; Raw template tags surviving a build and rendered to the reader.

How to fix it.

  1. Find the pages the report names and look at how their body copy is marked up.
  2. Put prose in <p> elements. Text held only in <div>, table cells or image alt is text a parser has to guess about.
  3. Replace text baked into images with real text, keeping the image if it earns its place.
  4. Give each page enough body prose to stand on its own rather than relying on the page it links to.

How to confirm it is fixed. View source on the page and confirm the paragraphs are paragraphs.

Reference.