Repository navigation
lens: follow the sitemap robots.txt declares, and stop one probe outranking the score - #402
Merged
Merged
Conversation
…anking the score (#396) Two bugs a 64-site survey against Cloudflare's Agent Readiness scanner turned up on 2026-08-15, both of which made this rubric claim things it had not observed. SITEMAP. The probe only ever asked /sitemap.xml, and then passed any 200 it got back. Both halves were wrong. robots.txt is the authority on where a sitemap lives (RFC 9309 2.2.3) and the convention is only a convention, so Stripe's real sitemap at /sitemap/sitemap.xml read as missing, along with 13 others in the same corpus; that was the single largest source of disagreement with Cloudflare, which follows the directive. Meanwhile a site serving its SPA shell or a soft-404 at /sitemap.xml scored a valid sitemap, which is the worse of the two because it manufactures a pass rather than missing one. lensSitemapShape now decides whether a response IS a sitemap (urlset, sitemapindex, or the plain-text form sitemaps.org blesses), and lensSitemapDeclared reads the directive, through the same SSRF guard a visitor-supplied target gets. The follow-up probe is conditional, so a site whose sitemap sits where the convention says keeps the parallel fan-out untouched. It counts entries before trimming the body, because the discovery blob is cached under DISCOVERY_MAX_BYTES and going over that ceiling does not error, it just silently stops caching. Verified against the sites that were failing: stripe, netlify (cross-host), mistral and theguardian now pass, vercel is unchanged at the conventional path, and target and airbnb come back UNKNOWN rather than fail because their sitemaps are gzipped and this probe reads text. theguardian is one Cloudflare itself misses, since the declaration is http://. A 403 stays unknown too: x.ai has a sitemap and refuses our bot, which is our fact rather than theirs. LEVEL LADDER. lensProbeNlweb called any non-404 JSON at /ask "NLWeb-shaped", and an NLWeb door is an ACTION surface, which is the top rung. So 4 of the 6 sites this rubric published as Agent-Native earned it from a /ask that answered 410, 412, 429 or 401 -- one decade-dead API and three bot walls. A door now has to answer; a 401 counts only when it names its scheme, the same rule lensProbeMcp already applies at /mcp, and a 5xx reads unknown. The ladder also read one signal per rung while the score read twenty, so the two contradicted each other in public: github.com at 13/100 and walmart.com at 27/100 were both labelled Agent-Native. A rung may now claim at most one step beyond what the score supports -- one step rather than zero, because a site can genuinely ship a working agent interface while publishing none of the metadata the other checks look for, and that site should outrank its score. What it may not do is outrun the whole rubric by four rungs on one probe. When the cap bites, levelNote says so and rides the SSR badge as a title, which nav.js already renders as an infotip. Five contract tests, each naming the case it exists to stop.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
aadhar-sh | 379cbf2 | Aug 15 2026, 12:23 PM |
Contributor
Wire-size diff vs merge base
Brotli q11, the quality the edge serves. Every number here is deterministic: Client assetsNo change. 20 files, 73.98 KiB Brotli total. PagesNo change. 48 files, 496.95 KiB Brotli total. Worker bundle271.96 KiB → 273.06 KiB gzip (+1.10 KiB, +0.4%) Largest module deltas (raw bytes in bundle):
Dictionary deltas (
|
oddharsh
added a commit
that referenced
this pull request
Aug 16, 2026
…verse (#416) This gotcha claimed twice that the Copilot check fails on every diff, and one of those claims went in a few hours ago as the general lesson. It is wrong. The run always fires; whether it REQUESTS A MODEL depends on the diff. The job prints a file exclusion list on every run, covering js, json, mjs, ts, py, rs, yaml, yml, html, xml, go, java, c/cpp, rb and more. `.md`, `.sh` and `.css` are NOT on it. So a docs diff survives the filter and gets a `claude-opus-4.6` request, while a pure JavaScript diff is filtered to nothing and the run exits clean. Both paths print `Sessions disabled: not supported for code scanning yet`, which this session initially read as the discriminator and is not; the discriminating line is `Creating copilot-sdk session with model`, which only the failing path reaches. Correlates 12 of 12: the five runs of 2026-08-15 (#399 jpg/json and #402 js/mjs passed; #398, #400 and #401 all carried md and failed), plus all seven earlier failures checked backwards. #313 is the "CSS-only" diff this gotcha already singles out, and css is not on the list either. #351's note here called its entire diff a `${{ }}` move in workflow files; it touched md and mjs too, and the md is what drew the request. The prose-only argument is unchanged in conclusion and inverted in mechanism: a docs PR reddens this because markdown is one of the few things the detector still looks at, rather than because the agent choked on prose. The entitlement root cause is untouched. Records the method failure too, since it is the transferable part. Nine entries were built by pattern-matching outcomes across PRs without once reading which files each diff touched, and "fails on everything" was an artifact of that sample. One uncollected column explained the whole shape. Co-authored-by: Aadharsh Pannirselvam <19518661+oddharsh@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs a 64-site survey against Cloudflare's Agent Readiness scanner turned up on 2026-08-15. Both made this rubric claim things it had not observed, and both were visible only because a second rubric disagreed.
Sitemap
The probe only ever asked
/sitemap.xml, and then passed any 200 it got back. Both halves were wrong.It asked the wrong URL. robots.txt is the authority on where a sitemap lives (RFC 9309 §2.2.3); the convention is only a convention. Stripe's real sitemap at
/sitemap/sitemap.xmlread as missing, along with 13 others in the corpus — the single largest source of disagreement with Cloudflare, which follows the directive.It didn't read what came back. A site serving its SPA shell or a soft-404 at
/sitemap.xmlscored a valid sitemap. This is the worse of the two, because it manufactures a pass rather than missing one.lensSitemapShapenow decides whether a response is a sitemap (urlset,sitemapindex, or the plain-text form sitemaps.org blesses).lensSitemapDeclaredreads the directive, through the same SSRF guard a visitor-supplied target gets. The follow-up probe is conditional, so a site whose sitemap sits where the convention says keeps the parallel fan-out untouched, and it counts entries before trimming the body — the discovery blob is cached underDISCOVERY_MAX_BYTES, and going over that ceiling does not error, it just silently stops caching.Verified against the sites that were failing:
/sitemap/sitemap.xmlwww.netlify.com/sitemap-index.xml(cross-host)/sitemap-index.xmlhttp://www.theguardian.com/sitemaps/news.xml/sitemap.xml— unchanged.gz, and this probe reads texttheguardian is one Cloudflare itself misses, because the declaration is
http://.Level ladder
lensProbeNlwebcalled any non-404 JSON at/ask"NLWeb-shaped", and an NLWeb door is an action surface, which is the top rung. So 4 of the 6 sites this rubric published as Agent-Native earned it from a/askthat answered 410, 412, 429 or 401 — one decade-dead API and three bot walls.A door now has to answer. A 401 counts only when it names its scheme (the rule
lensProbeMcpalready applies at/mcp, so the two door probes finally agree about what a locked door is), and a 5xx readsunknown.The ladder also read one signal per rung while the score read twenty, so the two contradicted each other in public: github.com at 13/100 and walmart.com at 27/100 were both labelled Agent-Native. A rung may now claim at most one step beyond what the score supports.
One step rather than zero on purpose: a site can genuinely ship a working agent interface while publishing none of the metadata the other checks look for, and that site should outrank its score. What it may not do is outrun the whole rubric by four rungs on one probe.
When the cap bites,
levelNotesays so rather than quietly publishing a smaller number, and rides the SSR badge as atitle, which nav.js already renders as an infotip.Checks
pnpm run lintclean,pnpm run buildclean,pnpm run routes:check134 routes / 0 failures, contract suite 282 pass / 0 fail including 5 new tests, each naming the case it exists to stop.Note both fixes move scores DOWN for some sites and UP for others;
/lens/censussnapshots taken before this will not be comparable to ones taken after.