Repository navigation
feat(research): follow same-site subpages in web_read - #454
Conversation
github-actions
Bot
added
size:L
vouch:trusted
|
What: Scoped proof passed on PR #454 head Base:
The MCP tests verify default matching, cycles, normalized duplicate URLs, overrides/disable, page/text/time limits, partial failures, cached continuation, concurrent sharing/freshness, scoped redirect refusal, public/private destinations and exact origin counts. The orchestration integration drains No complete-repository or live multi-environment/adoption claim is made; CI and #447 own those respectively. No version bump under project policy. |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Independent review of PR #454 at head ffe993bef5e973c01142dcb484ef572daeffeff2 (base 763156d7667fc56fce088a06b40a10e7cd88ce47, feat/423-web-read-core) found no blocking findings, only two low-severity cache/budget inefficiencies.
Why: The subpage slice meets #441's acceptance: default and overridable literal matching, normalized same-origin depth-one candidates, scoped redirect refusal, per-page provenance, shared 20,000-character and 30-second budgets with explicit continuation, and retained Effect Cache cancellation.
So what: review/independent is success on this SHA. The two notes are optional follow-ups for the author; merge stays with the user after #449 lands.
Findings
1. Low, non-blocking: a scoped refusal shadows a valid explicit-read cache entry
apps/server/src/webResearch/WebResearch.ts:356-369, with PageCache.get({ url }) returning only the newest row for a URL (apps/server/src/webResearch/PageCache.ts:58).
Trigger:
web_read({ url: "https://a.example/camere", followSubpages: false }), where/camere302s to an external booking engine. This stores an unrestrictedokentry with a seven-day TTL and an off-originfinalUrl.web_read({ url: "https://a.example/" })links/camere. Traversal correctly refuses to reuse that entry, so a restricted fetch writes a newerunsupportedrow withrestrictedToOrigin.- Repeat step 1 within seven days.
cache.get({ url })now returns the refusal row. The explicit-read branch rejects it (restrictedToOriginset, off-originfinalUrl), so the read requestsa.example/camereand the external target again instead of reusing the still-valid entry from step 1.
Effect: the explicit result is correct (no poisoning), but a TTL-valid read pays origin requests and reports rung: "http". The handler test covers only the refusal-then-explicit order, not explicit, then refusal, then explicit. One possible repair is to select the newest row compatible with the caller's restriction.
2. Low, non-blocking: a redirect onto an already-returned page is refetched and duplicated
apps/server/src/webResearch/WebResearch.ts:470-503.
fetched skips a candidate only when the candidate's own URL equals an earlier final URL. A later candidate that redirects onto the root or an earlier child is fetched in full. Examples: the root links /camere then /rooms, and /rooms 302s to /camere; or /camere 302s to /. The restricted fetch requests the alias and then the target again (the hop loop at lines 218-251 does not know the pages this read already holds), and subpages receives text already returned. That spends a page slot and part of the shared output budget on duplicate text, which can push real pages into pendingSubpages. The existing alias test covers only the alias-before-target order.
Cleared
- Default pattern,
followSubpages: false, case-insensitive literal|override without regex evaluation, empty-term rejection, andmaxSubpages1-6 schema bounds. - Normalization of fragments, host case, default ports and dot segments; root self-links and the root's redirect endpoints are skipped; children's links are never followed.
- External, credentialed, non-HTTP and private literal links are excluded by origin. Same-origin rebinding still goes through
PublicHttpand returnsblocked_local. Cross-origin HTTP and meta-refresh redirects are refused before robots or any request to the new origin. - Scoped refusals are never returned to explicit reads, and restricted flight keys never join unrestricted flights.
- Effect 4.0.4
Cache:PageRequestkeys hash and compare structurally.timeoutOptionraces in a forked fiber, so a timed-out child releases its awaiter, and the lookup is interrupted only when no other reader waits. Root sharing is unchanged. - Budgets: root plus child text never exceeds
maxChars(at most 20,000). The deadline runs from read start; the TestClock test ends at exactly 30,000 ms with the slow child interrupted.pendingSubpagesis a correct contiguous slice of at most six,unvisitedSubpagescounts the rest, and every limit sets a truncation reason. - Partial failures keep successful siblings, each with its own status,
httpStatus, hash and cache id. - Old cache entries decode because
linksandrestrictedToOriginare optional; anchorless roots are refetched only when following. Cache-id and nonzero-offset reads never traverse. - Claude and Codex T3 MCP tool timeouts are 65 minutes, so the 30-second budget fits. The MCP tool is the only consumer of
WebReadResult. - Scope: server/MCP only. No recursion, rendering, forwarding, search, session or UI changes.
Proof checked
/tmp/chromeria-441-tests-final.log: 4 files, 61 tests passed;/tmp/chromeria-441-proof-headmatches this head.- Typecheck log: suggestions only, no
error TS, none inwebResearch. - Final lint and knip logs show no findings. Their exit codes appear only in the author's proof note, not in the logs.
- I did not rerun checks. GitHub CI Lint, Typecheck and Test were still running during this review.
- Both findings come from reading the exact head source; I did not execute them.
Reviewed by Claude Opus 5.5 (claude-opus-5-5) through the Claude Code harness in T3 Code, dispatched at xhigh effort. Read-only review: no checkout, branch change or repository edit.
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
|
What: Both low-severity findings from the Opus review are accepted and reproduced by new MCP tests. The first regression reproduces explicit read, then restricted refusal, then explicit read: the original head incorrectly reports The cache owner will select the newest entry compatible with the caller's origin policy using indexed metadata. The disposable index gains nullable origin columns while existing cache ids remain readable. Restricted redirects can reuse a compatible fresh target, preserving the target body's actual fetch time and expiry; redirect freshness participates in restricted in-flight identity. Traversal skips an already-returned successful final URL and counts alias attempts toward the page budget. This does not change #449's branch or unrestricted shared-fetch cancellation. Elon record
|
ffe993b to
ff27f54
Compare
|
What: Scoped proof passed on PR #454 head Base:
The MCP tests verify default matching, cycles, normalized duplicate URLs, overrides/disable, page/text/time limits, partial failures, cached continuation, concurrent sharing/freshness, scoped redirect refusal, public/private destinations and exact origin counts. The orchestration integration drains No complete-repository or live multi-environment/adoption claim is made; CI and #447 own those respectively. No version bump under project policy. Both findings from review 5563998782 were reproduced before fixing and now pass. Added cases also cover cached redirect targets with forced freshness, alias publication after expired failures, root queue expiry without false origin cache failures, and reopening the original three-column index. All latest core queue/active-HTTP/cache-startup/race regressions pass, with single-page queue intent explicit and a separate default traversal deadline test. Opus follow-up review is running on this exact head. |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Follow-up independent review of PR #454 at head ff27f54bf76f55e9078e306818b5d01e7c9c10d6 (base 362cf419dcf4fe13027f9b3d958ab2e091336287, feat/423-web-read-core) found no blocking findings. Both earlier findings are fixed, and two new low-severity notes remain.
Why: Policy-aware cache selection and alias deduplication work as claimed, the rebase keeps every #449 round-2 guarantee, and the cumulative #441 acceptance holds. One note is a cache-poisoning path from hostile page content; the other is a leftover refetch case from finding 2.
So what: review/independent is success on this SHA. Both notes are optional author follow-ups; merge stays with the user after #449 lands.
Findings
1. Low, non-blocking: a hostile page can store a cache body the cache cannot read back
Locations: apps/server/src/webResearch/extract.ts:98 stores every anchor's label and absolute URL beside the text. apps/server/src/webResearch/PageCache.ts:91 reads bodies back with a fixed 32 MiB gunzip limit. apps/server/src/webResearch/WebResearch.ts:458 (and :290 for redirect targets) calls cache.get before anything else, and nothing catches its error.
Trigger: a page under the 4 MiB HTTP cap whose anchors are filled with C0 control characters, e.g. <a href=/> followed by 256 \u0001 characters, repeated about 15,000 times. JSON.stringify escapes each control character to 6 bytes, and the new links array repeats every label, so the cached JSON is twice the core's size.
Evidence: a disposable probe with the real extract, classify and PageCache's gunzip limit. A 4,050,055-byte page classifies ok with 4,170,000 text characters. Its body without links (the core's shape) is 23.4 MB and reads back. Its body with links is 47.0 MB, and reading it fails with ERR_BUFFER_TOO_LARGE. The threshold is about 2.9 MB of such HTML. The MCP path from that point follows from source; I did not run it end to end.
Effect: the first read succeeds and stores the body. After that, every read of that URL fails with orchestration_error ("The research page cache could not be read or written"), including maxAgeHours: 0, because every path calls get before it can store a newer row. If that page is a same-origin child, every default traversal of its root fails outright, losing the root and all successful siblings, until someone deletes web-cache. Only hostile same-origin content triggers this; ordinary ASCII link-heavy pages stay near 3x their HTML size. Possible repairs: bound the stored links, size the read limit for the largest body the write path can produce, or treat an unreadable URL row as a cache miss.
2. Low, non-blocking: an alias onto a page known only by its final URL still costs one extra request
Locations: apps/server/src/webResearch/WebResearch.ts:289-307 checks only for a row whose url equals the redirect target. apps/server/src/webResearch/PageCache.ts:112-113 indexes each page by its request URL only.
Trigger: the root redirects (e.g. https://hotel.example/ to https://www.hotel.example/it/, or /start to /home/), and a matching child redirects or meta-refreshes to the root's final URL. A second case: two aliases, /rooms and /suites, both redirect to /camere/, and /camere/ is not itself a candidate. The alias's target hop finds no row for /home/ (or /camere/), so it requests that page again before the traversal's duplicate check drops the result.
Effect: output stays deduplicated and the slot is counted as designed. The only cost is one extra origin request, plus at least 3 s of spacing out of the 30 s budget, per such alias. The new tests cover only targets that have their own URL row (an unredirected root /, or /camere). The /start to /home/ test has no child redirecting to /home/. Found from source; not executed.
Earlier findings, rechecked
- Finding 1 (policy shadowing) is fixed. Unrestricted lookups select
restriction_origin IS NULL OR final_origin = <url origin>. Restricted lookups selectrestriction_origin = S OR final_origin = S OR final_origin IS NULL. Both matchacceptsOrigin, so the newest compatible row wins. A restricted fetch only ever targets its own origin, sorestriction_originis always NULL or S. The new MCP case runs explicit read, then scoped refusal, then both again: it returns the original cache ids with no new requests. Scoped refusals still never reach explicit reads, and target reuse cannot spread a refusal, because a refusal's final URL is off-origin. - Finding 2 (late aliases) is fixed for targets that have their own row. HTTP and meta-refresh aliases onto a returned child or an unredirected root skip duplicate text, with the exact request list asserted. The reverse order skips via
fetched. Alias attempts count towardmaxSubpages, andpendingSubpagesstays a correct contiguous slice.maxAgeHours: 0disables target reuse (isFreshis false at 0), and the flight-key freshness agrees. Reused targets keep their bodyfetchedAtandexpiresAt, andrequestedAtstops a forced flight from accepting an alias row with an older body. Separating publication order from body age makes the recovered-alias case correct.
Rebase onto 362cf41
request's active-HTTP budget, kept separate from queue and spacing, is unchanged. The disabled-cache service on open failure is kept: the migration runs insidemake, and its errors map toPageCacheError("open"). The late storage recheck is kept and now also enforces origin policy, links and forced freshness.- The 30 s root deadline applies only when following.
followSubpages: falseand nonzero-offset URL reads keep the core's unbounded queue wait. An expired root returnsWebResearchBudgetError, mapped toorchestration_errorwith retry guidance. The interrupted flight writes no cache row, as the Crawl-delay 40 case shows. This is a deliberate, documented trade-off: with following on (the default), same-origin reads queued past 30 s get a retryable error instead of waiting. The existing queue and active-HTTP tests opt out explicitly, and their oracles are unchanged.
Cumulative acceptance, cleared
Default pattern and limits (at most 6, schema 1-6), disable, and case-insensitive literal | overrides with empty terms rejected. Normalized depth-one candidates from the root's final URL, with cycles and duplicates skipped and children never traversed. External, private and credentialed links excluded, and off-origin redirects refused before robots or any request. Per-page provenance and failures that keep successful siblings. A 30 s total deadline and a shared 20,000-character output budget with explicit reasons and pending continuation. Cache-id and nonzero-offset reads never traverse. Old index columns are added and old bodies decode. Scope is server/MCP only: no recursion, rendering, forwarding, search, session or UI work.
Proof checked
/tmp/chromeria-441-tests-final.log: 4 files, 74 tests passed. It was written at 19:27:05, after the 19:25:29 amend that produced this head, and/tmp/chromeria-441-proof-headmatches.- The local typecheck, lint and knip logs (19:24:32 to 19:24:45) predate that amend, so they cannot be tied to this commit. That gap is covered by GitHub CI on this exact SHA (run 39149558306): Typecheck, Lint, Test, Test Server 1-6 and Test Web all pass.
- Proof gaps, not defects: the old-index test reopens a downgraded index only by
cacheId. No test covers a URL lookup over legacy NULL-metadata rows, or a follow read refetching an anchorless legacy root. The source handles both. - I did not rerun the suite. The only execution was the disposable probe in
/tmpdescribed in finding 1.
Reviewed by Claude Opus 5.5 (claude-opus-5-5) through the Claude Code harness in T3 Code, at xhigh effort. Read-only review: no checkout, branch change or repository edit.
|
What: Both new findings from the second independent review are reproduced and repaired in the #441 layer. Review: #454 (review), on
Requirements and who asked: The user requested findings fixed; #441 requires bounded caching, shared budgets and duplicate suppression. |
|
What: The final #441 candidate passes 77 focused MCP, network and integration tests after both second-round findings were fixed. Head: All final checks ran after the commit on a clean checkout of this exact head:
The two new MCP regressions failed on Tests exercise the registered MCP boundary with the real reader and compressed cache, external HTTP fixtures, TestClock/Deferred milestones and no sleeps or polling. Orchestration integrations drain Local logs: |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Round 3 independent review of PR #454 at head d2d6dd3d7094ad9bed92e69dcf0c145e1cdb88a4 (base 362cf419dcf4fe13027f9b3d958ab2e091336287, feat/423-web-read-core) found no blocking findings. Both round-2 findings are fixed. One new low-severity note remains.
Why: The final-URL lookup removes the duplicate alias requests, and the shared 64 MiB ceiling stops unreadable cache bodies from being published. That ceiling can still be reached by crafted nested anchors, and a child that hits it fails the whole traversal.
So what: review/independent is success on this SHA. The note is an optional author follow-up. Merge stays with the user after #449 lands.
Finding
1. Low, non-blocking: a crafted same-origin child can still exceed the 64 MiB ceiling, and its write error fails the whole traversal
Locations: apps/server/src/webResearch/extract.ts:98 stores a label of up to 256 characters for every anchor, nested ones included. apps/server/src/webResearch/PageCache.ts:113-117 rejects JSON over MAX_CACHE_JSON_BYTES (:22). apps/server/src/webResearch/WebResearch.ts:419 yields cache.put without recovery. The child load at WebResearch.ts:606-609 does not catch it either, and apps/server/src/mcp/toolkits/webResearch/handlers.ts:15 maps it to orchestration_error.
Mechanism: in HTML, <a> elements cannot nest. In MathML foreign content the parser does nest them. Each ancestor's label is the first 256 characters of its inner text, so one run of 256 control characters under 256 nested anchors is repeated in about 32,900 label characters, each escaped to 6 JSON bytes. The PR body says the 64 MiB ceiling "accommodate[s] JSON escaping in retained labels and text". That holds only while each label character also appears once in the text, which nesting breaks.
Trigger and evidence: a disposable probe used this head's real extract, classify and CachedPage JSON encoder. The page was <math> + 256 x <a href=x> + 256 x \u0001 + </math>, repeated.
| Chains | HTML bytes | Status | Serialized JSON bytes | put |
|---|---|---|---|---|
| 260 | 735,679 | ok |
64,996,347 | accepted |
| 270 | 763,969 | ok |
67,496,187 | rejected |
| 300 | 848,839 | ok |
74,995,707 (2,534,908 without links) |
rejected |
All three are far below the 4 MiB HTTP cap and the 4M-character text cap.
Effect: the oversized body is never published, so the round-2 persistence is gone. But when such a page is a matching same-origin child, PageCacheError("write") fails the entire web_read with "The research page cache could not be read or written." The root and every successful sibling are dropped. The child is never cached, so every later default traversal of that root requests the child again and fails again. Reads with followSubpages:false still work. This contradicts the PR body's "failures preserve successful siblings", and the message blames the cache for what is a page-size failure. Only deliberately crafted same-origin content triggers it; ordinary pages do not nest anchors. I followed the error path from source and did not run it end to end through MCP.
Possible repairs, if wanted: bound the stored link metadata separately from the text, or turn an oversized page write into an uncached per-page failure so the siblings survive.
Round-2 findings, rechecked
- Unreadable cached body: fixed for the reported mechanism. The writer now rejects any JSON that the reader's 64 MiB
gunziplimit could not return, before the body or the index row is written (PageCache.ts:113-131). Reads therefore always cover writes. The 11,000-anchor MCP regression is about 34 MB of JSON, so it would have failed under the old 32 MiB read limit. The note above is the only remaining path to the limit. - Aliases onto final-URL-only targets: fixed.
redirectTarget(PageCache.ts:75-79) matchesurlorfinal_urlwith the same restricted-origin filter. Only restricted redirect hops use it (WebResearch.ts:289-290). Unrestricted URL reads still match onlyurl(:80-87).- A
final_urlmatch always hasfinal_originequal to the traversal origin, so thestatus === "ok"gate (:129) is what keeps refused redirects from standing in for a target. The new foreign-site regression covers this. - Successful restricted rows always end on their restriction origin, so no row can cross policies.
- An unrestricted successful row that reached the target through a cross-origin redirect holds the target's real body, and robots was checked on every hop.
- The new test asserts the exact eight-request list. Under ff27's URL-only lookup,
/home/and/camere/would each be requested twice. - Legacy rows get
final_urlNULL without any body scan. They only lose alias reuse.
- A
- The integration test's in-memory cache returns
nullforredirectTarget. That only disables reuse; the queue and active-HTTP oracles are unchanged. - The legacy-index reopen test now also reads by URL over NULL-metadata rows, closing that round-2 proof gap. A follow read that refetches an anchorless legacy root is still untested; the source handles it.
Cumulative acceptance
The base is unchanged at 362cf419 (#449's live head), and the only changes since round 2 are the two cache changes above, so the round-2 acceptance clearance still holds:
- Default pattern, at most 6 subpages, disable, and literal case-insensitive
|overrides. - Normalized depth-one candidates and cycle and duplicate skipping.
- External, private and credentialed destinations refused, along with off-origin redirects.
- Per-page provenance, the 30 s deadline including the root queue, the shared 20,000-character output, and explicit truncation with pending continuation.
- Cache-id and offset reads never traverse.
- Server/MCP scope only.
The tool description and orchestrationInstructions.ts guidance remain accurate.
Proof checked
- The commit is dated 19:48:24 +0200.
/tmp/chromeria-441-r3-proof-identity(19:48:58) holds this head and tree471672c3621e3217321bdb3ca16314369bdbe733, matchinggit log. /tmp/chromeria-441-r3-tests-final.log: started 19:48:59; 4 files and 77 tests passed in 15.74 s.- The typecheck log (19:49:15) has 0 errors, only existing suggestions. The lint log has no findings and the knip log is empty. All three were written after the commit, which closes the round-2 timing gap. The logs do not record exit codes, but their output agrees with exit 0.
- GitHub CI on this exact SHA passed: Typecheck, Lint, Test, Test Server 1-6, Test Web, Build and Check. One duplicate "Native fingerprint diff" run was cancelled and its rerun passed.
- I did not rerun the suite. The only execution was the disposable
/tmpprobe described in the finding.
Reviewed by Claude Opus 5.5 (claude-opus-5-5) through the Claude Code harness in T3 Code, at xhigh effort. Read-only review: no checkout, branch change or repository edit.
|
What: The nested-anchor finding is reproduced through MCP and repaired at the extraction boundary. Accepted finding: #454 (review), on The new registered-MCP regression uses 270 chains of 256 MathML anchors containing control characters, between two normal same-site children. It failed on that head with the reported cache error, losing the entire traversal. The fix gives retained link URLs, labels and per-entry overhead an independent 4 Mi-character extraction budget. This bounds metadata even when ancestor labels repeat descendant text. Existing extraction error handling returns a visible The 4 Mi-character text and metadata budgets together serialize to at most approximately 48 MiB under worst-case six-byte JSON escaping, plus small fixed provenance fields, below the existing shared 64 MiB cache ceiling. The ceiling is unchanged. Requirements and who asked: The user requested findings fixed; #441 requires bounded extraction and per-page failures preserving the root and successful siblings. |
d2d6dd3 to
d3cf0f5
Compare
|
What: The rebased #441 candidate passes 83 focused tests and all scoped static checks, with the nested-anchor finding fixed. Head: All final checks ran after committing this exact head on a clean checkout. Each log records the exact head and
Third-review finding and repair: the real-cache MCP regression failed on The rebase retains the new core's 60-second origin-queue limit, uncached backoff refusals, separate 30-second active-HTTP budget and cache-open diagnostics. The 24-read semaphore-contention test initially failed under default subpage following because this layer's total traversal deadline is shorter. It now explicitly sets Tests exercise the registered MCP boundary with real reader/cache and external HTTP fixtures, TestClock/Deferred milestones and no sleeps or polling. Integrations drain Local evidence: |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Round 4 independent review of PR #454 at head d3cf0f548ee922115e9c91f84b7b1bcdbe2c5447 (base 7a4b1f1537928d7193ca090d726547f788a2e5fc, feat/423-web-read-core) found no blocking findings. All five earlier findings are fixed, the rebase keeps the new core queue fixes, and one new low-severity note remains.
Why: The new link-metadata budget stops nested-anchor amplification, but it also rejects flat, link-dense pages that the core reads, including reads with followSubpages:false.
So what: review/independent is success on this SHA, and the note is an optional author follow-up. #449 has since moved to a9da7866dc, so this child needs a rebase, affected proof and a fresh exact-head review. Merge stays with the user after #449 lands.
Finding
1. Low, non-blocking: link-dense flat pages now fail extraction where the core returned them
Locations:
apps/server/src/webResearch/extract.ts:36setsMAX_LINK_CHARSequal to the 4 Mi-character text budget.extract.ts:101-103charges every HTTP(S) anchorhref + label + 32characters.- The throw becomes
PageExtractionErroratapps/server/src/webResearch/WebResearch.ts:379-382, thenprovider_errorat:411-412.
fetchPage always extracts, so this also applies with followSubpages:false.
Mechanism: an anchor costs about label + href + 4 characters of text ([label](href)), but label + href + 32 characters of metadata. On a page whose text is mostly anchors, the link budget runs out before the text budget. No nesting is needed.
Trigger and evidence: a disposable /tmp probe ran the real extract and classify from the base and from this head on the same HTML.
| Page | HTML bytes | Core (7a4b1f1) |
This head |
|---|---|---|---|
36,000 x <li><a href="/strutture/agriturismo-000123-toscana">Agriturismo 123 Toscana</a></li> |
3,084,958 | ok, 3,264,888 chars |
ok |
| 38,000 of the same | 3,256,958 | ok, 3,446,888 chars |
throws Research page links exceed the extraction limit. |
| 45,000 of the same | 3,858,958 | ok, 4,083,888 chars |
throws |
80,000 x <a href=/a>b</a> |
1,280,068 | ok, 2,240,000 chars |
throws |
All of these are under the 4 MiB HTTP body cap and the 4 Mi-character text cap.
Effect: a page with about 37k ordinary anchors or more returns provider_error with no text, where the core returned its full text. That is about 3.2 MB of HTML, or about 1.2 MB with very short links. Default reads and followSubpages:false reads are both affected. When the page is the root, its subpages are lost too. The review brief says followSubpages:false keeps the core's single-page behavior; for this page shape it does not.
Severity is low: only very large HTML sitemaps and index pages are this dense, and venue homepages are not. I did not run this end to end through MCP. The source path from the throw to provider_error is direct.
Possible repairs, if wanted:
- Give the metadata the remaining headroom. Worst-case text JSON is 24 MiB, so about 6.5 Mi characters of metadata still fit under 64 MiB. That covers ordinary anchors up to the 4 MiB body cap.
- Or keep the text and report truncated link metadata visibly, instead of failing the page.
Earlier findings, rechecked on this head
- Scoped refusals shadowing explicit reads: fixed. Unrestricted URL lookups skip restricted rows unless their final origin matches (
PageCache.ts:80-84).acceptsOriginrepeats the check (WebResearch.ts:145-148). The MCP regression covers both policy orders. - Late aliases refetching and duplicating output: fixed. Restricted hops reuse fresh compatible targets (
WebResearch.ts:320-339), keeping the target's realfetchedAt/expiresAt(:428-436). The traversal dedupes successful final URLs and still counts the attempt (:643-661). - Escaped labels over the old 32 MiB read limit: fixed. Writes reject JSON over 64 MiB before publication, and reads allow the same 64 MiB (
PageCache.ts:22,:102,:113-117). - Final URLs known only through aliases: fixed.
redirectTargetmatchesurlorfinal_urlunder the origin filter (PageCache.ts:75-79). Onlyokrows advertisefinal_url(:129). - Nested anchors exceeding the write ceiling: fixed.
- Text is at most 4 Mi characters, at no more than 6 JSON bytes each: 24 MiB.
- A link entry serializes to at most
21 + 2*href + 6*labelbytes, because URLs need only\\escapes. That is below6 * (href + label + 32), so all links together take at most 24 MiB. - The total therefore stays under the shared 64 MiB ceiling.
- The MCP regression keeps the root and both siblings, and the next read is served from cache.
Rebase onto 7a4b1f1
- Core queue code untouched. The child diff leaves alone the core's
requestqueue and 60 s acquisition deadline (WebResearch.ts:180-238), the zero-TTL robots refusal (:288-292) and theOriginQueueWaitErrorcatch (:414-420).if (cacheable) yield* cache.put(page)also covers the reused-target path (:457). - One core test edited, oracle unchanged. The 24-read contention case gains
followSubpages: falseand still proves the single-page contract.- The Retry-After and Crawl-delay cases pass unedited under default following, because those refusals are immediate.
/tmp/chromeria-441-r4-rebase-queue.logshows the default-follow version failing onExit.isSuccess, as the author described.
- Deliberate behavior change. With default following, a root that waits 30 to 60 s now returns
WebResearchBudgetErrorinstead of the core's uncachedchallenge. The tool description and PR body document this, and an MCP test covers it.followSubpages:falsekeeps the core's 60 s wait. - Timeouts cannot cache an interruption. The pinned
effect@4.0.4Cache.getdrops interrupted lookups (Cache.ts:641-647). A shared flight keeps running while another reader awaits it (:686-697).PageRequestkeys compare structurally, so restricted and unrestricted flights stay separate.
Cumulative acceptance
Met:
- Default pattern, at most 6 subpages, disable, and literal case-insensitive
|overrides without regex evaluation. - Normalized depth-one candidates, with cycles and duplicates skipped.
- External, private, credentialed and off-origin-redirect destinations refused, with public-network validation on every fetch.
- Shared cache, limiter and flight cancellation.
- Per-page provenance, with failures that keep successful siblings.
- A 30 s total deadline that includes the root.
- A shared 20,000-character output, explicit
truncationReasons, boundedpendingSubpages, andcacheId/nextOffsetcontinuation. - Cache-id and offset reads never traverse.
- Old three-column indexes reopen.
- Server/MCP scope only.
Docs:
- The tool description and
orchestrationInstructions.tsremain accurate. - The
docs/map.mdref follows the secondary-PR rule, since #454 does not close the primary #423. docs/fork-features.mdneeds no new PR entry, because the carried file set is unchanged.
Considered and not actionable:
pendingSubpagescan list an alias of a page already returned.- A forced refresh can fetch a redirect target twice in one traversal.
- An anchorless legacy row can race a concurrent
followSubpages:falseflight, but legacy rows exist only in unreleased core caches.
Proof checked
- Identity. The commit is dated 20:06:55 +0200.
/tmp/chromeria-441-r4-proof-identity(20:07:17) holds this head and treeb4758f3f5a17a701df31054eb860fb5fbf50bd47, matchinggit rev-parse. - Tests. 4 files and 83 tests passed in 18.35 s.
- Static checks. The typecheck, lint and knip logs were written after the commit. Each ends with this head and
exit_code=0. The typecheck log has only existing suggestions. - CI on this exact SHA passed: Typecheck, Lint, Test, Test Server 1-6, Test Web, Build, Check and Release Smoke. One duplicate Native fingerprint diff run was cancelled, and its rerun passed.
- Not rerun. I did not rerun the suite. The only execution was the disposable
/tmpextraction probe above.
Stack note
feat/423-web-read-core moved to a9da7866dc ("fix(research): cancel queue deadline on acquisition", 20:17 +0200), after this candidate's base. That commit changes request and the OriginQueueWaitError catch, and adds 135 lines of integration tests. This review does not cover it. The child needs a rebase, affected proof and a new exact-head review. This verdict applies only to d3cf0f5 on 7a4b1f1.
Reviewed by Claude Opus 5.5 (claude-opus-5-5) through the Claude Code harness in T3 Code, at xhigh effort. Read-only review: no checkout, branch change or repository edit.
|
What: The dense-page finding is reproduced at MCP and repaired by measuring serialized link metadata. Accepted finding: #454 (review), on The 80,000-link MCP regression failed on that head: The unchanged text bound requires at most 24 MiB under worst-case JSON escaping, so 32 MiB of encoded metadata plus fixed provenance stays below the existing 64 MiB cache ceiling. This replaces the estimate rather than guessing another raw-character threshold. Nested amplification remains an explicit per-page extraction failure; no anchors are silently dropped and no storage errors are hidden. Requirements and who asked: The user requested findings fixed; #441 must preserve usable single-page reads and bound metadata without losing healthy siblings. |
d3cf0f5 to
c3e6698
Compare
|
What: The rebased #441 candidate passes 89 focused tests and scoped static checks, with all six prior findings fixed. Head: All final checks ran after committing this exact head on a clean checkout. Every log records this head and
Fourth-review finding and repair: the 80,000-link MCP case failed on Rebased only the child onto #449's Registered MCP tests use the real reader/cache and external HTTP fixtures. Orchestration integrations drain Local evidence: |
a9da786 to
fe1bfe3
Compare
c3e6698 to
5175cca
Compare
|
What: The #441 layer now passes 89 focused tests and scoped static checks on #449's native-main-based head. Head:
The frozen offline dependency install completed in 12.3 s with no downloads or tracked changes. Installed Effect is 4.0.5. Root AGENTS.md and the unchanged direction triad were reloaded after rebase. All final checks ran on the clean exact head after dependency refresh. Each check log records the exact head and
The six findings and their failing-before/passing-after regressions are retained unchanged and all pass here. Registered MCP tests use the real reader/cache and external HTTP fixtures; integrations drain effects and verify unchanged thread state. TestClock/Deferred milestones, no sleeps or polling. Latest core queue/active-HTTP regressions retain their assertions and single-page intent. No test was removed or weakened. The previous review attempt was interrupted with no pending child runs after detecting the parent move, before a verdict. Its old-head pending status was marked superseded/error. A new review/status applies to this current candidate. Complete repository checks remain CI-owned. No merge, release, deployment, restart, browser or live-userdata changes. Local evidence: |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Round 6 independent review of PR #454 at head 5175cca6464da3fec1c56eb921b46dc9dec7026b (base fe1bfe38579aba2e81390e829ad66fdc7eda402b, feat/423-web-read-core) found no findings, blocking or otherwise.
Why: The sixth finding's fix now counts real serialized JSON bytes. No non-nested page that the core reads as ok can exceed the 32 MiB link budget. All five earlier fixes, and the rebase onto the native-main core, hold on Effect 4.0.5.
So what: review/independent is success on this SHA. #449 must land first, and merge stays with the user.
Reviewer: Claude Opus 5.5 (claude-opus-5-5) through Claude Code. xhigh effort was requested, but the session does not show me its effort setting, so I cannot confirm it. Read-only: no checkout, no edits, no test reruns.
Sixth finding (dense flat pages): fixed
extract.ts:99-104chargesBuffer.byteLength(JSON.stringify({url,label})) + 1per HTTP(S) anchor, starting from 2 for[]. This is the real array JSON plus at most one byte. The cache encodes withSchema.fromJsonString, which in pinnedeffect@4.0.5is a plainJSON.stringifywith no spacing (SchemaGetter.js:979), in the sameurl, labelfield order. The accounting therefore matches what is stored.- No flat page the core reads can fail this budget. Per non-nested anchor:
- Metadata is at most
22 + 2*href + 6*labelbytes. URLs need at most\\escapes, and labels arecompact(inner).slice(0,256). - That anchor's text
[X](href)is at least4 + href + max(label, 1)characters, or4 + 2*hrefwhen the label is empty. - So metadata is at most 6 times the anchor's own text. Flat anchors' text pieces are disjoint and total at most 4 Mi characters, so metadata stays at or below 24 MiB, under the 32 MiB budget.
- Only nested anchors (MathML, or
<a>inside table cells) count the same text more than once. That is finding 5's deliberate visibleprovider_errorcase.
- Metadata is at most
- The write ceiling still holds. Text JSON is at most 24 MiB, links at most 32 MiB, and the title (500 characters,
extract.ts:73) and URLs (4096 characters,WebResearch.ts:88) add a few KB. That stays under the shared 64 MiB write/read ceiling (PageCache.ts:22,102,113). Redirect-reuse aliases copy an already-bounded body. - Tests discriminate the fix. The 80,000
<a href="x">Rooms</a>case writes about 4.5 MB of links and about 3.0 M characters of text. The old raw estimate charged about 5.8 M characters and failed it. The case also checks text, continuation and a cached default reread with two requests. The 11,000-label control-character case stores about 35 MB of JSON and reads it back. The 270 x 256 nested-anchor case keeps the root and both siblings, then serves the next read from cache.
Earlier findings, rechecked on this head
- Scoped refusals no longer shadow explicit reads. URL lookups filter rows by origin policy (
PageCache.ts:75-87), andacceptsOriginrepeats the check (WebResearch.ts:145-148). Restricted rows always haverestriction_origin = origin(url), because children are filtered to the root's final origin. - Late aliases are reused and deduplicated. Restricted hops reuse fresh, compatible targets, keeping the target's real
fetchedAtandexpiresAt(WebResearch.ts:321-339,431-438). The loop deduplicates by successful final URL and still counts the attempt (:655-663). - The 64 MiB ceiling is shared by writes and reads, and an oversized write is rejected before the index row is published.
redirectTargetmatches the requested URL or the stored final URL. Onlyokrows advertisefinal_url(PageCache.ts:129).- Nested anchors produce a per-page
provider_error. The root and siblings are kept, and no unreadable row is persisted.
Rebase and runtime
git range-diff a9da7866dc..c3e66983de fe1bfe3857..5175cca646shows all 8 authored commits as=.git diff a9da7866dc fe1bfe3857leaveswebResearch/, the MCP toolkit and the integration test unchanged. The base changes only comments inMcpHttpServer.tsandpublicProxy.test.ts, and the web research layer wiring is identical.- Relied-on Effect sources are byte-identical between 4.0.4 and 4.0.5:
Cache.ts,Equal.ts,Hash.ts,Data.ts,SchemaGetter.ts,SchemaTransformation.tsandSemaphore.ts.- Interrupted lookups are still dropped (
Cache.ts:641-647), and a shared flight survives one reader leaving (:682-697). PageRequestkeys still compare structurally through defaultEqual/Hash, so restricted and unrestricted flights stay separate.- The
Effect.tsandinternal/effect.tschanges are transaction metadata and interrupt-cause bookkeeping. They do not touch the timeout and race paths used here.
- Interrupted lookups are still dropped (
URL.parseis available on the supported runtimes: the engine is^24.13.1and Electron is 44.4.5.
Proof inspected
/tmp/chromeria-441-r6-proof-identityholds this head and tree964569e42b, which matchesgit rev-parse HEAD^{tree}.- The test, typecheck, lint and knip logs each end with this head and
exit_code=0. The test log shows 4 files and 89 tests passed. The typecheck log has only pre-existing suggestions elsewhere. - The logs do not record their commands. The file and command list comes from the durable proof comment.
git diff --checkis clean, and the added tests contain no sleeps or timers.
Cumulative acceptance (#441)
These all hold in the code and in the MCP fixtures:
- depth one, at most 6 pages, the default pattern, disable and literal case-insensitive
|override with no regex - normalisation, deduplication and cycles
- refusal of external/private destinations and off-origin redirects
- the shared cache, limiter and in-flight cancellation
- per-page provenance
- partial page failures
- the 30 s total deadline including the root, with uncached root queue expiry
- the shared 20k output limit, with
truncationReasons,pendingSubpagesandunvisitedSubpages - no traversal for cacheId or nonzero-offset reads
Scope stays server/MCP only. The T3_CODE_ORCHESTRATION_INSTRUCTIONS text stays accurate, and the docs/map.md secondary ref follows the map rule.
Cleared, not findings
- When the root uses the whole 20k output budget, no child is fetched and every candidate is listed as pending. This is documented as explicit continuation.
- Origins with
Crawl-delayof 30 s or more returnWebResearchBudgetErroron the first default read. The retry, orfollowSubpages:false, succeeds, as the tool description says. - A legacy row whose final URL is off-origin can still win the SQL
LIMIT 1and then failacceptsOrigin. The only cost is a refetch.
Not verified: live network behavior and end-to-end MCP clients. Neither is required for this layer.
fe1bfe3 to
0072aec
Compare
5175cca to
3886489
Compare
|
What: PR #454 is retargeted to merged main and all refreshed local proof passes on the exact candidate.
Every command below ran with
Commands: node_modules/.bin/vp test run apps/server/src/mcp/toolkits/webResearch/handlers.test.ts apps/server/src/mcp/WebResearch.integration.test.ts apps/server/src/webResearch/PublicHttp.test.ts apps/server/src/htmlRender/publicProxy.test.ts
node_modules/.bin/tsc-rs --noEmit -p apps/server/tsconfig.json
node_modules/.bin/vp lint apps/server/src/mcp/WebResearch.integration.test.ts apps/server/src/mcp/toolkits/webResearch/handlers.test.ts apps/server/src/mcp/toolkits/webResearch/tools.ts apps/server/src/webResearch/PageCache.ts apps/server/src/webResearch/WebResearch.ts apps/server/src/webResearch/extract.ts apps/server/src/webResearch/schemas.ts
node_modules/.bin/knip --workspace apps/server --exports --preprocessor ./scripts/knip-schemas.ts --no-config-hints
bash scripts/fork-check.sh --upstream upstream
git diff --check c1d3960a1d1c0da46e714553ec32bf08b3136844...HEADDecisive test output: The registered MCP tests use the real reader/compressed cache with HTTP fixtures. Integrations drain The initial optional fork-check invocation incorrectly supplied fork main as its upstream baseline and consequently classified fork files as upstream edits. That invocation was discarded (retained at Limits: local verification is scoped under repository policy; complete repository checks remain CI-owned. No live network, real external MCP-client, browser/rendering, multi-environment or adoption proof is claimed for this server/MCP slice; those are separate #423 tickets. There is no frontend/browser behavior in this diff. No deployment, restart, machine/service/config activation or version bump. User approval covers the ordered merges, but the parent alone executes #454's merge after current gates pass. |
lukemaj
left a comment
lukemaj
left a comment
There was a problem hiding this comment.
What: Round 7 independent review of PR #454 passes with no blocking findings. The rebase is faithful, and I have one low, optional old-cache note.
Why: The research source, tests and dependencies are byte-identical to the round 6 candidate. The merged core and the incoming #457 work are intact, and current-head proof and CI are green.
So what: The #423 parent can carry #454 through its ordered merge once its own gates pass. The note below needs no change before merge.
Candidate
- PR #454, OPEN, target
main. - Head
388648949738ea8320bc4a780d96b39b55db94cb, tree9a6be0a97db74150f52e02a8697c75856ab3c775. - Base
c1d3960a1d1c0da46e714553ec32bf08b3136844(#449 squash), tree7964231a16f885ef6c5b59f9d8ccab6a4b22c3f9, parent45a30107ba. - I re-read the live PR and remote refs just before posting. Both match.
Rebase integrity (verified)
git range-diff fe1bfe3857..5175cca646 c1d3960a1d..3886489497shows=for all eight child commits. The only new commit is map-only commit 9.- Diff from base to head: the same 8 files and +1335/−96 as fe1→old-head. The research, MCP and htmlRender paths have zero diff from old head
5175cca646to head. - The old head and this head differ only in #457's observer source/tests, the
fork-features.mdregistry anddocs/map.md. None of these files is in base→head, so the incoming main work is byte-identical to base.pnpm-lock.yamlhas no diff; SHA-25682d9eede…3dfcmatches the proof logs. - The merged core is complete:
feat/423-web-read-coreat0072aecc89(local and origin, preserved) has the same tree as basec1d3960a1d. Research paths are identical between fe1→base. The #457 changes touch no research code. - Map net change:
WR: #423 PR #449→WR: #423 PR #454. This follows the secondary-ref rule, because #454 closes #441, not #423. proof/441-reviewed-5175cca646is retained.
Complete-candidate review
I read the full base→head diff and the surrounding owners: WebResearch.ts (limiter, request, fetchPage, flights, loadPage, read), PageCache.ts, extract.ts, schemas.ts, tools.ts, handlers.ts and PublicHttp bounds. I rechecked all six earlier fixes against the code and their MCP regressions:
- Policy-compatible cache selection.
- Alias/returned-page deduplication, with the alias still counted against the page budget.
- A shared 64 MiB write/read ceiling, rejected before blob/index publication.
- Successful-only
final_urladvertisement, with true body age preserved. - A separate link-metadata budget that keeps the root and siblings.
- Exact serialized UTF-8 link accounting. The bound holds: body ≤4 MiB, text ≤4 Mi chars (≤24 MiB escaped), links ≤32 MiB, title ≤500, so the total stays under 64 MiB.
Acceptance holds: default/override/disable, ≤6 same-origin depth-one pages, refused external/private destinations and redirects, a shared limiter/cache, per-page provenance, partial failures, a 30 s budget, a shared 20k output, explicit truncation and continuation, and no traversal on cacheId or nonzero-offset reads. Structural PageRequest keys are proved by the concurrent-shared-fetch test (3 observed requests).
Findings
Low, non-blocking, optional. An alias redirect can turn an anchorless pre-upgrade cache body into an entry that claims to have no links.
- Where:
apps/server/src/webResearch/WebResearch.ts:321-338, withPageCache.ts:76-79. - Trigger:
- A restricted child A redirects within the origin to T.
- T's only fresh entry was written by the core-only build, so it has no
links. The legacy index row matchesfinal_origin IS NULL, andacceptsOriginpasses via T's same-originfinalUrl. - The reuse returns
links: stored.links ?? [], and A is published withlinks: []. - A later explicit
web_read({url: A})accepts that entry, because therequireLinkscheck atWebResearch.ts:505treats the defined[]as having links. It returns the cached page and follows no subpages. It reportstruncated:falseandunvisitedSubpages:0.
- Contrast: a direct read of T correctly refetches and upgrades.
- Impact: at most 7 days of missing automatic traversal for explicit reads of such an alias. No text is lost, because the anchors stay as markdown links in
text, andmaxAgeHours:0bypasses it. - Why it doesn't block: it is reachable only when a core-only build wrote cache entries before this layer merges.
- Evidence: traced in the source; I did not run it.
- Optional fix: add
stored.links !== undefinedto the redirect-reuse condition.
Nothing else blocking or actionable. I cleared these concerns:
- A failed child's
finalUrlsuppresses a later identical candidate, but the failure stays visible. - Legacy rows can shadow older compatible rows, which only causes a refetch.
- Flights shared by intent cannot hand a links-less page to a follow reader, because only legacy entries lack
linksand the flight recheck is stricter. WebResearchBudgetErrormaps toorchestration_errorwith retry guidance.
Verification
- I inspected the proof logs myself, not only their names. SHA-256 digests match comment 6254094064 for tests, typecheck, lint, knip, fork and diff.
- Each log records the exact base/head/tree, the command, the lockfile digest, Warsaw times,
worktree_unchanged=trueandexit_code=0. - Results: 89/89 tests in 4 files; typecheck 0 errors (245 suggestions, none new from this diff); lint on 7 files; knip exports.
- Fork check reports
fork-features: OK (46 features)andfork-check: OKagainst upstream merge-based0918f1659. The discarded invalid-base fork log is acknowledged and was not used. - The tests use no sleeps or polling and TestClock/Deferred only. The core queue/active-HTTP regressions keep their assertions; the only change to an existing core test adds the legacy-index migration path.
- GitHub CI on this exact head:
Check,Lint,Build and Typecheck,Test,Test Server 1-8andFork Stack Modelall succeeded. The parent still owns final CI acceptance. - Limits: I reran nothing and ran no probe. There is no live network, external MCP client, browser or multi-environment proof, and this slice requires none. I wrote nothing beyond this review and the
review/independentstatus.
Reviewer: Claude Opus 5.5 (claude-opus-5-5) through the Claude Code harness in T3 Code, as a read-only delegated task with no further delegation. xhigh effort was requested; I cannot observe the runtime effort setting from inside the session.
|
What: PR #454 merged into main as Merge input: head Evidence: 89-test scoped proof, CI39158544642, Fork39158544596, Opus R7 PASS, and transfer report. Latest The reviewer’s optional legacy-cache alias note is nonblocking and is retained under #447. No core-only build was deployed in this task. The #441 owner retains cleanup ownership of its workspace and will preserve only resources still needed by #442/#443; root does not remove another active owner's workspace. Core/rebase proof refs remain retained while descendants establish their final predecessor boundaries. This is an active-stack retention exception, not completed cleanup. #443 stays open for pre-activation cache lifecycle work. No deployment, staging, restart, machine configuration, public release or version bump is part of this landing. |
|
What: #455 does not need #441's historical branch or worktree; its own exact history is retained independently. Verified |
|
What: #443 releases its dependency on #441 and #442's historical branches/workspaces, including their old core references. Final integration identity/authority. No other owner's branch/workspace was modified by this task. This confirmation releases only #443's dependencies and is not cleanup authority over any other owner's work. |
Follow matching same-origin depth-one subpages through the shared research service, cache and limiter, with per-page provenance, bounded output and partial failures. Closes #441. Refs #423. Proof: 89 focused MCP/runtime tests with drained effects and no sleeps; scoped typecheck/lint/exports/fork checks; CI39158544642 and Fork39158544596 passed on388648949738ea8320bc4a780d96b39b55db94cb. Independent review: Claude Opus5.5 R7, review5564609099, PASS on exacthead. User-approved ordered landing coordinated on411. No deployment/restart or version bump. Authored with Codex through T3 Code; reviewed with Claude Opus5.5 through Claude Code in T3 Code.
What:
web_readnow includes matching same-site depth-one subpages with separate provenance and bounded continuation.Why: Room and event facts often live beyond the homepage, so homepage-only reads miss research evidence.
So what: The parent merged this as
def07e920a27c4d266b47cf15ecde4be0db70c27and #441 is closed; dependent owners released their refs, the remote/proof branches are retired, and only this thread's active local checkout remains retained.Closes #441. Part of the approved #423 detour and its review/user decisions.
Merged candidate
main, exact basec1d3960a1d1c0da46e714553ec32bf08b3136844, tree7964231a16f885ef6c5b59f9d8ccab6a4b22c3f9.388648949738ea8320bc4a780d96b39b55db94cb, tree9a6be0a97db74150f52e02a8697c75856ab3c775.fe1bfe38579aba2e81390e829ad66fdc7eda402band preserved all eight child commits from prior reviewed head5175cca6464da3fec1c56eb921b46dc9dec7026b; range-diff marks every replay unchanged. A ninth map-only commit removes the merged feat(research): read public pages with shared cache and provenance #449 reference and retains this open PR. The old child remains at localproof/441-reviewed-5175cca646, and the core branch is preserved.feat/441-web-read-subpagesand its exclusive workspace were edited.def07e920a27c4d266b47cf15ecde4be0db70c27, parentc1d3960a1d1c0da46e714553ec32bf08b3136844, with tree exactly equal to this reviewed head. GitHub reports MERGED and feat(research): follow same-site subpages in web_read #441 CLOSED. The parent continues feat(research): render thin pages with the isolated browser #453 then feat(research): share page reads through a linked environment #455; this task performed no merge, deployment or restart.Behavior
Initial URL reads follow at most six matching depth-one anchors. Same-site is conservatively limited to the final root's origin, including HTTP and meta-refresh redirects. URL parsing canonicalizes fragments, host case, default ports and dot segments. Root cycles, duplicate URLs and aliases onto returned pages are skipped; children are not traversed. Redirects reuse compatible fresh targets, including targets known only by an earlier alias, while preserving actual body age. Alias attempts count toward the page limit.
followSubpages:falsedisables following.subpagePatternoverridescamere|alloggi|appartament|ospitalit|rooms|soggiorn|eventi|prezzi|tariffewith case-insensitive literal alternatives separated by|, matched against path/query or anchor label.maxSubpagesaccepts 1-6. No arbitrary regex evaluation.The existing service/cache/limiter own every fetch. Each child has its own URL/final URL/time/rung/status/hash/cache id and text continuation; failures preserve successful siblings. Cache lookup chooses the newest origin-policy-compatible entry. The disposable index gains nullable origin/final-URL/publication metadata, keeping old cache ids and compressed bodies readable. Only successful bodies advertise reusable final URLs, so a foreign refused redirect cannot substitute for a target. Publication order is independent of body age.
Text retains the core's 4 Mi-character extraction limit. Link metadata has a separate 32 MiB budget measured in actual serialized UTF-8 JSON bytes, preserving dense flat pages while bounding nested-label amplification. Exceeding extraction limits produces a visible per-page failure and preserves siblings. Cache writes and bounded decompression share a 64 MiB serialized-body ceiling; oversized writes fail before publication. No state database migration.
Traversal has a 30-second total deadline including root origin queue time, and shares
maxChars(default/maximum 20,000) across returned text. Root expiry produces an explicit uncached deadline error with retry guidance. With following disabled, core single-page queue/active-HTTP budgets remain intact. Partial results exposetruncated,truncationReasons, up to sixpendingSubpagesandunvisitedSubpages. Continue each page throughcacheId/nextOffset, or read pending URLs with following disabled. Full text and anchors stay cached. Cache-id and nonzero-offset reads do not traverse.Proof and review
5175cca6464da3fec1c56eb921b46dc9dec7026b, not this new main-based candidate. All six prior review findings were accepted and fixed with failing-before/passing-after MCP regressions: policy cache shadowing, late alias duplication, escaped metadata cache read/write mismatch, final targets known only through aliases, nested-label amplification, and rejection of dense flat pages by an inaccurate metadata estimate./tmp/chromeria-441-r7-*.loginclude commands, exact base/head/tree, frozen-lockfile digest, Warsaw times and exit codes. The current proof comment records commands, results and log hashes.OrchestrationEffectWorkerV2before/after, preserves thread state and measures bounded wire output/continuation. TestClock/Deferred only; no sleeps or polling. Existing core queue/active-HTTP/cancellation tests and all six review regressions remain unchanged.review/independentissuccess, creatorlukemaj, linked to that review. One optional low legacy-cache alias note was explicitly accepted for #447 by the parent before merge; it is not claimed fixed. The new T3 task explicitly requested xhigh; the reviewer could not independently observe its runtime effort setting. Current CI passed: lint, test, all eight server shards, build/typecheck and Check succeeded; unaffected web/mobile/Rust/release-smoke jobs were skipped by project CI. Fork passed on the same head after the explicitly authorized same-SHA close/reopen triggered both workflows. No earlier verdict or CI result is transferred to the new SHA.388648949738ea8320bc4a780d96b39b55db94cband main baselinec1d3960a1d1c0da46e714553ec32bf08b3136844: zero change in all measured Codex/Claude wire, decoded-live and message-count metrics, with every enforced ceiling passing.Scope and limits
Server/MCP behavior only. Web, desktop and mobile retain generic MCP presentation. Provider adapters, WebSocket contracts, command-palette/keybinding entry points and connection routing do not change; shared-host forwarding is #443. No custom UI, browser/computer use, rendering, session access, search, live network/external-client or multi-environment/adoption claim. No browser proof is applicable to this slice. Website impact: none, because no website content or routes change. No version bump or machine/service/config activation.
Finalization
The implementation, exact-head proof, independent review, parent-owned merge and Issue closure are complete. Issue #441 owns the final handoff and cleanup proof. No version bump or activation was performed.
Verified disposable task scripts, redundant published bodies, rebase patch copies and historical review probes were retired. Raw proof logs and identities remain available. Cleanup does not touch the core/sibling branches, other owners' workspaces, shared installs or live state.
Both dependent owners released their #441 needs: #442 explicitly after #453 merged as
f27d8ee146295ed2b76d8d15d415dd7cff574df5, and #443 in its final integration handoff. No open PR targets this branch. Remotefeat/441-web-read-subpagesand localproof/441-reviewed-5175cca646were deleted with exact-SHA guards and verified absent. The local branch remains at388648949738ea8320bc4a780d96b39b55db94cb, with its upstream unset, solely because this active T3 thread still binds/home/luke/.t3/worktrees/chromeria/feat-441-web-read-subpages. Retire that local branch/checkout after the active binding is retired. Raw proof and GitHub delivery records remain preserved; no core/sibling branch, other workspace, shared install or runtime was changed. The cleanup update and exact command/after-state proof are on #441.Elon record
WebResearch.readremains the owner. The final gate is exact-head review and CI after the squash-parent rebase.Implemented with GPT-6.1-Sol (max) through the Codex harness in Chromeria; independent review uses Claude Opus 5.5 through Claude Code, with xhigh requested and verdict/status bound to the exact head.