fix(discovery): one exhaustive surface is not an artifact inventory - #2116
Conversation
Discharges the unresolved review thread stranded on merged PR #1260 ("Do not treat a sitemap as proof that a rung is absent"). The reviewer is right, and the proof is internal to the file. The artifact ladder's absence rule earned `probed-and-not-existing` "against a surface that enumerates the publisher's own artifacts completely: a `sitemap.xml` (or its index), the in-repo docs tree, a releases or asset listing" -- a disjunction of three surfaces -- and outcome-gate criterion 9 restated it as "such as a sitemap or the in-repo docs tree", explicitly single-surface. "Completely" was the intended guard and was never made operative. So a publisher that omits PDFs from its sitemap, parks model cards on an asset host, or keeps them off the docs tree let a run record rung 1 as absent, descend to the announcement post, and pass criterion 9 with the system card unread. The contradiction needed no external evidence. The ladder's own preamble scopes the surfaces apart -- "(The doc-index probe below enumerates *pages*; this ranks *artifact classes*.)" -- and the doc-index table stamped `sitemap.xml` "Exhaustive -- every URL ... enumerate ALL pages". Pages. Rung 1 for a model/benchmark claim is "the system or model card, often a PDF". `probed-and-not-existing` is now earned exactly two ways: the surfaces checked TOGETHER cover every first-party surface an artifact of that claim class plausibly lives on, or the publisher declares its own inventory complete for the class. "Plausibly, for that class" is a stated bound, so the outcome stays reachable -- a class a publisher only ships in-repo is still settled by the tree alone. Short of either, the rung is unresolved: a Gap naming surfaces checked and unchecked. An adversarial verifier then landed two real hits on that fix, both addressed here. The Output Format fetch log had no `unresolved` slot and called does-not-exist "the bypass outcome a probe alone can establish", so the reviewer's path completed anyway through the nearest legal value; the vocabulary is five-valued now and the probe-settles-nonexistence phrasing is struck. And the eval clause added to case 8 could not fire against its own library-behavior fixture, so it is generalized and case 12 supplies a real model-card fixture. Proven by A/B control on the two texts rather than by grep: fresh agents given only the pre-fix rules answered probed-and-not-existing 2 of 3; given only the post-fix rules, unresolved 3 of 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Claude finished @kyle-sexton's task in 46s —— View job Security review complete
No security issues found. This PR touches six files, all of which are non-executable content:
There are no GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets or token handling, and no permission or trust-boundary changes anywhere in this diff. The entire change is confined to the research skill's documentation of when an "artifact absence" may be concluded — a correctness/behavioral fix to agent reasoning guidance, not a security-relevant surface. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61a2355244
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude finished @kyle-sexton's task in 3m 37s —— View job Review complete
SummaryThis PR closes a review thread stranded on merged PR #1260 by tightening the What holds up:
One inline finding posted (non-blocking): the sentence appended to case 8's expectations ( I did not independently re-run Branch: |
|
Claude has reviewed this PR 4 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Version collision only. Main shipped discovery 0.13.0 (the bare-alias removal) while this branch held 0.12.3, so this entry takes 0.13.1 and both changelog entries are kept in full — they cover unrelated changes. Verified: no conflict markers remain, plugin.json parses, changelog versions are monotonic with no duplicates, no MD012. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 25s —— View job Review complete
SummaryRe-reviewed at HEAD Cross-file consistency holds up. I traced the five-surface list (docs
This is a real fix for the reviewer's original complaint on #1260: the single-surface reading is gone everywhere it previously appeared, including the load-bearing outcome-gate criterion, not just the discipline-file prose. One new finding, inline on I did not independently re-run Branch: |
|
Claude finished @kyle-sexton's task in 41s —— View job Security review complete
No security issues found. All six changed files are non-executable content, and none introduce a new trust boundary, data flow, or execution surface:
No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission, config, or trust-boundary changes anywhere in this diff. Consistent with the two prior automated security-review passes on this PR (job runs 31340120546) which reached the same conclusion on earlier commits — this pass re-verified against the current HEAD ( Branch: |
…cuous clause Both are the failure this PR itself named: an added test that cannot fail is not an oracle. Case 12 asked the run to sweep a fictional vendor's first-party surfaces with `files` empty and nothing on the web to fetch, so the only correct outcome was "premise unverifiable" — never the one-clean-surface-is-not-absence behaviour the case exists to grade. It now embeds the surfaces in the prompt the way case 11 embeds its dispatch payload: a docs sitemap that parses cleanly and holds no model card, a launch post linking an un-enumerated sibling research domain, an asset host that 403s on listing, and no vendor completeness declaration. That is a state where recording rung 1 as probed-and-not-existing is wrong and unresolved-plus-Gap is right, so the expectations can now discriminate. Case 8's appended clause could not fire there at all: its claims are library-behaviour comparisons whose rung 1 is "the source itself" per discipline.md's rung list, not a model card that might sit on a sitemap, an asset host, or a sibling domain. No run against that fixture would consider probed-and-not-existing off a single sitemap scan. Dropped rather than reworded — case 12 is where the clause is exercisable, and it is exercisable there now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…o fix/discovery-stranded-1260
|
Claude finished @kyle-sexton's task in 38s —— View job Security review complete
No security issues found. Reviewed the full diff (
No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission or config changes anywhere in this diff. Consistent with the three prior automated security-review passes on earlier commits of this PR, which reached the same conclusion — this pass re-verified against current HEAD and the finding stands. (Note: this task's instructions asked for a security review specifically — the eval-fixture realism concerns raised by Codex and prior Claude review comments on Branch: |
|
Claude finished @kyle-sexton's task in 1m 55s —— View job Review complete
SummaryRe-reviewed at HEAD Case 12's fixture is now real ( Case 8's clause is dropped, not reworded ( Cross-file consistency still holds across the other four files ( No new findings. Both previously open items are resolved; I found nothing else to flag in this pass. I did not independently re-run Branch: |
Grading the fixture against a deliberately-wrong answer and a correct one exposed the same defect this case exists to punish, now inside the case itself. The wrong answer failed 5 of 8 rows, but the correct answer could not pass: two rows graded UNGRADEABLE against BOTH answers, so no run of any quality could clear the case. Row 3 was gated on "before any absence verdict". A correct run issues no absence verdict at all, so the gate never opened. It now grades the accounting every run performs: research.vantor.ai and assets.vantor.ai must each appear somewhere as swept or as named-unchecked. It also closes the escape the row allowed, since prompt clause (d) states no vendor completeness declaration exists. Row 6's antecedent was "If the model card IS found" — unreachable here, because the fixture has no card to find. Replaced with the branch that is reachable: a title, snippet, or sitemap index entry LOCATES a rung and never reads it, so no condition and no fetched-and-lacking-the-claim outcome may be sourced from the sitemap's 47 URLs. The found-card protocol belongs in a case where a card is actually found, not as a clause that cannot fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 3m 18s —— View job Review complete
SummaryRe-reviewed at HEAD This commit's fix is sound. Per the commit message, grading case 12 against a deliberately-wrong and a deliberately-correct answer found two of the eight expectation rows
I traced both new rows against the case 12 prompt ( No new findings. Verification note: I could not execute Branch: |
|
Claude finished @kyle-sexton's task in 1m 12s —— View job Security review complete
No security issues found. Reviewed the full diff (
No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission, config, or trust-boundary changes anywhere in this diff. Consistent with all five prior automated security-review passes on earlier commits of this PR ( Branch: |
No linked issue
Summary
Discharges the one unresolved review thread left stranded on merged PR #1260 (
PRRT_kwDOTCGFQM6TzDQs,plugins/discovery/skills/research/context/discipline.md): "Do not treat a sitemap as proof that arung is absent." The reviewer is right, and the proof is internal to the file.
The artifact ladder's absence rule earned
probed-and-not-existing"against a surface thatenumerates the publisher's own artifacts completely: a
sitemap.xml(or its index), the in-repo docstree, a releases or asset listing" — a disjunction of three surfaces — and outcome-gate criterion 9
restated it as "such as a sitemap or the in-repo docs tree", explicitly single-surface. "Completely"
was the intended guard and was never made operative. So a publisher that omits PDFs from its
sitemap, parks model cards on an asset host, or keeps them off the docs tree let a run record rung 1
as absent, descend to the announcement post, and pass criterion 9 with the system card unread.
The contradiction needed no external evidence. The ladder's own preamble scopes the two surfaces
apart — "(The doc-index probe below enumerates pages; this ranks artifact classes.)" — and the
doc-index table stamped
sitemap.xml"Exhaustive — every URL … enumerate ALL pages". Pages.Rung 1 for a model/benchmark claim is "the system or model card, often a PDF". Two paragraphs
apart, the file licensed an absence claim about PDFs from a surface it itself scoped to pages.
Fix
probed-and-not-existingis now earned exactly two ways and no others:class plausibly lives on — docs
sitemap.xmlor its index, in-repo docs tree, releases or assetlisting, download/asset host, a sibling first-party domain the publisher itself links to; or
Short of either, the rung is unresolved — a Gap naming surfaces checked and unchecked.
Plausibly, for that class is a stated bound rather than decoration: the surfaces the class actually
uses, never every surface imaginable, so a claim class a publisher only ever ships in-repo is still
settled by the tree alone and the outcome stays reachable. Making the bar unsatisfiable would be the
opposite failure, and was tested for explicitly (below).
Six sites, every one of which carried the single-surface reading or the vocabulary that let a run
fall back into it:
skills/research/context/discipline.md— the absence rule itself.skills/research/SKILL.md— outcome-gate criterion 9. The load-bearing one; a fix touching onlythe discipline file would leave the complaint reproducing verbatim through the gate text.
skills/research/context/discipline.md— the doc-index table'ssitemap.xmlrow, rescoped to"Exhaustive for that host's listed pages … NOT an artifact inventory". This is the cell that
licensed the bad inference in the first place.
skills/research/SKILL.md— the Output Format fetch-log vocabulary (see the verifier section).skills/research/context/gotchas.md— the failure mode its curated-vs-exhaustive bullet washiding: a curated index is not the only non-proof of absence.
skills/research/evals/evals.json— the criterion-9 grading clause, plus a new case 12 with afixture the clause can actually fire against.
Verification
Pre-fix control (the point of the exercise — a check that passes on both trees proves nothing).
A grep control would only prove the edit landed, so the control is a behavioral A/B holding
everything but the rule text constant. Two structurally identical bundles — criterion 9 + the
absence rule + the rung list + the doc-index table — pre-fix in one, post-fix in the other.
Identical isolated question to both: the docs
sitemap.xmlparsed cleanly and lists no model cardor PDF, nothing else was checked, may rung 1 be recorded
probed-and-not-existing?, with the restof the ladder stipulated as not at issue.
probed-and-not-existingunresolvedThe pre-fix majority reasoned exactly as the reviewer predicted: "the rule names … a
sitemap.xml(or its index), the in-repo docs tree, a releases or asset listing … singular and disjunctive … the
unchecked asset/releases host is a substitutable alternative, not an outstanding obligation." Prose,
so this is a model-judgment control rather than a deterministic one, reported with its sample size.
An earlier control run was discarded as contaminated and is not counted above: it bundled the
rung-1 absence question with a separate question about sourcing from the announcement, which drags
in rungs 2-4 and made every answer a composite. One of those samples nonetheless said the quiet part
out loud — "Rung 1 may well be recordable as
probed-and-not-existinghere" — before answering NOon the composite.
Unboundedness probe. A separate fresh agent was asked whether the new bar is satisfiable at all
or has become an impossible standard:
VERDICT: SATISFIABLE, on the grounds that the plausible-surfaceset is a closed enumeration and "'Plausibly, for that class' is the bound and it is a real one"
bounds the recursion, with the publisher-declaration path as a second exit.
Adversarial verifier — landed two real hits, both fixed here. A fresh-context agent got the
finding text and the diff, no rationale, and was told to refute the claim that the finding was
discharged. Verdict on the first round:
PARTIALLY DISCHARGED.does-not-exist, fetched-and-lacking, unreachable-after-escalation — with no slot for
unresolved, while describing does-not-exist as "the bypass outcome a probe alone can establish"and asserting "nonexistence is what a probe settles". Both phrases contradicted the tightened
criterion 9 fourteen lines above them, and the reviewer's exact path completed straight through
the gap: sweep one surface, land on
unresolvedper criterion 9, find no legal slot for it,write the nearest legal value, pass the gate. The verifier's own calibration: pre-fix
unresolvedwas a corner case, post-fix it is the default whenever the sweep is incomplete, so the
unrepresentable state went from latent to operative. The vocabulary is five-valued now,
unresolvedmarked as the default and explicitly not a licence to descend; theprobe-settles-nonexistence phrasing is struck; criterion 9's "exactly one of three outcomes" is
corrected to four.
claims, whose rung 1 is "the source itself", so the off-sitemap-PDF scenario it grades could not
arise from that fixture and the clause passed vacuously. "An added test that cannot fail is not an
oracle." The clause is generalized to any claim class, and case 12
absence-of-a-rung-needs-more-than-one-clean-surfacesupplies a fixture that presents the realscenario.
context/gotchas.mdstill framed the whole absence trap as curated-vs-exhaustive,leaving the implicature that a miss on the exhaustive surface is not silence. Bullet added.
reviewer's scenarios (omitted PDFs, another first-party domain, model cards outside the docs tree)
are covered on four surfaces each.
Gates.
scripts/affected-tests.sh --runover the diff (10 suites selected, all mapped — nounmapped-file error);
scripts/check-changelog-parity.sh --check-bump origin/main→ OK, no versioncollision (
discovery0.12.2 → 0.12.3, main still carries 0.12.2);check-skill.sh research→PASS, 0 errors;
check-evals-quality.sh→ PASS, 0 warnings;markdownlint-cli2on all four changedMarkdown files → 0 errors.
Branch merged
origin/main(not rebased) before push.Related
absence vocabulary, and criterion 9.
PRRT_kwDOTCGFQM6TzDQson feat(discovery): artifact ladder, large-doc fetch, absence enumeration #1260, still unresolved at time of writing.discovery0.12.2 → 0.12.3 with its per-plugin changelog entry (changelog parity).