Skip to content

fix(discovery): one exhaustive surface is not an artifact inventory - #2116

Merged
kyle-sexton merged 6 commits into
mainfrom
fix/discovery-stranded-1260
Aug 10, 2026
Merged

fix(discovery): one exhaustive surface is not an artifact inventory#2116
kyle-sexton merged 6 commits into
mainfrom
fix/discovery-stranded-1260

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

No linked issue

Summary

Discharges the one unresolved review thread left stranded on merged PR #1260 (PRRT_kwDOTCGFQM6TzDQs,
plugins/discovery/skills/research/context/discipline.md): "Do not treat a sitemap as proof that a
rung is absent."
The reviewer is right, and the proof is internal to the file.

The artifact ladder's absence rule earned probed-and-not-existing "against a surface that
enumerates the publisher's own artifacts completely: a sitemap.xml (or its index), the in-repo docs
tree, a releases or asset listing" — a disjunction of three surfaces — and outcome-gate criterion 9
restated it as "such as a sitemap or the in-repo docs tree", explicitly single-surface. "Completely"
was the intended guard and was never made operative. So a publisher that omits PDFs from its
sitemap, parks model cards on an asset host, or keeps them off the docs tree let a run record rung 1
as absent, descend to the announcement post, and pass criterion 9 with the system card unread.

The contradiction needed no external evidence. The ladder's own preamble scopes the two surfaces
apart — "(The doc-index probe below enumerates pages; this ranks artifact classes.)" — and the
doc-index table stamped sitemap.xml "Exhaustive — every URL … enumerate ALL pages". Pages.
Rung 1 for a model/benchmark claim is "the system or model card, often a PDF". Two paragraphs
apart, the file licensed an absence claim about PDFs from a surface it itself scoped to pages.

Fix

probed-and-not-existing is now earned exactly two ways and no others:

  1. the surfaces checked together cover every first-party surface an artifact of that claim
    class
    plausibly lives on — docs sitemap.xml or its index, in-repo docs tree, releases or asset
    listing, download/asset host, a sibling first-party domain the publisher itself links to; or
  2. the publisher itself declares its chosen inventory complete for that class.

Short of either, the rung is unresolved — a Gap naming surfaces checked and unchecked.

Plausibly, for that class is a stated bound rather than decoration: the surfaces the class actually
uses, never every surface imaginable, so a claim class a publisher only ever ships in-repo is still
settled by the tree alone and the outcome stays reachable. Making the bar unsatisfiable would be the
opposite failure, and was tested for explicitly (below).

Six sites, every one of which carried the single-surface reading or the vocabulary that let a run
fall back into it:

  • skills/research/context/discipline.md — the absence rule itself.
  • skills/research/SKILL.md — outcome-gate criterion 9. The load-bearing one; a fix touching only
    the discipline file would leave the complaint reproducing verbatim through the gate text.
  • skills/research/context/discipline.md — the doc-index table's sitemap.xml row, rescoped to
    "Exhaustive for that host's listed pages … NOT an artifact inventory". This is the cell that
    licensed the bad inference in the first place.
  • skills/research/SKILL.md — the Output Format fetch-log vocabulary (see the verifier section).
  • skills/research/context/gotchas.md — the failure mode its curated-vs-exhaustive bullet was
    hiding: a curated index is not the only non-proof of absence.
  • skills/research/evals/evals.json — the criterion-9 grading clause, plus a new case 12 with a
    fixture the clause can actually fire against.

Verification

Pre-fix control (the point of the exercise — a check that passes on both trees proves nothing).
A grep control would only prove the edit landed, so the control is a behavioral A/B holding
everything but the rule text constant. Two structurally identical bundles — criterion 9 + the
absence rule + the rung list + the doc-index table — pre-fix in one, post-fix in the other.
Identical isolated question to both: the docs sitemap.xml parsed cleanly and lists no model card
or PDF, nothing else was checked, may rung 1 be recorded probed-and-not-existing?
, with the rest
of the ladder stipulated as not at issue.

Bundle probed-and-not-existing unresolved
PRE-fix 2 of 3 1 of 3
POST-fix 0 of 3 3 of 3

The pre-fix majority reasoned exactly as the reviewer predicted: "the rule names … a sitemap.xml
(or its index), the in-repo docs tree, a releases or asset listing … singular and disjunctive … the
unchecked asset/releases host is a substitutable alternative, not an outstanding obligation."
Prose,
so this is a model-judgment control rather than a deterministic one, reported with its sample size.

An earlier control run was discarded as contaminated and is not counted above: it bundled the
rung-1 absence question with a separate question about sourcing from the announcement, which drags
in rungs 2-4 and made every answer a composite. One of those samples nonetheless said the quiet part
out loud — "Rung 1 may well be recordable as probed-and-not-existing here" — before answering NO
on the composite.

Unboundedness probe. A separate fresh agent was asked whether the new bar is satisfiable at all
or has become an impossible standard: VERDICT: SATISFIABLE, on the grounds that the plausible-surface
set is a closed enumeration and "'Plausibly, for that class' is the bound and it is a real one"
bounds the recursion, with the publisher-declaration path as a second exit.

Adversarial verifier — landed two real hits, both fixed here. A fresh-context agent got the
finding text and the diff, no rationale, and was told to refute the claim that the finding was
discharged. Verdict on the first round: PARTIALLY DISCHARGED.

  • Hit 1. The Output Format section's fetch-log vocabulary was four-valued — carries-the-claim,
    does-not-exist, fetched-and-lacking, unreachable-after-escalation — with no slot for
    unresolved
    , while describing does-not-exist as "the bypass outcome a probe alone can establish"
    and asserting "nonexistence is what a probe settles". Both phrases contradicted the tightened
    criterion 9 fourteen lines above them, and the reviewer's exact path completed straight through
    the gap: sweep one surface, land on unresolved per criterion 9, find no legal slot for it,
    write the nearest legal value, pass the gate. The verifier's own calibration: pre-fix unresolved
    was a corner case, post-fix it is the default whenever the sweep is incomplete, so the
    unrepresentable state went from latent to operative. The vocabulary is five-valued now,
    unresolved marked as the default and explicitly not a licence to descend; the
    probe-settles-nonexistence phrasing is struck; criterion 9's "exactly one of three outcomes" is
    corrected to four.
  • Hit 2. The eval clause was attached to case 8, a RabbitMQ/Kafka/NATS comparison — library-behavior
    claims, whose rung 1 is "the source itself", so the off-sitemap-PDF scenario it grades could not
    arise from that fixture and the clause passed vacuously. "An added test that cannot fail is not an
    oracle." The clause is generalized to any claim class, and case 12
    absence-of-a-rung-needs-more-than-one-clean-surface supplies a fixture that presents the real
    scenario.
  • Hit 3 (secondary). context/gotchas.md still framed the whole absence trap as curated-vs-exhaustive,
    leaving the implicature that a miss on the exhaustive surface is not silence. Bullet added.
  • The verifier explicitly failed to land the unboundedness attack, and confirmed all three of the
    reviewer's scenarios (omitted PDFs, another first-party domain, model cards outside the docs tree)
    are covered on four surfaces each.

Gates. scripts/affected-tests.sh --run over the diff (10 suites selected, all mapped — no
unmapped-file error); scripts/check-changelog-parity.sh --check-bump origin/main → OK, no version
collision (discovery 0.12.2 → 0.12.3, main still carries 0.12.2); check-skill.sh research
PASS, 0 errors; check-evals-quality.sh → PASS, 0 warnings; markdownlint-cli2 on all four changed
Markdown files → 0 errors.

Branch merged origin/main (not rebased) before push.

Related

kyle-sexton and others added 2 commits August 9, 2026 18:36
Discharges the unresolved review thread stranded on merged PR #1260
("Do not treat a sitemap as proof that a rung is absent"). The reviewer
is right, and the proof is internal to the file.

The artifact ladder's absence rule earned `probed-and-not-existing`
"against a surface that enumerates the publisher's own artifacts
completely: a `sitemap.xml` (or its index), the in-repo docs tree, a
releases or asset listing" -- a disjunction of three surfaces -- and
outcome-gate criterion 9 restated it as "such as a sitemap or the
in-repo docs tree", explicitly single-surface. "Completely" was the
intended guard and was never made operative. So a publisher that omits
PDFs from its sitemap, parks model cards on an asset host, or keeps
them off the docs tree let a run record rung 1 as absent, descend to
the announcement post, and pass criterion 9 with the system card unread.

The contradiction needed no external evidence. The ladder's own
preamble scopes the surfaces apart -- "(The doc-index probe below
enumerates *pages*; this ranks *artifact classes*.)" -- and the
doc-index table stamped `sitemap.xml` "Exhaustive -- every URL ...
enumerate ALL pages". Pages. Rung 1 for a model/benchmark claim is
"the system or model card, often a PDF".

`probed-and-not-existing` is now earned exactly two ways: the surfaces
checked TOGETHER cover every first-party surface an artifact of that
claim class plausibly lives on, or the publisher declares its own
inventory complete for the class. "Plausibly, for that class" is a
stated bound, so the outcome stays reachable -- a class a publisher
only ships in-repo is still settled by the tree alone. Short of
either, the rung is unresolved: a Gap naming surfaces checked and
unchecked.

An adversarial verifier then landed two real hits on that fix, both
addressed here. The Output Format fetch log had no `unresolved` slot
and called does-not-exist "the bypass outcome a probe alone can
establish", so the reviewer's path completed anyway through the
nearest legal value; the vocabulary is five-valued now and the
probe-settles-nonexistence phrasing is struck. And the eval clause
added to case 8 could not fire against its own library-behavior
fixture, so it is generalized and case 12 supplies a real model-card
fixture.

Proven by A/B control on the two texts rather than by grep: fresh
agents given only the pre-fix rules answered probed-and-not-existing
2 of 3; given only the post-fix rules, unresolved 3 of 3.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 46s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review changed files for security issues
  • Post findings

No security issues found.

This PR touches six files, all of which are non-executable content:

  • plugins/discovery/.claude-plugin/plugin.json — a version-string bump (0.12.20.12.3), no schema or permission changes.
  • plugins/discovery/CHANGELOG.md — prose changelog entry, no code.
  • plugins/discovery/skills/research/SKILL.md, context/discipline.md, context/gotchas.md — skill-instruction prose refining the artifact-ladder absence rule (probed-and-not-existing vs unresolved) and its outcome-gate wording. No shell commands, tool invocations, URLs to fetch, or file paths are introduced; the changes only tighten epistemic guidance text that an agent reads as instructions for how to reason about source-completeness during research.
  • plugins/discovery/skills/research/evals/evals.json — adds one eval fixture (case 12) with a fictional vendor ("Vantor AI") and benchmark scenario used purely as grading-rubric test data; it is not fetched, executed, or interpreted as a live URL/command.

There are no GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets or token handling, and no permission or trust-boundary changes anywhere in this diff. The entire change is confined to the research skill's documentation of when an "artifact absence" may be concluded — a correctness/behavioral fix to agent reasoning guidance, not a security-relevant surface.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61a2355244

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/discovery/skills/research/evals/evals.json
@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 37s —— View job


Review complete

Summary

This PR closes a review thread stranded on merged PR #1260 by tightening the probed-and-not-existing absence rule from "against any one of three named surfaces" to "surfaces checked together cover every first-party surface for that claim class, or the publisher declares its inventory complete" — otherwise unresolved. I checked the six changed files for internal consistency across the (now cross-referencing) outcome-gate criterion 9, the absence rule, the doc-index table, gotchas.md, and evals.json, plus the version bump and changelog.

What holds up:

  • Criterion 9 in SKILL.md (four outcomes: probed-and-not-existing / unresolved / fetched-and-lacking-the-claim / existing-but-unreachable) and the Output Format fetch-log vocabulary (five outcomes, adding carries-the-claim) are internally consistent with each other and with the count labels used ("four", "five") — verified by grep across the file.
  • The five-surface list (docs sitemap.xml/index, in-repo docs tree, releases/asset listing, download/asset host, sibling first-party domain) is repeated identically across discipline.md, SKILL.md criterion 9, and the CHANGELOG.md entry.
  • The sitemap.xml doc-index table row (discipline.md line 183) is correctly rescoped to "pages," not "artifact inventory," which is the actual contradiction the original review thread flagged — table syntax checks out (5 pipes / 4 columns).
  • evals.json is well-formed: case IDs are sequential/unique (1–12), and new case 12 gives criterion 9's tightened rule a fixture it can actually fire against (a model-card/benchmark claim, distinct from case 8's library-behavior claims).
  • Version bump (0.12.20.12.3) matches origin/main's current 0.12.2, and the CHANGELOG.md entry follows the existing format/heading conventions used by prior entries.
  • Per CLAUDE.md's fresh-docs mandate scope rule, none of this PR's changes touch a plugin manifest field, marketplace schema, hook contract, or skill frontmatter — it's all skill/context prose body and eval fixture data, so the mandate's docs-fetch-and-cite requirement doesn't apply here.

One inline finding posted (non-blocking): the sentence appended to case 8's expectations (evals.json line 115) restates the new unresolved-vs-absent clause against a RabbitMQ/Kafka/NATS library-comparison fixture whose rung 1 is "the source itself," not a sitemap/model-card surface — so it can't actually fire there, the same vacuous-oracle problem the PR's own "Hit 2" finding called out for the clause it sits beside. New case 12 already exercises this clause with a fixture that can fail it.

I did not independently re-run scripts/affected-tests.sh, check-changelog-parity.sh, check-skill.sh, check-evals-quality.sh, or markdownlint-cli2 — the PR description's reported results for those are author-claimed and unverified by me.

Branch: fix/discovery-stranded-1260

Comment thread plugins/discovery/skills/research/evals/evals.json Outdated
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 4 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

Version collision only. Main shipped discovery 0.13.0 (the bare-alias removal)
while this branch held 0.12.3, so this entry takes 0.13.1 and both changelog
entries are kept in full — they cover unrelated changes.

Verified: no conflict markers remain, plugin.json parses, changelog versions are
monotonic with no duplicates, no MD012.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 25s —— View job


Review complete

  • Gather context (diff, changed files, CLAUDE.md/REVIEW.md rules)
  • Understand the request
  • Review changed files for correctness, architecture, error handling, maintainability
  • Post inline findings
  • Post summary comment

Summary

Re-reviewed at HEAD 793d661 (this dispatch fired on a synchronize event). The two commits since the prior review (61a2355793d661, i.e. a013d20 merged in) touch only the frontmatter-name refactor and a version-number ripple (0.12.30.13.1 to stay ahead of origin/main, which had independently advanced to 0.13.0) — no substantive change to the six files this PR's own diff touches. plugin.json's 0.13.1 still matches the new ## [0.13.1] CHANGELOG.md heading, so that resolves cleanly.

Cross-file consistency holds up. I traced the five-surface list (docs sitemap.xml/index, in-repo docs tree, releases/asset listing, download/asset host, sibling first-party domain) and the outcome-count vocabulary across all touched files:

  • Criterion 9 (SKILL.md line 184) correctly says four outcomes for rungs above the sourcing rung (which by definition never carry the claim): probed-and-not-existing / unresolved / fetched-and-lacking-the-claim / existing-but-unreachable.
  • The Output Format fetch-log vocabulary (SKILL.md line 198) correctly says five — the sourcing rung's own entry adds carries-the-claim. That's not a contradiction, it's two different populations (rungs above vs. the rung itself), and both files agree with each other.
  • discipline.md's rescoped sitemap.xml doc-index row ("Exhaustive for that host's listed pages… NOT an artifact inventory") points back at the absence rule earlier in the same file, table syntax intact (4 columns).
  • gotchas.md's new bullet, evals.json case 12's expectations, and the CHANGELOG.md entry all restate the same five-surface list and the same two ways to earn probed-and-not-existing without drifting.

This is a real fix for the reviewer's original complaint on #1260: the single-surface reading is gone everywhere it previously appeared, including the load-bearing outcome-gate criterion, not just the discipline-file prose.

One new finding, inline on evals.json:172 (case 12, absence-of-a-rung-needs-more-than-one-clean-surface): the fixture invents a fictional vendor ("Vantor AI") with files: [], so a run actually executed against this prompt has nothing real to sweep and can't exercise the scenario the case claims to test — the same vacuous-oracle shape the PR's own "Hit 2" finding diagnosed for case 8, just relocated. This also contradicts the PR's own CHANGELOG.md claim that case 12 "puts a real model-card fixture in front of the run." Non-blocking, same severity class as the still-open case-8 finding from my prior review pass on this PR (evals.json:115 — the sentence appended there can't fire against a RabbitMQ/Kafka/NATS library-comparison fixture whose rung 1 isn't a sitemap-indexable artifact). Neither finding has been addressed as of 793d661; both remain outstanding alongside Codex's independent flag of the same case-12 issue.

I did not independently re-run scripts/affected-tests.sh, check-changelog-parity.sh, check-skill.sh, check-evals-quality.sh, or markdownlint-cli2 — I did verify evals.json's JSON structure and table pipe-counts by reading the files directly, but the PR description's tool-run results remain author-claimed and unverified by me.

Branch: fix/discovery-stranded-1260

@claude

claude Bot commented Aug 9, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 41s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review changed files for security issues
  • Post findings

No security issues found.

All six changed files are non-executable content, and none introduce a new trust boundary, data flow, or execution surface:

  • plugins/discovery/.claude-plugin/plugin.json — version bump only (0.13.00.13.1).
  • plugins/discovery/CHANGELOG.md — prose changelog entry.
  • plugins/discovery/skills/research/SKILL.md, context/discipline.md, context/gotchas.md — skill-instruction prose tightening the artifact-ladder absence rule (probed-and-not-existing vs the new unresolved outcome) and expanding the fetch-log vocabulary from four values to five. No shell commands, tool invocations, URLs to fetch, or file paths are introduced or altered — this only changes the epistemic bar an agent applies when reasoning about whether a first-party artifact (e.g. a model card) can be concluded absent from partial evidence (e.g. a single sitemap scan).
  • plugins/discovery/skills/research/evals/evals.json — extends one existing case's expectation text and adds a new case 12, all fictional grading-rubric fixture data (a "Vantor AI" benchmark scenario) — not fetched, executed, or interpreted as a live URL/command.

No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission, config, or trust-boundary changes anywhere in this diff. Consistent with the two prior automated security-review passes on this PR (job runs 31340120546) which reached the same conclusion on earlier commits — this pass re-verified against the current HEAD (793d661) and the finding stands.

Branch: fix/discovery-stranded-1260

Comment thread plugins/discovery/skills/research/evals/evals.json
kyle-sexton and others added 2 commits August 9, 2026 20:35
…cuous clause

Both are the failure this PR itself named: an added test that cannot fail is
not an oracle.

Case 12 asked the run to sweep a fictional vendor's first-party surfaces with
`files` empty and nothing on the web to fetch, so the only correct outcome was
"premise unverifiable" — never the one-clean-surface-is-not-absence behaviour
the case exists to grade. It now embeds the surfaces in the prompt the way case
11 embeds its dispatch payload: a docs sitemap that parses cleanly and holds no
model card, a launch post linking an un-enumerated sibling research domain, an
asset host that 403s on listing, and no vendor completeness declaration. That
is a state where recording rung 1 as probed-and-not-existing is wrong and
unresolved-plus-Gap is right, so the expectations can now discriminate.

Case 8's appended clause could not fire there at all: its claims are
library-behaviour comparisons whose rung 1 is "the source itself" per
discipline.md's rung list, not a model card that might sit on a sitemap, an
asset host, or a sibling domain. No run against that fixture would consider
probed-and-not-existing off a single sitemap scan. Dropped rather than
reworded — case 12 is where the clause is exercisable, and it is exercisable
there now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 38s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review changed files for security issues
  • Post findings

No security issues found.

Reviewed the full diff (git diff origin/main...HEAD) at HEAD 452a788. All six changed files are non-executable content, none introduce a new trust boundary, data flow, or execution surface:

  • plugins/discovery/.claude-plugin/plugin.json — version-string bump only (0.13.00.13.1), no schema/permission changes.
  • plugins/discovery/CHANGELOG.md — prose changelog entry, no code.
  • plugins/discovery/skills/research/SKILL.md, context/discipline.md, context/gotchas.md — skill-instruction prose tightening the artifact-ladder absence rule (probed-and-not-existing vs the new unresolved outcome) and expanding the fetch-log vocabulary from four values to five. No shell commands, tool invocations, URLs to fetch, or file paths are introduced or altered — this only changes the epistemic bar an agent applies when deciding whether a first-party artifact (e.g. a model card) can be concluded absent from partial evidence (e.g. a single sitemap scan).
  • plugins/discovery/skills/research/evals/evals.json — extends case 8's expectation text and adds new case 12 (absence-of-a-rung-needs-more-than-one-clean-surface), a fictional-vendor ("Vantor AI") grading-rubric fixture with files: []. This is inert test data, not fetched, executed, or interpreted as a live URL/command — no injection surface.

No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission or config changes anywhere in this diff. Consistent with the three prior automated security-review passes on earlier commits of this PR, which reached the same conclusion — this pass re-verified against current HEAD and the finding stands.

(Note: this task's instructions asked for a security review specifically — the eval-fixture realism concerns raised by Codex and prior Claude review comments on evals.json:172 are correctness/test-quality findings, not security issues, so they're out of scope here and left to the code-review lane.)

Branch: fix/discovery-stranded-1260

@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 55s —— View job


Review complete

  • Gather context (diff, changed files, CLAUDE.md guidelines)
  • Review changed files for correctness, architecture, error handling, maintainability
  • Post inline findings (none — see below)
  • Post summary comment

Summary

Re-reviewed at HEAD 452a788 (dispatched on synchronize). Diff against origin/main is unchanged in scope from prior passes — still the same 6 files — but commit 53466e6 (landed since my last review at 793d661) directly addresses both open findings from that pass and Codex's finding:

Case 12's fixture is now real (evals.json:170). The prompt now embeds the first-party surfaces directly — a docs sitemap that parses cleanly and lists no model card, a launch post linking an un-enumerated sibling domain, an asset host that 403s, no completeness declaration — following the same pattern case 11 (evals.json:154) uses for state a live fetch can't produce. This gives the run something concrete to sweep, so expected_output and the expectations can actually discriminate pass/fail rather than the only correct answer being "premise unverifiable." Confirmed by reading the fixture directly; this resolves both Codex's finding and my own prior inline comment on this line.

Case 8's clause is dropped, not reworded (evals.json:115). The appended sentence that couldn't fire against a RabbitMQ/Kafka/NATS library-comparison fixture (rung 1 = "the source itself," never a sitemap-indexable artifact) is removed entirely rather than patched in place, leaving case 12 as the sole (and now exercisable) owner of that clause. This resolves my prior inline finding on this line.

Cross-file consistency still holds across the other four files (SKILL.md criterion 9 and the five-valued fetch-log vocabulary, discipline.md's absence rule and rescoped sitemap.xml table row, gotchas.md's new bullet, and CHANGELOG.md) — no regressions introduced by the merge of origin/main (a013d20, a large unrelated frontmatter-name refactor) or by this fixture fix. Version bump (0.13.00.13.1) still matches the new ## [0.13.1] heading and origin/main's current 0.13.0, so no changelog-parity collision. evals.json case IDs remain sequential and unique (1–12).

No new findings. Both previously open items are resolved; I found nothing else to flag in this pass.

I did not independently re-run scripts/affected-tests.sh, check-changelog-parity.sh, check-skill.sh, check-evals-quality.sh, or markdownlint-cli2 — I verified file contents, table structure, and case-ID sequencing directly via the diff and file reads, but the PR description's own tool-run results remain author-claimed and unverified by me.

Branch: fix/discovery-stranded-1260

Grading the fixture against a deliberately-wrong answer and a correct one
exposed the same defect this case exists to punish, now inside the case
itself. The wrong answer failed 5 of 8 rows, but the correct answer could not
pass: two rows graded UNGRADEABLE against BOTH answers, so no run of any
quality could clear the case.

Row 3 was gated on "before any absence verdict". A correct run issues no
absence verdict at all, so the gate never opened. It now grades the accounting
every run performs: research.vantor.ai and assets.vantor.ai must each appear
somewhere as swept or as named-unchecked. It also closes the escape the row
allowed, since prompt clause (d) states no vendor completeness declaration
exists.

Row 6's antecedent was "If the model card IS found" — unreachable here, because
the fixture has no card to find. Replaced with the branch that is reachable:
a title, snippet, or sitemap index entry LOCATES a rung and never reads it, so
no condition and no fetched-and-lacking-the-claim outcome may be sourced from
the sitemap's 47 URLs. The found-card protocol belongs in a case where a card
is actually found, not as a clause that cannot fire.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 18s —— View job


Review complete

  • Gather context (diff, changed files, prior review threads)
  • Review the new commit (8ff69cd) since the last review pass
  • Check whether prior findings are addressed / re-verify cross-file consistency
  • Post inline findings (none — see below)
  • Post summary comment

Summary

Re-reviewed at HEAD 8ff69cd (dispatched on synchronize). Confirmed via git diff 452a788..8ff69cd --stat that this commit touches only evals.json (2 insertions / 2 deletions), both inside case 12's expectations array — everything else in the six-file diff is unchanged from the 452a788 pass, where I found cross-file consistency held (criterion 9's four/five-outcome vocabulary, the five-surface list repeated across discipline.md/SKILL.md/CHANGELOG.md, the rescoped sitemap.xml doc-index row) and both prior open findings (case 12's vacuous fixture, case 8's unfireable clause) were resolved.

This commit's fix is sound. Per the commit message, grading case 12 against a deliberately-wrong and a deliberately-correct answer found two of the eight expectation rows UNGRADEABLE against both — a real defect in the oracle, same failure family as the PR's own "Hit 2" and the prior files: [] finding, just relocated one level deeper (the fixture became real in 53466e6/452a788, but two of its grading clauses still couldn't fire).

  • Row 3 (evals.json:176) was gated on "before any absence verdict" — a correct run issues no absence verdict at all (it lands on unresolved), so that gate never opened for the right answer. Replaced with a check that's gradable regardless of outcome: does the run's accounting mention research.vantor.ai and assets.vantor.ai (swept or named-unchecked) — both directly checkable against fixture clauses (b)/(c), and the vendor-completeness escape is explicitly closed by clause (d).
  • Row 6 (evals.json:179) was conditioned on "if the model card IS found" — unreachable in a fixture with no card to find. Replaced with the branch that's actually reachable here: a listing/snippet/index entry locates a rung but isn't a substitute for reading it, so no fetched-and-lacking-the-claim outcome may be sourced from the sitemap's 47 titles alone.

I traced both new rows against the case 12 prompt (evals.json:170) and confirmed each is satisfiable by the golden expected_output (evals.json:171) and checkable without ambiguity — no regression of the vacuous-oracle problem this PR has been chasing across three prior rounds. IDs remain sequential/unique (1–12, verified by grep), and the file's JSON structure is intact (read directly, not parsed — see verification note below).

No new findings.

Verification note: I could not execute check-evals-quality.sh, check-skill.sh, markdownlint-cli2, or a JSON parser in this sandbox — every non-git/read-only command in this session required approval that wasn't available, including python3 -c and the repo's own lint scripts run read-only. I verified structure and content by reading the files and git show/git diff directly instead. The PR description's reported gate results remain author-claimed and unverified by me, consistent with every prior pass on this PR.

Branch: fix/discovery-stranded-1260

@claude

claude Bot commented Aug 10, 2026

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 1m 12s —— View job


Security review complete

  • Gather context (PR diff, changed files)
  • Review changed files for security issues
  • Post findings

No security issues found.

Reviewed the full diff (git diff via gh pr diff 2116) at HEAD 8ff69cd. All six changed files are non-executable content, and none introduce a new trust boundary, data flow, or execution surface:

  • plugins/discovery/.claude-plugin/plugin.json — version-string bump only (0.13.00.13.1), no schema/permission changes.
  • plugins/discovery/CHANGELOG.md — prose changelog entry, no code.
  • plugins/discovery/skills/research/SKILL.md, context/discipline.md, context/gotchas.md — skill-instruction prose tightening the artifact-ladder absence rule (probed-and-not-existing vs the unresolved outcome) and expanding the fetch-log vocabulary from four to five values. No shell commands, tool invocations, URLs to fetch, or file paths are introduced or altered — this only changes the epistemic bar an agent applies when deciding whether a first-party artifact (e.g. a model card) can be concluded absent from partial evidence (e.g. a single sitemap scan).
  • plugins/discovery/skills/research/evals/evals.json — extends case 8's expectation text (trailing punctuation only, no substantive change from the prior HEAD) and case 12's fixture, which now embeds a concrete (fictional-vendor) surface inventory directly in the prompt field rather than relying on files. This is inert grading-rubric text — not fetched, executed, shelled out, or interpreted as a live URL/command — so it carries no injection surface distinct from any other eval prompt already in this file.

No GitHub Actions workflow changes, no scripts, no dependency/action pins, no secrets/token/credential handling, and no permission, config, or trust-boundary changes anywhere in this diff. Consistent with all five prior automated security-review passes on earlier commits of this PR (61a2355, 793d661, 452a788), which reached the same conclusion — this pass re-verified against current HEAD 8ff69cd (test-only commit adding case 12 fixture polish) and the finding stands.

Branch: fix/discovery-stranded-1260

@kyle-sexton
kyle-sexton merged commit e47964c into main Aug 10, 2026
33 checks passed
@kyle-sexton
kyle-sexton deleted the fix/discovery-stranded-1260 branch August 10, 2026 04:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant