You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
cli-flag-verify catches one class of written-content hallucination — a --flag that does not exist in the binary's real --help. One adjacent class
has a mechanical oracle and no guard:
Class
Oracle
Enforceability tier
A /plugin:skill or agent reference that does not resolve
glob plugins tree
Detect-then-judge
It is detect-then-judge, not deterministic, and ships declared that way. Globbing the plugins tree is exact only inside this repo; plugins must be
repo-agnostic, and in a consumer repo the reference may name a plugin from
another marketplace or one not installed. Per the org convention conventions/engineering/enforceability-tiers.md, detect-then-judge is advisory
plus a human verdict and never an auto-fix — which the guard already is.
Excluded because already covered
Link and in-repo anchor resolution — lychee.toml sets include_fragments = "full".
Hallucinated CLI flags — owned by cli-flag-verify.
A version string disagreeing with its owning manifest. This issue
originally scoped a third guard for this class. It was dropped during the
build, on enumeration rather than judgment:
The only shape it has in this repo — a ## [<version>] CHANGELOG heading
versus the plugin manifest — is already covered deterministically by scripts/check-changelog-parity.sh --check-bump, a required CI gate that
additionally requires the entry be newly added versus the base ref.
Searched for an uncovered prose surface: *.md under plugins/ and docs/
(both a version|plugin-adjacent regex and a bare-semver regex), every
string value in all plugins/*/.claude-plugin/plugin.json and .claude-plugin/marketplace.json outside the version field, plugins/*/hooks/*.sh, .github/**/*.{yml,yaml}, scripts/, and .claude/. Not checked: *.mjs outside scripts/, and untracked files.
Every hit is either a third-party version (Claude Code CLI, firecrawl
upstream, DeDRM, Calibre, boris) that no local manifest can adjudicate, or a
claim shape a manifest-compare oracle gets wrong: historical
(before the 0.6.0 split), a minimum floor
(plugins/work-items/skills/work/SKILL.md cites implementation 0.9.0+,
currently true and explicitly documented there as not a manifest version
dependency), or planned (CREATE: 0.1.0 in a topic PLAN).
A guard whose only surface a required CI gate already owns supplies no signal
that gate does not already produce, while adding a second place the rule can
drift — the one mechanism per concern principle in PLUGIN-PHILOSOPHY.md's
validation section. Its residual surface it would additionally misread. Recorded
here rather than shipped.
(Correction: an earlier revision cited "a silently skipped feature is a defect"
here. That doctrine concerns a promised feature doing nothing when a runtime
prerequisite is absent, not redundancy. The deletion criterion is "supplies no
additional signal" — a coverage test, not a correctness one.)
An asserted repo-relative path that does not exist. Built, measured, and withdrawn. Swept across all 975 tracked markdown files it fired on 23.7% of
them with zero true positives across 389 findings; 72% were consumer-project
config paths a marketplace correctly lacks. Fixing the three dominant causes
still left ~4% firing at zero true positives, so a repo-root filesystem test is
the wrong oracle for a repo whose docs are largely about other repos' trees. The
full measurement lives on guardrails: rescope asserted-path-verify — 23.7% firing rate, zero true positives on a full-corpus sweep #1314 for rescoping.
Direction (no open decision)
One advisory PostToolUse guard on Write|Edit, following the cli-flag-verify pattern exactly — diff-scope only (scan what the call wrote,
never re-read from disk), per-guard userConfig kill switch defaulting true, statusMessage on each handler, telemetry on meaningful outcomes, and a
co-located test with MUST-fire, MUST-stay-quiet, kill-switch, empty-stdin,
missing-prerequisite, and telemetry cases.
hook::require_jq's second argument must be unique plugin-wide — require-jq-notice-isolation.test.sh discovers hooks by glob and fails on
duplicates.
Do not add helpers to lib/hook-utils.sh unless unavoidable — that forces a
sync across every carrying plugin plus a version bump and CHANGELOG entry in
each.
Boy Scout, same PR: the README guard count and its tables are stale — they omit block-convention-violation. Re-measure the wired set at authoring time rather
than trusting a prior count.
Acceptance
The guard ships advisory (exit 0), independently toggleable
Its enforceability tier is stated where it is documented; the reference
guard is not described as deterministic
hook::require_jq keys are unique plugin-wide
scripts/check-silent-skips.sh passes with no new annotation exemptions
lib/hook-utils.sh is unchanged
README guard counts and tables match the wired set, measured not asserted (11, reconciled three ways)
Version bump plus CHANGELOG entry (0.15.0)
This was generated by AI during a /planning:interview session on the fact-check capability.
Observation
cli-flag-verifycatches one class of written-content hallucination — a--flagthat does not exist in the binary's real--help. One adjacent classhas a mechanical oracle and no guard:
/plugin:skillor agent reference that does not resolveIt is detect-then-judge, not deterministic, and ships declared that way. Globbing the plugins tree is exact only inside this repo; plugins must be
repo-agnostic, and in a consumer repo the reference may name a plugin from
another marketplace or one not installed. Per the org convention
conventions/engineering/enforceability-tiers.md, detect-then-judge is advisoryplus a human verdict and never an auto-fix — which the guard already is.
Excluded because already covered
Link and in-repo anchor resolution —
lychee.tomlsetsinclude_fragments = "full".Hallucinated CLI flags — owned by
cli-flag-verify.A version string disagreeing with its owning manifest. This issue
originally scoped a third guard for this class. It was dropped during the
build, on enumeration rather than judgment:
## [<version>]CHANGELOG headingversus the plugin manifest — is already covered deterministically by
scripts/check-changelog-parity.sh --check-bump, a required CI gate thatadditionally requires the entry be newly added versus the base ref.
*.mdunderplugins/anddocs/(both a
version|plugin-adjacent regex and a bare-semver regex), everystring value in all
plugins/*/.claude-plugin/plugin.jsonand.claude-plugin/marketplace.jsonoutside theversionfield,plugins/*/hooks/*.sh,.github/**/*.{yml,yaml},scripts/, and.claude/. Not checked:*.mjsoutsidescripts/, and untracked files.upstream, DeDRM, Calibre, boris) that no local manifest can adjudicate, or a
claim shape a manifest-compare oracle gets wrong: historical
(
before the 0.6.0 split), a minimum floor(
plugins/work-items/skills/work/SKILL.mdcitesimplementation 0.9.0+,currently true and explicitly documented there as not a manifest version
dependency), or planned (
CREATE: 0.1.0in a topic PLAN).A guard whose only surface a required CI gate already owns supplies no signal
that gate does not already produce, while adding a second place the rule can
drift — the one mechanism per concern principle in
PLUGIN-PHILOSOPHY.md'svalidation section. Its residual surface it would additionally misread. Recorded
here rather than shipped.
(Correction: an earlier revision cited "a silently skipped feature is a defect"
here. That doctrine concerns a promised feature doing nothing when a runtime
prerequisite is absent, not redundancy. The deletion criterion is "supplies no
additional signal" — a coverage test, not a correctness one.)
An asserted repo-relative path that does not exist. Built, measured, and
withdrawn. Swept across all 975 tracked markdown files it fired on 23.7% of
them with zero true positives across 389 findings; 72% were consumer-project
config paths a marketplace correctly lacks. Fixing the three dominant causes
still left ~4% firing at zero true positives, so a repo-root filesystem test is
the wrong oracle for a repo whose docs are largely about other repos' trees. The
full measurement lives on guardrails: rescope asserted-path-verify — 23.7% firing rate, zero true positives on a full-corpus sweep #1314 for rescoping.
Direction (no open decision)
One advisory
PostToolUseguard onWrite|Edit, following thecli-flag-verifypattern exactly — diff-scope only (scan what the call wrote,never re-read from disk), per-guard
userConfigkill switch defaulting true,statusMessageon each handler, telemetry on meaningful outcomes, and aco-located test with MUST-fire, MUST-stay-quiet, kill-switch, empty-stdin,
missing-prerequisite, and telemetry cases.
hook::require_jq's second argument must be unique plugin-wide —require-jq-notice-isolation.test.shdiscovers hooks by glob and fails onduplicates.
Do not add helpers to
lib/hook-utils.shunless unavoidable — that forces async across every carrying plugin plus a version bump and CHANGELOG entry in
each.
Boy Scout, same PR: the README guard count and its tables are stale — they omit
block-convention-violation. Re-measure the wired set at authoring time ratherthan trusting a prior count.
Acceptance
guard is not described as deterministic
hook::require_jqkeys are unique plugin-widescripts/check-silent-skips.shpasses with no new annotation exemptionslib/hook-utils.shis unchangedThis was generated by AI during a /planning:interview session on the fact-check capability.