Skip to content

guardrails: add a claim-verification guard (plugin-skill reference) #1270

Description

@kyle-sexton

Observation

cli-flag-verify catches one class of written-content hallucination — a
--flag that does not exist in the binary's real --help. One adjacent class
has a mechanical oracle and no guard:

Class Oracle Enforceability tier
A /plugin:skill or agent reference that does not resolve glob plugins tree Detect-then-judge

It is detect-then-judge, not deterministic, and ships declared that way. Globbing the plugins tree is exact only inside this repo; plugins must be
repo-agnostic, and in a consumer repo the reference may name a plugin from
another marketplace or one not installed. Per the org convention
conventions/engineering/enforceability-tiers.md, detect-then-judge is advisory
plus a human verdict and never an auto-fix — which the guard already is.

Excluded because already covered

  • Link and in-repo anchor resolutionlychee.toml sets
    include_fragments = "full".

  • Hallucinated CLI flags — owned by cli-flag-verify.

  • A version string disagreeing with its owning manifest. This issue
    originally scoped a third guard for this class. It was dropped during the
    build, on enumeration rather than judgment:

    • The only shape it has in this repo — a ## [<version>] CHANGELOG heading
      versus the plugin manifest — is already covered deterministically by
      scripts/check-changelog-parity.sh --check-bump, a required CI gate that
      additionally requires the entry be newly added versus the base ref.
    • Searched for an uncovered prose surface: *.md under plugins/ and docs/
      (both a version|plugin-adjacent regex and a bare-semver regex), every
      string value in all plugins/*/.claude-plugin/plugin.json and
      .claude-plugin/marketplace.json outside the version field,
      plugins/*/hooks/*.sh, .github/**/*.{yml,yaml}, scripts/, and
      .claude/. Not checked: *.mjs outside scripts/, and untracked files.
    • Every hit is either a third-party version (Claude Code CLI, firecrawl
      upstream, DeDRM, Calibre, boris) that no local manifest can adjudicate, or a
      claim shape a manifest-compare oracle gets wrong: historical
      (before the 0.6.0 split), a minimum floor
      (plugins/work-items/skills/work/SKILL.md cites implementation 0.9.0+,
      currently true and explicitly documented there as not a manifest version
      dependency), or planned (CREATE: 0.1.0 in a topic PLAN).

    A guard whose only surface a required CI gate already owns supplies no signal
    that gate does not already produce, while adding a second place the rule can
    drift — the one mechanism per concern principle in PLUGIN-PHILOSOPHY.md's
    validation section. Its residual surface it would additionally misread. Recorded
    here rather than shipped.

    (Correction: an earlier revision cited "a silently skipped feature is a defect"
    here. That doctrine concerns a promised feature doing nothing when a runtime
    prerequisite is absent, not redundancy. The deletion criterion is "supplies no
    additional signal" — a coverage test, not a correctness one.)

  • An asserted repo-relative path that does not exist. Built, measured, and
    withdrawn. Swept across all 975 tracked markdown files it fired on 23.7% of
    them with zero true positives across 389 findings; 72% were consumer-project
    config paths a marketplace correctly lacks. Fixing the three dominant causes
    still left ~4% firing at zero true positives, so a repo-root filesystem test is
    the wrong oracle for a repo whose docs are largely about other repos' trees. The
    full measurement lives on guardrails: rescope asserted-path-verify — 23.7% firing rate, zero true positives on a full-corpus sweep #1314 for rescoping.

Direction (no open decision)

One advisory PostToolUse guard on Write|Edit, following the
cli-flag-verify pattern exactly — diff-scope only (scan what the call wrote,
never re-read from disk), per-guard userConfig kill switch defaulting true,
statusMessage on each handler, telemetry on meaningful outcomes, and a
co-located test with MUST-fire, MUST-stay-quiet, kill-switch, empty-stdin,
missing-prerequisite, and telemetry cases.

hook::require_jq's second argument must be unique plugin-wide —
require-jq-notice-isolation.test.sh discovers hooks by glob and fails on
duplicates.

Do not add helpers to lib/hook-utils.sh unless unavoidable — that forces a
sync across every carrying plugin plus a version bump and CHANGELOG entry in
each.

Boy Scout, same PR: the README guard count and its tables are stale — they omit
block-convention-violation. Re-measure the wired set at authoring time rather
than trusting a prior count.

Acceptance

  • The guard ships advisory (exit 0), independently toggleable
  • Its enforceability tier is stated where it is documented; the reference
    guard is not described as deterministic
  • hook::require_jq keys are unique plugin-wide
  • scripts/check-silent-skips.sh passes with no new annotation exemptions
  • lib/hook-utils.sh is unchanged
  • README guard counts and tables match the wired set, measured not asserted (11, reconciled three ways)
  • Version bump plus CHANGELOG entry (0.15.0)

This was generated by AI during a /planning:interview session on the fact-check capability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions