Skip to content

Measure whether /tdd:principles actually gets invoked during implementation (SkillUse telemetry, not evals) #3048

Description

@kyle-sexton

Spun out of Lane C (#2936). Refs #2933.

The claim, and why it is still open

#2936 asserted that "/tdd:principles exists but rarely gets invoked during implementation (known behavioral gap)."

While adjudicating that lane I first judged the claim STALE, on the grounds that the invocation is already wired — implement/SKILL.md names it at the "Before writing first test" chaining row, and grep finds 7 wired sites across the fleet. Validation showed that inverts the claim. Wiring is a property of text; invocation is runtime behavior. Seven wired sites plus rare invocation are not a contradiction — jointly they are evidence that wiring is not the lever. The presence of the instruction is exactly what makes a behavioral gap interesting.

Corrected disposition, now recorded in docs/upstream/aihero-shipping-course.md (Lane C): UNVERIFIED, with wiring confirmed at 7 sites and the behavioral half unmeasured.

Why not evals

The first disposal proposed was "file it as an evals item." That venue does not execute anything. plugins/evals/README.md:24-26:

No command in this plugin executes model-graded evals.

CI schema-validates eval content only. Filing there would produce JSON no runner runs.

The instrument that does exist

plugins/claude-ops/hooks/skill-usage-audit.sh writes a SkillUse event per skill invocation to skill-usage.jsonl, with a sibling skill-usage-expansion-audit.sh covering the expansion path (source: "tool" vs source: "expansion").

Recorded fields today: ts, event, skill, branch, project, project_id, hook, source.

Stated limit, up front: there is no caller attribution in that record. Nothing says which skill invoked tdd:principles, or that an implementation session was underway. The best available reading is co-occurrence — implement and tdd:principles events on the same branch within some window — which is a proxy, not proof. Any conclusion drawn must carry that caveat, and inflating the proxy into a measurement is the specific failure mode to avoid here.

What to build

  1. A query/report over skill-usage.jsonl that answers: across sessions where implementation:implement fired, in what fraction did tdd:principles also fire, and in what order. plugins/claude-ops/skills/observability is the natural home.
  2. The answer recorded against Lane C: TDD wiring + zero-assembly default chain #2936's premise in the SSOT — either the gap is real (and then the remedy is a separate item, since more wiring is already ruled out as the lever) or it is not.
  3. Optional and worth costing separately: adding caller context to the SkillUse record so co-occurrence can be replaced with attribution. That widens a telemetry schema and touches the hook-telemetry convention, so it is a decision, not a detail.

Acceptance criteria

  • A repeatable reading of implement→tdd co-occurrence exists, with its proxy limitation stated in the output itself
  • Lane C: TDD wiring + zero-assembly default chain #2936's premise is settled in the SSOT as measured, with the number, or explicitly recorded as unmeasurable with the current schema
  • If unmeasurable, the schema-widening decision is filed as its own item rather than assumed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.priority: needs-triageDefault until a priority tier is assigned.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions