You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#2936 asserted that "/tdd:principles exists but rarely gets invoked during implementation (known behavioral gap)."
While adjudicating that lane I first judged the claim STALE, on the grounds that the invocation is already wired — implement/SKILL.md names it at the "Before writing first test" chaining row, and grep finds 7 wired sites across the fleet. Validation showed that inverts the claim. Wiring is a property of text; invocation is runtime behavior. Seven wired sites plus rare invocation are not a contradiction — jointly they are evidence that wiring is not the lever. The presence of the instruction is exactly what makes a behavioral gap interesting.
Corrected disposition, now recorded in docs/upstream/aihero-shipping-course.md (Lane C): UNVERIFIED, with wiring confirmed at 7 sites and the behavioral half unmeasured.
Why not evals
The first disposal proposed was "file it as an evals item." That venue does not execute anything. plugins/evals/README.md:24-26:
No command in this plugin executes model-graded evals.
CI schema-validates eval content only. Filing there would produce JSON no runner runs.
The instrument that does exist
plugins/claude-ops/hooks/skill-usage-audit.sh writes a SkillUse event per skill invocation to skill-usage.jsonl, with a sibling skill-usage-expansion-audit.sh covering the expansion path (source: "tool" vs source: "expansion").
Stated limit, up front: there is no caller attribution in that record. Nothing says which skill invoked tdd:principles, or that an implementation session was underway. The best available reading is co-occurrence — implement and tdd:principles events on the same branch within some window — which is a proxy, not proof. Any conclusion drawn must carry that caveat, and inflating the proxy into a measurement is the specific failure mode to avoid here.
What to build
A query/report over skill-usage.jsonl that answers: across sessions where implementation:implement fired, in what fraction did tdd:principles also fire, and in what order. plugins/claude-ops/skills/observability is the natural home.
The answer recorded against Lane C: TDD wiring + zero-assembly default chain #2936's premise in the SSOT — either the gap is real (and then the remedy is a separate item, since more wiring is already ruled out as the lever) or it is not.
Optional and worth costing separately: adding caller context to the SkillUse record so co-occurrence can be replaced with attribution. That widens a telemetry schema and touches the hook-telemetry convention, so it is a decision, not a detail.
Acceptance criteria
A repeatable reading of implement→tdd co-occurrence exists, with its proxy limitation stated in the output itself
Spun out of Lane C (#2936). Refs #2933.
The claim, and why it is still open
#2936 asserted that "
/tdd:principlesexists but rarely gets invoked during implementation (known behavioral gap)."While adjudicating that lane I first judged the claim STALE, on the grounds that the invocation is already wired —
implement/SKILL.mdnames it at the "Before writing first test" chaining row, and grep finds 7 wired sites across the fleet. Validation showed that inverts the claim. Wiring is a property of text; invocation is runtime behavior. Seven wired sites plus rare invocation are not a contradiction — jointly they are evidence that wiring is not the lever. The presence of the instruction is exactly what makes a behavioral gap interesting.Corrected disposition, now recorded in
docs/upstream/aihero-shipping-course.md(Lane C): UNVERIFIED, with wiring confirmed at 7 sites and the behavioral half unmeasured.Why not evals
The first disposal proposed was "file it as an evals item." That venue does not execute anything.
plugins/evals/README.md:24-26:CI schema-validates eval content only. Filing there would produce JSON no runner runs.
The instrument that does exist
plugins/claude-ops/hooks/skill-usage-audit.shwrites aSkillUseevent per skill invocation toskill-usage.jsonl, with a siblingskill-usage-expansion-audit.shcovering the expansion path (source: "tool"vssource: "expansion").Recorded fields today:
ts,event,skill,branch,project,project_id,hook,source.Stated limit, up front: there is no caller attribution in that record. Nothing says which skill invoked
tdd:principles, or that an implementation session was underway. The best available reading is co-occurrence —implementandtdd:principlesevents on the samebranchwithin some window — which is a proxy, not proof. Any conclusion drawn must carry that caveat, and inflating the proxy into a measurement is the specific failure mode to avoid here.What to build
skill-usage.jsonlthat answers: across sessions whereimplementation:implementfired, in what fraction didtdd:principlesalso fire, and in what order.plugins/claude-ops/skills/observabilityis the natural home.SkillUserecord so co-occurrence can be replaced with attribution. That widens a telemetry schema and touches the hook-telemetry convention, so it is a decision, not a detail.Acceptance criteria