Repository navigation
Description
Coding agents relearn the same things every session: where a function lives, which command fails here and what worked instead, what the person already said "from now on" about. Tracekit's signed records already show each of these, across sessions. This adds a Lessons part to TIPS (tracekit tips and the TIPS page of tracekit view).
What it does
- Finds lessons in the signed records, deterministically, from runs that verify:
lookup: the same search, then the same file opened, in 3+ sessions on 2+ days.fix: the same error, then the same small change that worked, 3+ times in 2+ sessions.told: a standing instruction the person typed ("from now on", "always", "never", "remember"), from the signed trusted prompt record only, never pasted or fetched text.
- Never takes the agent's own words as a lesson. Text comes from templates over evidence values, or the person's typed sentence. A value that only untrusted content brought into a run (its decision's
provenance[]) drops the occurrence. A lesson that rests on a deny or a hold, or talks about holds, approvals or the policy, is never given to the agent. - A person approves each lesson (
tracekit tips approve L-xxxxxxxx; alsoretire,restore,add). Lessons are Claude Code auto-memory notes with aMEMORY.mdindex, kept in Tracekit's data dir (which the agent is denied), with git as history. The agent's owntracekit tips approveis denied byTK-D012. - Gives approved lessons to the next Claude Code session at its first prompt (and again after a compaction or resume), ranked by BM25 against the prompt within an 8,000-byte budget, and signs a
tracekit:lessonsobserve naming which lesson ids it gave and the block's hash.tracekit tips --for-agent [--query WORDS] [--json]is the lookup for any agent. - Measures each lesson on the detector that found it: before approval vs the sessions the signed observe proves got it, shown once each side has 5 sessions.
- E23, an offline eval: planted lessons are found; a path, command or host only a fetched page brought gives zero agent-facing lessons; a deny or hold followed by a rewritten retry gives zero lessons.
Not in this change
Probe lessons, claim lessons, the repeated-approval card for the person, cost tips as lessons, subagent and other-harness injection, an optional model step to reword or grade lessons, skill drafts, a holdout measurement.
Added: find, fix, measure for every tip
Every TIPS item, not just lessons, closes the same loop on signed records:
- Find: each tip has a stable id, the detector that found it, and its rate per session or per 100 calls, citing its records.
- Fix: a lesson is fixed when the person approves it. For anything else, the person runs
tracekit tips mark IDonce they've changed something (a setting, a script, a tool). That records a signedtracekit:tipsobserve with the id. The agent's ownmarkis denied, like its ownapprove. - Measure: the same detector, before the fix and after it, on signed records. It's shown once each side has 5 sessions; a worse result is shown in the warning colour. It's labelled "before and after on the same detector, not a controlled test".
Action-layer checks, measured first on a real week (38,064 signed calls, 3 sessions, 289 subagents):
- Slow tools: build, as its own issue once this lands.
tool.resultcarries noduration_ms, so time a call as its signedtool.resulttsminus its decision'sts(signer clocks, not the agent's). Leave out failed calls, asks, subagent launches and waits for a person. Split Bash by command family. Share of a run is the union of call intervals over its wall time, since subagents overlap. Waits and polling took 13.8% of run time, and tests and builds 8.6%. - Retried unchanged: dropped. 0 of 343 failed calls were retried identically and failed again.
- Loop: deferred. An argument commitment is salted per decision, so the record can't say two calls had the same arguments. It needs a comparable call identity first. Only 7 non-polling loops were found.
- Re-read: deferred. All repeated Reads were paging, and overlapping re-reads were worth about $0.50 for the week.
The existing three cost tips get ids and the same before-and-after.
Activity
TIPS: lessons from signed runs, approved by a person, given to the ne…