Skip to content

TIPS: lessons from signed runs, approved by a person, given to the next session #859

Closed

Description

@LoopGlitch26

Coding agents relearn the same things every session: where a function lives, which command fails here and what worked instead, what the person already said "from now on" about. Tracekit's signed records already show each of these, across sessions. This adds a Lessons part to TIPS (tracekit tips and the TIPS page of tracekit view).

What it does

  1. Finds lessons in the signed records, deterministically, from runs that verify:
    • lookup: the same search, then the same file opened, in 3+ sessions on 2+ days.
    • fix: the same error, then the same small change that worked, 3+ times in 2+ sessions.
    • told: a standing instruction the person typed ("from now on", "always", "never", "remember"), from the signed trusted prompt record only, never pasted or fetched text.
  2. Never takes the agent's own words as a lesson. Text comes from templates over evidence values, or the person's typed sentence. A value that only untrusted content brought into a run (its decision's provenance[]) drops the occurrence. A lesson that rests on a deny or a hold, or talks about holds, approvals or the policy, is never given to the agent.
  3. A person approves each lesson (tracekit tips approve L-xxxxxxxx; also retire, restore, add). Lessons are Claude Code auto-memory notes with a MEMORY.md index, kept in Tracekit's data dir (which the agent is denied), with git as history. The agent's own tracekit tips approve is denied by TK-D012.
  4. Gives approved lessons to the next Claude Code session at its first prompt (and again after a compaction or resume), ranked by BM25 against the prompt within an 8,000-byte budget, and signs a tracekit:lessons observe naming which lesson ids it gave and the block's hash. tracekit tips --for-agent [--query WORDS] [--json] is the lookup for any agent.
  5. Measures each lesson on the detector that found it: before approval vs the sessions the signed observe proves got it, shown once each side has 5 sessions.
  6. E23, an offline eval: planted lessons are found; a path, command or host only a fetched page brought gives zero agent-facing lessons; a deny or hold followed by a rewritten retry gives zero lessons.

Not in this change

Probe lessons, claim lessons, the repeated-approval card for the person, cost tips as lessons, subagent and other-harness injection, an optional model step to reword or grade lessons, skill drafts, a holdout measurement.

Added: find, fix, measure for every tip

Every TIPS item, not just lessons, closes the same loop on signed records:

  1. Find: each tip has a stable id, the detector that found it, and its rate per session or per 100 calls, citing its records.
  2. Fix: a lesson is fixed when the person approves it. For anything else, the person runs tracekit tips mark ID once they've changed something (a setting, a script, a tool). That records a signed tracekit:tips observe with the id. The agent's own mark is denied, like its own approve.
  3. Measure: the same detector, before the fix and after it, on signed records. It's shown once each side has 5 sessions; a worse result is shown in the warning colour. It's labelled "before and after on the same detector, not a controlled test".

Action-layer checks, measured first on a real week (38,064 signed calls, 3 sessions, 289 subagents):

  • Slow tools: build, as its own issue once this lands. tool.result carries no duration_ms, so time a call as its signed tool.result ts minus its decision's ts (signer clocks, not the agent's). Leave out failed calls, asks, subagent launches and waits for a person. Split Bash by command family. Share of a run is the union of call intervals over its wall time, since subagents overlap. Waits and polling took 13.8% of run time, and tests and builds 8.6%.
  • Retried unchanged: dropped. 0 of 343 failed calls were retried identically and failed again.
  • Loop: deferred. An argument commitment is salted per decision, so the record can't say two calls had the same arguments. It needs a comparable call identity first. Only 7 non-polling loops were found.
  • Re-read: deferred. All repeated Reads were paging, and overlapping re-reads were worth about $0.50 for the week.

The existing three cost tips get ids and the same before-and-after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions