feat(headless): validate task tools with runtime-policy A/B #317

Activity

1. Astro-Han

   added

   enhancementNew feature or requestNew feature or request

   on Jun 26, 2026
2. Astro-Han commented on Jun 29, 2026

   

   Astro-Han

   on Jun 29, 2026

   ContributorAuthor

   More actions

   Construction plan & frozen experiment design

   This plan was pressure-tested with an independent second-opinion review before posting; the statistical and wiring corrections from that pass are folded in. It supersedes the original A/B Plan and Decision Gate wording above on two points (margin and baseline) — see §5.

   0. Frozen experiment design (pre-registration — fixed before any run)

   Arms (two only):

   • A (baseline): no task-class tools at all. Heavy-task progress tools (todo_update / inventory_submit) are off in both arms for this experiment, so A is a clean "no ledger" baseline rather than "existing todo, no new task_*". This intentionally narrows the original "keep existing heavy-task progress as the baseline" wording; rationale in §5.
   • B: A + task_create / task_update / task_list / task_get + capped task-state replay.
   • Both arms run with active tool-result prune at 2048 (current default). The only intended policy difference is the task tools. B's larger tool schema, extra tool_call/tool_result events, and any extra steps are part of the task-tool intervention's footprint — they are not treated as confounders, but for that reason no post-treatment subset (e.g. "attempts where B actually activated tasks", "attempts where active prune fired") is used for the primary judgment.

   Primary decision (computed on the full accumulated sample): B−A pass-rate difference with a confidence interval, read as four outcomes:

   • superior — CI lower bound > 0
   • non-inferior — CI lower bound in (−10pp, 0]
   • inferior — CI upper bound < −10pp (strong evidence of meaningful harm)
   • inconclusive — anything else (CI too wide / straddles the margin)

   Margin = 10pp, matching the phase-1 framing already used for the active-prune A/B (#340). This is the margin a ~240-pair sample can actually resolve. A 5pp margin was considered and rejected: at ~240 pairs the one-sided lower-bound half-width is ~7–8pp, so a true 5pp effect would not be resolvable; resolving 5pp would need on the order of 1000+ pairs per arm (~4× the cost). The run's purpose is therefore to detect a large regression, estimate the effect size, and opportunistically surface a superiority signal — not to prove a 5pp lift.

   Framework gap (ships in the implementation PR as "necessary for measurability"): the current A/B decision type is non_inferior / inferior / inconclusive with a point-estimate rule; it has no superior label and does not use the CI-lower-bound tiers above. Implementing this readout requires extending ab-summary.ts / ab-types.ts / ab-render.ts. The same PR should also switch the CI to a paired/clustered estimator: the design is paired (same task, both arms), but the framework currently uses an independent two-proportion (Newcombe/Wilson) interval, which discards the pairing — and precision is the binding constraint here.

   Diagnostics only (never gate promotion): expected-duration buckets (le30 / 30to45 / 45to60 / gt60), active-prune subset, task-activation subset. Long-horizon analysis is sliced on the pre-treatment expected-duration axis, never on realized trace/turn/token length (which is post-treatment and would bias the subgroup).

   Pre-registered guards (fixed before the run):

   • Low activation: if fewer than 50% of B attempts create at least one task, the only allowed conclusion is "the intervention did not take effect" — not "task tools have no value".
   • Caps: total rounds ≤ 3 (round 1 ≈ 80 pairs, target ~240 total); hard cost cap to be set against budget before freeze; futility stop after round 1 only when the round-1 point estimate ≤ −10pp and its CI excludes 0 on the negative side (stop for a clearly bad direction, never an early "superior" call); task-state replay ≤ 1024 tokens; task_list returns ≤ 20 active tasks (≤ 1024 tokens); per-turn task-update cap surfaced as the stale/repeated-update metric; per-attempt max 50 tasks.

   Metrics: primary = official verifier pass rate. Secondary = tokens / cost / duration / turns, incomplete / timeout / budget-exhausted rate, task activation rate, repeated-or-stale task-update rate.

   1. Phase 1 — task tool group (runtime primitive, off by default)

   • New runtime sibling module (e.g. packages/runtime/src/task-ledger-tools.ts); not added to buildBuiltinTools(). Made a runtime primitive (not headless-only) so desktop can enable it later once it passes the gate.
   • Tools: task_create / task_update / task_list / task_get, with terse descriptions to discourage over-planning.
   • State machine: pending / in_progress / blocked / completed / cancelled; blocked is first-class with reason/evidence; invalid transitions and duplicate IDs are rejected.
   • Mutations recorded as runtime tool events; the latest task state is projectable. Layering: runtime owns the ledger tool factory + projection/replay; headless maps task events into its task-run store via an optional recorder callback. Runtime does not import headless task-run types.
   • Enablement: registered only when an experiment flag is set. This requires adding a new Harbor context-env key (e.g. MAKA_CONTEXT_TASK_TOOLS) to the whitelist plus its snapshot/fingerprint, and a tool-assembly point in the Harbor backend path — the existing whitelist only covers context-budget/prune/compact/archive keys, so this is net-new wiring, not a free reuse.
   • Tests: create/update/list/get, invalid transitions, duplicate IDs, off-state registers nothing.

   2. Phase 2 — capped task-state replay

   • Project the latest task state back into retry/resume prompts under a hard token cap.
   • This is the channel through which the ledger's value is realized: with active prune on, raw task_list results may be pruned from context, so the durable signal must come from the capped replay, not from tool-result messages persisting in-context.
   • Ships in the same PR as Phase 1 (the replay is part of the ledger's recoverable semantics). Tests cover the projection cap.

   3. Phase 3 — full run as accumulating rounds

   • Use the runtime-policy A/B harness (runtime-policy-ab-run.ts). Target a full run over the reused Terminal-Bench task pool, executed as multiple accumulating rounds (the harness already has a round concept); rounds stratify into one final estimate, as the prune A/B did.
   • Round 1 (~40 tasks × 2 reps ≈ 80 pairs, covering all duration buckets) doubles as an interim direction check. Interim looks are for futility only (see guard above); no early superiority claim, to avoid optional-stopping inflation. Final four-outcome decision is computed on the full accumulated sample.
   • Reuse result-export / runtime-policy-ab reporting. maka-eval outputs stay git-excluded.

   4. PRs, issue body, out of scope

   • Implementation PR: Phase 1 + Phase 2 + the env wiring + the A/B decision/report extensions (superior label, CI-lower-bound tiers, paired CI) + tests. One coherent, end-to-end-verifiable change.
   • Results: posted as a comment on this issue with the report and the four-outcome decision; no separate PR unless report scaffolding itself changes.
   • Issue body: a minimal edit will add a pointer to this comment and flag the two refinements it supersedes (margin fixed at 10pp with a four-outcome readout; baseline runs heavy-task progress tools off in both arms).
   • Out of scope (unchanged from the non-goals above): no desktop UI this phase, no default product enablement, no arm C (free-text todo), no plan-reminder/automation, no goal integration, no subagent task assignment, no cross-session task database.
3. Astro-Han commented on Jul 5, 2026

   

   Astro-Han

   on Jul 5, 2026

   ContributorAuthor

   More actions

   Revised implementation scope: CRUD-lite Task tools as an experimental harness primitive

   After #537, the remaining #317 work should not directly promote the desktop session task ledger, and it should not invent a benchmark-only checkpoint mechanism. The experiment should answer a narrower, more general question:

   Do common model-facing Task/Todo tools provide net value as an agent harness primitive when paired with bounded task-state replay?

   First experimental baseline

   Use a single candidate arm:

   • A: no task-class tools and no task-state replay.
   • B: CRUD-lite Task tools + capped automatic task-state replay.

   The B-arm tool family should stay close to the original #317 and to common Task tool practice:

   • task_create
   • task_update
   • task_list
   • task_get

   Keep the first schema deliberately small: short task title/description, status (pending, in_progress, blocked, completed, cancelled), and minimal blocker/evidence text where needed. Defer priority, owner, dependencies, workflow semantics, cross-session task databases, and rich evidence references.

   Relationship to #537

   #537 is useful implementation groundwork, not the completed experiment:

   • reuse its session persistence direction, safe rendering/redaction patterns, and turn-tail injection seam;
   • do not expose the experiment tools in default desktop/runtime configuration;
   • add headless/Harbor experiment wiring behind an explicit flag;
   • make replay capped and experimental, not an unconditional product default;
   • collect task activation, task call, stale/repeated update, token/cost/duration, timeout, and pass-rate diagnostics.

   PR policy

   The implementation can live in one draft PR so the full experiment surface is reviewable together: tools, storage/projection, Harbor/runtime-policy flag, metrics, reporting, and tests. The PR remains experimental and off by default.

   Decision after the A/B run:

   • If the candidate has clear net value, mark the PR ready or follow up with a production enablement decision.
   • If it is ineffective or harmful, close the PR and delete the branch rather than merging unused task tools into production code.
   • If the result is inconclusive, keep the PR draft or narrow it to reusable harness/reporting pieces, still with no default production enablement.

   This keeps the experiment honest: code enters the repo only if the Task-tool harness primitive earns its keep.
4. Astro-Han

   added a commit that references this issue on Jul 6, 2026

   feat(headless): add experimental todo task-tool baseline

   ...

   3dca793
5. Nyvo-io commented on Jul 22, 2026

   

   Nyvo-io

   on Jul 22, 2026

   Contributor

   More actions

   I would like to take #317. I audited the current main branch before proposing the scope: the Core/Runtime Task Ledger and task_create/task_update/task_list/task_get tools now exist, while the Headless experiment still exposes the narrower todo_write surface and its current metrics are todo_write-specific. I found no open PR, assignee, or active upstream branch implementing the remaining experiment.

   My proposed implementation/evidence boundary is:

   1. Reuse the existing Core/Runtime Task Ledger and CRUD tool implementations; do not add a parallel ledger or benchmark-only task model.
   2. Add an explicit off-by-default Harbor/runtime-policy candidate arm that exposes the existing CRUD-lite tools with bounded automatic task-state replay. The baseline has no task-class tools or task replay.
   3. Extend cell/WAL/A-B evidence to report per-tool calls, activation, replay size, and repeated/stale update diagnostics without making post-treatment subsets authoritative for the primary decision.
   4. Audit the frozen statistical contract against the current paired non-inferiority implementation and add only the missing decision/report behavior needed by feat(headless): validate task tools with runtime-policy A/B #317, including a distinct superior outcome and fail-closed low-activation/incomplete-evidence handling.
   5. Cover flag-off behavior, tool wiring, replay budgets, restart/resume evidence, metric reconstruction, and decision boundaries with focused tests, then run the full relevant Headless suite.
   6. Run a single-task Harbor/Docker smoke first to prove arm separation and evidence extraction. Only after that, run the pre-registered paired benchmark under an explicit hard cost cap and publish the exact environment, artifacts, and four-outcome result.

   I will keep product defaults unchanged, preserve official verifier authority, and avoid Desktop UI, workflow, subagent assignment, or cross-session task-database scope.

   Before opening a branch, I would like to confirm two points:

   • should the current CRUD-lite candidate replace the todo_write experiment for this issue, or should both remain selectable experimental profiles?
   • should the implementation stay draft until the full paired benchmark is complete, as the latest PR policy comment specifies?

   If this boundary matches the current intent, please assign me. I will wait for confirmation and assignment before starting implementation.
6. Astro-Han commented on Aug 23, 2026

   

   Astro-Han

   on Aug 23, 2026

   ContributorAuthor

   More actions

   Closing this experiment definition as superseded by #2605. The reusable Task Ledger and CRUD tools now exist, but the Headless runtime-policy arm, todo_write metrics, retry projection, and benchmark wiring named here were removed with the old Headless architecture.

   The product question—whether Task tools earn their token and action-space cost—remains reasonable. A new experiment should reuse the current Task Ledger and run through @maka/eval and the Runtime Host rather than restoring a benchmark-specific execution path.

   Triage and wording were prepared with OpenAI Codex and reviewed by the maintainer.
7. Astro-Han

   closed this as not plannedon Aug 23, 2026