What is missing
BUILD_PERFORMANCE.md covers build speed. Nothing in the repository measures
runtime speed: no benches/, no criterion / iai / divan in Cargo.lock, no
.perf cases, no budget constants, no gate that fails when a user path gets
slower. Verified 2026-09-15.
So every claim about responsiveness in this milestone ("streaming is bursty",
"typing feels laggy", "make it smooth and fast") is currently unfalsifiable. The
stream-pacing change in 9b41aeaac was reproduced and measured with a scratch
harness that was thrown away; nothing keeps that measurement honest next month.
What good looks like, learned from deepseek-harness
refs/dsh/benchmarks/AGENTS.md runs required repository-level performance gates
whose measured path crosses package ownership. The transferable parts, in the
order I would take them:
- Organize by measured user path, not by package tree. One directory per
path. DSH has session-open, agent-continuation, conversation-fold,
active-stream-reconnect, terminal-io, long-session-browser.
- Fixed synthetic inputs from reviewed constants. "Never use recorded
Sessions, user material, ambient repositories, or network services."
- Budgets are reviewed source constants. "Environment variables must not
override performance budgets."
- Say how the verdict is reached. "Report enough raw and aggregate
measurements to explain each verdict, including whether a budget uses a
median, minimum, absolute value, or ratio."
- Separate expectations by machine. Reference-machine expectations are
recorded separately from the shared CI time scale and variance headroom, and
the time scale is never applied to memory or to dimensionless ratios.
- Fresh process, private root, bounded child for any wall-clock or
retained-memory sample; remove owned roots after failure as well as success.
- One named benchmark per real defect class.
active-stream-reconnect
exists because a reconnect carrying a 100,000-delta reasoning prefix has to
fold without stalling — a specific shape, not a generic "streaming is fast".
The decision this needs first
A harness means a benchmark dependency, and criterion / iai / divan are all
absent from Cargo.lock. Adding one is a dependency decision on a repository
currently working to reduce dependency surface (#6151), so it should be a
deliberate choice rather than a drive-by. The alternatives are a dependency-free
Instant-based harness in a child process, or adopting one of the above.
Also unanswered: what wall-clock assertion can survive a busy developer machine,
and which user paths are actually gated versus merely measured. DSH answers both;
we have not.
First candidates, straight from this milestone
- Stream reveal throughput and latency: the per-beat advance, and the lag
between a provider delta arriving and its text being visible, at several
backlog sizes.
- TUI input latency: keystroke to painted frame.
- Session open on a long history — the TUI already virtualizes the
transcript (4c8e5de), so the cost is in the fold, not the render.
Not started. Filed because the milestone now contains speed claims that nothing
in the repository can check.
What is missing
BUILD_PERFORMANCE.mdcovers build speed. Nothing in the repository measuresruntime speed: no
benches/, no criterion / iai / divan inCargo.lock, no.perfcases, no budget constants, no gate that fails when a user path getsslower. Verified 2026-09-15.
So every claim about responsiveness in this milestone ("streaming is bursty",
"typing feels laggy", "make it smooth and fast") is currently unfalsifiable. The
stream-pacing change in
9b41aeaacwas reproduced and measured with a scratchharness that was thrown away; nothing keeps that measurement honest next month.
What good looks like, learned from deepseek-harness
refs/dsh/benchmarks/AGENTS.mdruns required repository-level performance gateswhose measured path crosses package ownership. The transferable parts, in the
order I would take them:
path. DSH has
session-open,agent-continuation,conversation-fold,active-stream-reconnect,terminal-io,long-session-browser.Sessions, user material, ambient repositories, or network services."
override performance budgets."
measurements to explain each verdict, including whether a budget uses a
median, minimum, absolute value, or ratio."
recorded separately from the shared CI time scale and variance headroom, and
the time scale is never applied to memory or to dimensionless ratios.
retained-memory sample; remove owned roots after failure as well as success.
active-stream-reconnectexists because a reconnect carrying a 100,000-delta reasoning prefix has to
fold without stalling — a specific shape, not a generic "streaming is fast".
The decision this needs first
A harness means a benchmark dependency, and criterion / iai / divan are all
absent from
Cargo.lock. Adding one is a dependency decision on a repositorycurrently working to reduce dependency surface (#6151), so it should be a
deliberate choice rather than a drive-by. The alternatives are a dependency-free
Instant-based harness in a child process, or adopting one of the above.Also unanswered: what wall-clock assertion can survive a busy developer machine,
and which user paths are actually gated versus merely measured. DSH answers both;
we have not.
First candidates, straight from this milestone
between a provider delta arriving and its text being visible, at several
backlog sizes.
transcript (
4c8e5de), so the cost is in the fold, not the render.Not started. Filed because the milestone now contains speed claims that nothing
in the repository can check.