Skip to content

No runtime performance gate exists: nothing measures or defends the speed of the user path #6193

Description

@Hmbown

What is missing

BUILD_PERFORMANCE.md covers build speed. Nothing in the repository measures
runtime speed: no benches/, no criterion / iai / divan in Cargo.lock, no
.perf cases, no budget constants, no gate that fails when a user path gets
slower. Verified 2026-09-15.

So every claim about responsiveness in this milestone ("streaming is bursty",
"typing feels laggy", "make it smooth and fast") is currently unfalsifiable. The
stream-pacing change in 9b41aeaac was reproduced and measured with a scratch
harness that was thrown away; nothing keeps that measurement honest next month.

What good looks like, learned from deepseek-harness

refs/dsh/benchmarks/AGENTS.md runs required repository-level performance gates
whose measured path crosses package ownership. The transferable parts, in the
order I would take them:

  1. Organize by measured user path, not by package tree. One directory per
    path. DSH has session-open, agent-continuation, conversation-fold,
    active-stream-reconnect, terminal-io, long-session-browser.
  2. Fixed synthetic inputs from reviewed constants. "Never use recorded
    Sessions, user material, ambient repositories, or network services."
  3. Budgets are reviewed source constants. "Environment variables must not
    override performance budgets."
  4. Say how the verdict is reached. "Report enough raw and aggregate
    measurements to explain each verdict, including whether a budget uses a
    median, minimum, absolute value, or ratio."
  5. Separate expectations by machine. Reference-machine expectations are
    recorded separately from the shared CI time scale and variance headroom, and
    the time scale is never applied to memory or to dimensionless ratios.
  6. Fresh process, private root, bounded child for any wall-clock or
    retained-memory sample; remove owned roots after failure as well as success.
  7. One named benchmark per real defect class. active-stream-reconnect
    exists because a reconnect carrying a 100,000-delta reasoning prefix has to
    fold without stalling — a specific shape, not a generic "streaming is fast".

The decision this needs first

A harness means a benchmark dependency, and criterion / iai / divan are all
absent from Cargo.lock. Adding one is a dependency decision on a repository
currently working to reduce dependency surface (#6151), so it should be a
deliberate choice rather than a drive-by. The alternatives are a dependency-free
Instant-based harness in a child process, or adopting one of the above.

Also unanswered: what wall-clock assertion can survive a busy developer machine,
and which user paths are actually gated versus merely measured. DSH answers both;
we have not.

First candidates, straight from this milestone

  • Stream reveal throughput and latency: the per-beat advance, and the lag
    between a provider delta arriving and its text being visible, at several
    backlog sizes.
  • TUI input latency: keystroke to painted frame.
  • Session open on a long history — the TUI already virtualizes the
    transcript (4c8e5de), so the cost is in the fold, not the render.

Not started. Filed because the milestone now contains speed claims that nothing
in the repository can check.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions