Skip to content

Gauntlet levels low/medium/full: gauntlet-plan.sh picks the pre-PR level from changed paths; merge stays full; miss-rate log #749

Closed

Description

@wasikarn

Follow-up to #735 (operator-approved 2026-10-30).

Why

Since #735, a push runs lint only, and the full gauntlet runs once, at merge. Before opening a PR, an agent has two choices today: nothing beyond lint, or the full gauntlet, which takes several minutes and queues on a shared lock. A middle level that runs only the tests a change can affect gives faster pre-PR confidence. The merge gate stays full.

Levels

Level Runs Used
low lint on changed files (scripts/_lib/gauntlet-lint.sh) every push (already the default)
medium low + harness-audit + only the tests mapped from the changed files an agent before opening a PR or after a validator fix
full the whole gauntlet every merge (merge-pr.sh, sync-pr-head.sh), not adjustable; or on request

Design

  1. scripts/gauntlet-plan.sh <base>..<head> (or <base> against HEAD) reads the changed paths. It maps them through a committed rules file and prints the level, the selected test files and one reason line per rule that fired.
    • Docs or *.md only → low.
    • A skill or agent file → medium + that skill's tests + audit.
    • hooks/gates/** → medium + that gate's tests.
    • Shared code (tests/_lib/, scripts/_lib/, scripts/run-gauntlet.sh, git-hooks/, hooks/ libs) → full.
    • A path no rule covers → full.
    • The highest level wins across files.
  2. scripts/run-gauntlet.sh --level medium --plan <base> (or an equivalent) runs only the planned tests plus validate, lint and audit. An agent may ask for a higher level than the plan suggests, never a lower one: a --level below the plan's is refused.
  3. The merge gate is unchanged: merge-pr.sh and sync-pr-head.sh always run full, and only a full unsharded run writes the pass cache (Push in under 30 s: fast pre-push, full gauntlet once at merge, pass cache by tree hash #735).
  4. Miss rate: on every merge-time full run, merge-pr.sh appends one JSON row to a local log under ~/.cache/mh/. The row holds the PR, the head, the plan's level and selected files, the failing files (from the gauntlet's failing: line), and miss = any failing file outside the selection when the level was below full. If the rate isn't close to 0, tighten the rules.

Done when

  • Tests cover the mapping:
    • docs-only gives low;
    • a gate change gives medium with that gate's tests;
    • a tests/_lib/ change gives full;
    • an unmapped path gives full;
    • mixed changes take the highest level.
  • --level below the plan's is refused.
  • A merge-time run writes one miss-log row, and the row says miss when a failing file lies outside the selection.
  • docs/reference/repo-gotchas.md and branching-model.md describe the levels.

Open

  • A sourced helper can create a dependency that a file-based mapping misses. That is why unmapped and shared paths fall back to full, and why the miss log exists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions