Skip to content

test(bench): enforce resource limits and normalize evidence with BenchExec #957

Closed

Description

@DecisionNerd

Objective

Make BenchExec the Linux authority for full-process wall, CPU, RSS, I/O, limits, and termination evidence.

Requirements

  • Versioned definitions and GraphForge tool-info integration.
  • Capture pressure and I/O; pin cores/memory where supported.
  • Normalize BenchExec and GraphForge phase telemetry while preserving both and detecting disagreement.
  • Validate and sanitize evidence; fail closed on missing mandatory fields.

Acceptance criteria

  • Parent and children are included.
  • Timeout, OOM, exit, signal, harness, and correctness failures remain distinct.
  • Fixtures cover normalization and malformed evidence.

BDD completion scenario

Given a runner that spawns children and exceeds a limit
When BenchExec supervises it
Then the complete tree stops
And normalized evidence reports the authoritative typed outcome.

Activity

  1. DecisionNerd commented on Aug 29, 2026

    @DecisionNerd
    ContributorAuthor

    Closure evidence:

    • PR test(bench): make BenchExec the Linux resource authority #988 exact head ffa81f9f52f3450ca418d9835f441c3985eaf770 merged as f6b25ee9fc0a38128ed2ce2a2986920679eb32d2 with CI Gate and all required checks green and zero unresolved review threads.
    • The merged benchmark surface includes the versioned graphforge-certification-v1 definition, public certification tool-info adapter, pinned CPU/memory/time limits, full-process wall/CPU/RSS/I/O/pressure normalization, preserved GraphForge phase telemetry, disagreement reporting, closed schemas, and fail-closed mandatory-field validation.
    • Regression fixtures distinguish timeout, oom, exit, signal, harness, and correctness, reject malformed/non-finite/missing/contradictory evidence, and validate the versioned definition and public runner command.
    • The former blocker spike(bench): prove local ReFrame and BenchExec admission #954 is closed by PR test(bench): require strict native admission evidence #989. Its real unprivileged Ubuntu 24.04 BenchExec admission ran the supported runexec path, exceeded the wall limit, terminated the detached descendant, and captured aggregate RSS, CPU, I/O, and pressure. The independently validated sanitized artifact has SHA-256 737d6def57ac8df61dce3d3e511a9615794ca42a838454d6a974740c7ae44263.

    Together, the merged normalization contract and real process-tree admission satisfy the issue requirements and BDD outcome without weakening or simulating the authority path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTest coverage and testing infrastructure

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions