Skip to content

benchmark: maintainer action-space schema + proposal validator #9260

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 1 of that issue names the action space; this issue fixes it as validated types. Without a closed, validated action space, two agents' outputs are not comparable and the scorer has nothing stable to score against.

Requirements

  1. Closed action set: merge | close | request_changes | label | hold. Closed, not extensible-by-string — an unrecognized action is a validation error, never a silently-ignored or zero-scored entry.
  2. Per-action parameters, typed: label carries the label set; close carries a reason class; request_changes carries the blocking concern; merge and hold carry no parameters. Parameters that do not apply to the action are a validation error.
  3. Prediction horizon is part of the task, not the agent's choice: the agent proposes the action it expects the maintainer to take within N days of T (N fixed per benchmark, recorded in the task).
  4. Abstention is first-class: an agent may decline to predict a work unit. Abstention is not a wrong answer — it is recorded and feeds the coverage metric defined in Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215's scoring semantics, so a precise-but-narrow agent is distinguishable from a broad-but-noisy one.
  5. Pure validator that never throws for ordinarily-invalid input, returning { valid: true; proposal } | { valid: false; errors: string[] } — mirroring validateAttestationEnvelope in packages/loopover-engine/src/calibration/attestation-envelope.ts exactly (including rejecting unknown keys rather than ignoring them).

Deliverables

  • Action-space + proposal types and their pure validator in packages/loopover-engine/src/calibration/ (public, so a candidate agent author can validate locally before submitting)
  • 100% branch-coverage unit tests, including every rejection path
  • Exported from the engine barrel

Expected outcome

Any agent, written by anyone, can emit a proposal this repo will accept or reject deterministically — and the scorer downstream can assume well-formed input.

References

Sub-issue of #9216. Validator pattern to mirror: attestation-envelope.ts's validateAttestationEnvelope. Coverage/abstention semantics: #9215. Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions