Skip to content

spec: a 14-field result record and a survey of seven published results #12970

Closed

Description

@gHashTag

A benchmark result is usually published as a number, a unit, a workload and a part name. This issue asks for
a compiler-checked t27 spec of the record that should travel with such a result: a 14-field result record,
each field stating how completely one published result discloses it (stated, partial, absent, or unknown),
plus a survey of seven results that other people published, encoded as data and pinned by tests. The
record and the survey are already public as prose at https://t27.ai/#/passport; this issue only moves them
into a spec the compiler checks, so the page and the spec cannot drift apart.

Existing specs checked first (no second spec where one exists): specs/verified/receipt.t27 and
specs/verified/run_record.t27 own the shape of t27's own run receipts, and specs/claw/lab_receipt.t27 and
specs/tri/lab/receipt.t27 own lab receipts. None of them describes how completely a published third-party
result discloses its conditions, so a new spec is the right home; it reads nothing from them and restates
nothing of theirs.

Scope: the record, not the measurement. The spec holds no metric definitions and no thresholds. Nothing is
published by this issue: the work stays on a local branch until the owner clears it.

Boundary

  • specs/benchmarks/result_record.t27
  • .trinity/seals/benchmarks_BenchmarksResultRecord.json

User Scenarios

  • Given the spec file when a maintainer runs its tests then the survey tally, the per-result
    counts and the anchoring counts are checked by the compiler instead of by eye.
  • Given a result that leaves one field absent when the comparison rule is asked whether a comparison
    against it is supported then the answer is no, while the result itself stays publishable.
  • Given one cell of the survey flipped to another state when the tests run then the tally test
    fails, so a silent edit of the data is caught.

Requirements

  • FR-001: The spec MUST define the 14 fields of the record and a per-field cell state of stated, partial or
    absent, and MUST keep an unknown (not measured) cell a distinct state that is never coerced to absent or
    stated.
  • FR-002: Each field MUST carry its anchoring (a case that still stands, a count that was withdrawn, or the
    argument alone), and the tests MUST pin the three group sizes.
  • FR-003: The spec MUST encode seven published results as data, each with its headline figure and its primary
    source as plain strings, and the tests MUST pin the cell tally, the absent count of each result, and the
    facts about the control result.
  • FR-004: The spec MUST encode the rule that a result missing a field stays publishable but a comparison
    against it is not supported, as a small pure function with a positive test and a negative control.
  • FR-005: The tests MUST include negative controls: a flipped cell makes the tally test fail, and the
    spec MUST state that one count was withdrawn without repeating the withdrawn figure.
  • FR-006: A comment in the spec MUST say that the three cases behind the record are not from neuromorphic
    hardware and that FPGA figures are post-route estimates from the tools, not silicon measurements.
  • FR-007: The spec MUST use test blocks for the cell sweep (heavy loops in invariant exhaust the Zig
    comptime branch budget), MUST be hand-written t27 only, and MUST compare real-valued claims by integer
    arithmetic or a stated tolerance.
  • FR-008: The spec MUST carry a provenance block as plain strings and integers: the page the survey was read
    from, the git revision of the source files that hold it (a revision that is not known is said so in a
    comment, never invented), the date it was read, the number of results, the number of cells and a spec
    version. The tests MUST check that the result and cell counts match the encoded data, and that no cell is
    stated for a result that has no source reference. Each of those rules MUST have a negative control that
    fails, and each failing check MUST name the invariant it protects. The spec MUST NOT carry a checksum of its
    own file: the seal carries it.

Success Criteria

  • t27c test-report specs/benchmarks/result_record.t27 exits 0 with 0 failed tests and 0 vacuous tests.
  • t27c check specs/benchmarks/result_record.t27 exits 0.
  • t27c seal specs/benchmarks/result_record.t27 --verify prints "all hashes MATCH" after the last edit.
  • git diff --stat origin/master...HEAD lists at most 2 files, all inside the Boundary above.
  • The spec file contains 0 lines of non-ASCII text and 0 mentions of the withdrawn count.

Activity

  1. gHashTag commented on Nov 17, 2026

    @gHashTag
    CollaboratorAuthor

    Spec created: specs/tri/result_record.t27 (typecheck OK, 1 test passes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions