Skip to content

Publish the complete coverage denominator instead of a scopeless standardized score #532

Closed

Description

@aviggiano

Context

Split out of the Aave v4 postmortem (#523) so the remaining work is protocol-agnostic and
independently trackable. #523 is closed as the historical postmortem for
aave-v4-v056-gpt55-xhigh-invariant-only-20260811-full-timeout.

In that run all three coverage nodes reported 100%, and the final campaign LCOV passed the same
covg-eval check: 39/39 analyzed function ranges, zero missing. The number was internally
consistent and still misleading: the denominator was 39 Recon-Magic-selected functions across
nine files, which excluded Spoke.sol and LiquidationLogic.sol entirely — the exact production
paths the campaign needed to reach.

A conservative post-hoc normalization over byte-identical non-dependency src/** files put real
line coverage at 896/2,248 (39.86%), with LiquidationLogic.sol at 2/222 (0.90%) where both hits
were a library declaration and a constant, no function body. Native instruction count was flat at
76,177 from the first status update to the end of a 58-minute campaign.

Problem

Two distinct defects, both in how coverage evidence is published rather than how it is computed:

  1. Coverage artifacts do not publish their denominator. A consumer cannot tell which files,
    functions, or ranges were analyzed, which were excluded and why, which are production source
    versus test or dependency, or which material components scored zero.
  2. A narrow selected-range score is presented as generic "standardized coverage". Without its
    scope, 100% reads as broad production coverage to every downstream consumer: the final
    report, the report bundle, eval scoring, and any human deciding whether a campaign is
    trustworthy.

Together these let a campaign publish 100% standardized coverage while every critical path
relevant to its own evaluation is untouched.

Expected behavior

Coverage artifacts and the final report carry the complete denominator, and no score is presented
without its scope.

Acceptance criteria

  • Coverage artifacts record the analyzed denominator explicitly: included files, excluded
    files with an exclusion reason, and the functions/ranges counted.
  • Coverage artifacts distinguish production source from test, harness, and dependency source,
    and attribute each counted range accordingly.
  • Coverage artifacts enumerate material components with zero coverage rather than letting
    them vanish from a narrow denominator.
  • Any published coverage percentage names its scope wherever it is rendered (artifact JSON,
    report.md, report bundle, eval scoring inputs). A bare 100% with no scope is rejected.
  • A production-source view and/or a declared-critical-path view is published alongside the
    selected-range score, so the two cannot be confused.
  • The check is protocol-agnostic: no Aave-specific file or contract names in the
    implementation or its gates.
  • Regression coverage for a campaign whose selected denominator excludes a declared critical
    component: the narrow score still computes, and the artifact and report both disclose the
    exclusion and the zero-coverage component.

Related: #523 (historical postmortem), #519, #524, #529, #530.

Activity

  1. added this to the v0.1.0 milestone on Aug 14, 2026
  2. aviggiano commented on Aug 14, 2026

    @aviggiano
    CollaboratorAuthor

    v0.1.0 relationship: #532 and #590 are complementary release blockers, not duplicates. #532 makes the full coverage denominator and scope visible; #590 rejects a vacuous/empty standardized measurement even when tooling exits zero. The release needs both denominator publication and non-vacuity enforcement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions