Repository navigation
Publish the complete coverage denominator instead of a scopeless standardized score #532
Description
Context
Split out of the Aave v4 postmortem (#523) so the remaining work is protocol-agnostic and
independently trackable. #523 is closed as the historical postmortem for
aave-v4-v056-gpt55-xhigh-invariant-only-20260811-full-timeout.
In that run all three coverage nodes reported 100%, and the final campaign LCOV passed the same
covg-eval check: 39/39 analyzed function ranges, zero missing. The number was internally
consistent and still misleading: the denominator was 39 Recon-Magic-selected functions across
nine files, which excluded Spoke.sol and LiquidationLogic.sol entirely — the exact production
paths the campaign needed to reach.
A conservative post-hoc normalization over byte-identical non-dependency src/** files put real
line coverage at 896/2,248 (39.86%), with LiquidationLogic.sol at 2/222 (0.90%) where both hits
were a library declaration and a constant, no function body. Native instruction count was flat at
76,177 from the first status update to the end of a 58-minute campaign.
Problem
Two distinct defects, both in how coverage evidence is published rather than how it is computed:
- Coverage artifacts do not publish their denominator. A consumer cannot tell which files,
functions, or ranges were analyzed, which were excluded and why, which are production source
versus test or dependency, or which material components scored zero. - A narrow selected-range score is presented as generic "standardized coverage". Without its
scope,100%reads as broad production coverage to every downstream consumer: the final
report, the report bundle, eval scoring, and any human deciding whether a campaign is
trustworthy.
Together these let a campaign publish 100% standardized coverage while every critical path
relevant to its own evaluation is untouched.
Expected behavior
Coverage artifacts and the final report carry the complete denominator, and no score is presented
without its scope.
Acceptance criteria
- Coverage artifacts record the analyzed denominator explicitly: included files, excluded
files with an exclusion reason, and the functions/ranges counted. - Coverage artifacts distinguish production source from test, harness, and dependency source,
and attribute each counted range accordingly. - Coverage artifacts enumerate material components with zero coverage rather than letting
them vanish from a narrow denominator. - Any published coverage percentage names its scope wherever it is rendered (artifact JSON,
report.md, report bundle, eval scoring inputs). A bare100%with no scope is rejected. - A production-source view and/or a declared-critical-path view is published alongside the
selected-range score, so the two cannot be confused. - The check is protocol-agnostic: no Aave-specific file or contract names in the
implementation or its gates. - Regression coverage for a campaign whose selected denominator excludes a declared critical
component: the narrow score still computes, and the artifact and report both disclose the
exclusion and the zero-coverage component.
Related: #523 (historical postmortem), #519, #524, #529, #530.
Activity
aviggiano commentedon Aug 14, 2026
v0.1.0 relationship: #532 and #590 are complementary release blockers, not duplicates. #532 makes the full coverage denominator and scope visible; #590 rejects a vacuous/empty standardized measurement even when tooling exits zero. The release needs both denominator publication and non-vacuity enforcement.