Skip to content

Reject vacuous standardized coverage outputs even when tooling exits zero #590

Closed

Description

@aviggiano

Summary

Ultrafuzz should machine-reject standardized coverage results with an empty target map, a zero denominator, or no production-attributed sources, even when the coverage tools exit successfully.

The v0.0.16 coverage schemas can represent a typed blocker and reject an empty reported metric list, but the runtime does not validate the semantic contents of the underlying recon-generate coverage and covg-eval outputs. Correct rejection still depends on the model noticing that the result is vacuous.

Evidence from #531

The Aave v4 stateful-invariant-coverage node observed:

  • recon-generate coverage exited 0 and wrote a two-byte {} Magic coverage map.
  • covg-eval magic/ echidna/ --return-json exited 0.
  • The result reported full_coverage: true while functions_analyzed was 0.
  • The agent correctly rejected the result, recorded coverage-tooling-blocked, and left the standardized percentage null.

That was a good model decision, not a runtime guarantee. Another producer could turn the same successful exit codes or vacuous "full coverage" result into a reported percentage.

Expected behavior

Coverage acceptance must depend on a nonempty, production-attributed denominator rather than process exit status or a tool's boolean "full coverage" field.

The node may complete successfully with a typed coverage blocker; this issue does not require a tooling problem to fail the entire workflow. It requires the unusable measurement to be impossible to publish as valid coverage.

Acceptance criteria

  • Capture and validate the exact standardized-coverage tool outputs as typed durable evidence.
  • Reject an empty coverage target/map such as {}, regardless of exit code.
  • Reject functions_analyzed == 0 or an equivalent zero denominator.
  • Reject a result with no source attributable to configured production roots.
  • A rejected result cannot use target-met or below-target and cannot publish a coverage percentage.
  • Record a typed coverage-tooling-blocked or production-source-attribution-blocked outcome with the raw evidence paths.
  • Propagate the blocker into campaign/report comparability.
  • Add regressions for: empty map plus exit 0; zero analyzed functions plus "full coverage"; harness-only sources; and a valid nonzero production-attributed result.
  • Keep the gate adapter-friendly so coverage backends other than Recon can supply the same denominator and source-attribution facts.

Related: #531 and #532.

Activity

  1. added this to the v0.1.0 milestone on Aug 14, 2026
  2. aviggiano commented on Aug 14, 2026

    @aviggiano
    CollaboratorAuthor

    v0.1.0 relationship: #532 and #590 are complementary release blockers, not duplicates. #532 makes the full coverage denominator and scope visible; #590 rejects a vacuous/empty standardized measurement even when tooling exits zero. The release needs both denominator publication and non-vacuity enforcement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions