Skip to content

ci: add trustworthy coverage ratcheting toward 90% #10867

Description

@ReubenBond

Goal

Turn the existing coverage report into a trustworthy non-regression ratchet which can advance Orleans from 78.54% to above 90% without encouraging exclusions or denominator manipulation.

Coverage report: https://github.com/dotnet/orleans/runs/98679059570

Current behavior

  • The complete PR CI matrix merges coverage from providers, CodeGen, .NET 8/10, Linux, Windows, and macOS.
  • The repository summarizer de-duplicates physical source lines and reports 78.54% (99,056/126,115).
  • Raw Cobertura aggregation reports lower values because linked/shared source appears in multiple assemblies.
  • Branch coverage is not published canonically.
  • The coverage check currently reports success unconditionally and has no trusted main-branch baseline or changed-line gate.
  • Only loaded maintained source appears in the report, so compiled-but-unloaded source needs an explicit universe check.

Work

  • Produce trusted same-matrix main and PR coverage artifacts with SDK/tool/config fingerprints.
  • Define canonical line and branch de-duplication for linked/shared source.
  • Verify the exact expected matrix artifact set before merging.
  • Build a compiled-source manifest so missing assemblies/files cannot silently improve coverage.
  • Add report-only overall, per-package, changed-line, and changed-branch comparisons.
  • Calibrate normal variance, then fail regressions.
  • Allow baselines to move upward automatically; require reviewed scope changes for downward movement.
  • Fail unexplained changes to source/module filters, compile inclusion, exclusions, or coverage attributes.
  • Add tests for missing artifacts, duplicate lines, source-universe drift, exclusions, and check conclusions.
  • Raise the repository floor incrementally as coverage work lands, ending above 90%.

Acceptance criteria

  • Coverage comparison is reproducible between main and pull requests.
  • The check fails verified coverage regressions after a report-only calibration period.
  • Canonical overall line, branch, package, and changed-code metrics are published.
  • Missing test partitions or maintained source fail the coverage job instead of shrinking the denominator.
  • The ratchet cannot be lowered without an explicit reviewed change and rationale.
  • Final enforced repository line coverage exceeds 90%.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions