Skip to content

Compare project coverage per Codecov flag - #954

Merged
timlichtenberg merged 2 commits into
mainfrom
tl/codecov-flag-project-status
Oct 5, 2026
Merged

timlichtenberg merged 2 commits into
mainfrom
tl/codecov-flag-project-status

Conversation

@timlichtenberg

@timlichtenberg timlichtenberg commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Description

codecov/project failed on #952 and #953 although both raised coverage in each flag (unit tests 92.14 to 92.15 and 92.16 %, nightly 95.45 to 95.52 and 95.53 %) and neither touched the lines it reported. On a main commit whose nightly session is carried forward, the combined Codecov report is not the union of its two flag reports: on a2c5c17, eight branch statements in interior_energetics count as hit or partial in the combined report and as missed in both the unit-tests and the nightly view. A PR head with real uploads then reads lower. Once main gets its own nightly upload the combined report agrees with the flags again (26f73aa: 11 mismatched lines before its nightly upload, 0 after).

This PR compares project coverage per flag, so each flag compares like with like:

  • codecov/project keeps its name and reads the unit-tests flag.
  • codecov/project/nightly is new and reads the nightly flag. Until the branch nightly has run, the head carries the nightly data of the base as it was at the head's first upload, so it can differ from the base if main got a newer nightly in between; the branch nightly settles it.
  • codecov/patch is unchanged and still reads all flags. Targets are unchanged.

A PR whose new code only the smoke or slow tiers cover now shows a lower codecov/project; codecov/project/nightly shows that code once the branch nightly has run.

Validation of changes

Checklist

  • I have followed the contributing guidelines
  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • My changes generate no new warnings or errors
  • I have checked that the tests still pass on my computer
  • I have updated the docs, as appropriate
  • I have added tests for these changes, as appropriate
  • I have checked that all dependencies have been updated, as required

The project status now compares the unit-tests flag and the nightly flag separately, each against the same flag on the base. The combined report on a main commit whose nightly session is carried forward does not equal the union of its own flag reports, so the combined comparison could fail on lines a PR never touched while both flags rose. Each flag compares like with like: the main baseline and the PR run the same unit selection, and nightly compares with nightly.

codecov/project keeps its name and now reads the unit-tests flag; codecov/project/nightly is new.
@timlichtenberg
timlichtenberg requested a review from a team as a code owner October 4, 2026 17:39
@codecov

codecov Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.54%. Comparing base (26f73aa) to head (7b99084).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #954   +/-   ##
=======================================
  Coverage   95.53%   95.54%           
=======================================
  Files         127      127           
  Lines       21160    21160           
  Branches     3595     3595           
=======================================
+ Hits        20216    20217    +1     
+ Misses        944      943    -1     
Flag Coverage Δ
nightly 95.53% <ø> (ø)
unit-tests 92.16% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@timlichtenberg
timlichtenberg merged commit 9b7bc6f into main Oct 5, 2026
41 checks passed
@timlichtenberg
timlichtenberg deleted the tl/codecov-flag-project-status branch October 5, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant