Repository navigation
ci(browser): say which check failed, and rank failures across runs - #1009
Merged
Merged
Conversation
…ne's reports A flaky or failed browser script was reported by name only. The console lines, the warning annotation, a new error annotation for a script that fails twice, the job summary and summary.json now carry the first error line of its log and the first stack frame in scripts/.
…ipts across recent runs bun scripts/ci-flakes.ts reads recent ci.yml runs through gh and lists each script and failing site with how often it failed and how often it passed only on a second run.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Summary
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage |  |
Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files. (2 skipped: … | Write docstrings for the functions missing them to satisfy the coverage threshold. |
-
Check name Status Explanation Title check The title clearly summarizes both primary changes: improved browser failure reporting and ranking failures across CI runs. Description check The description includes the required Change and Validation sections. It explains the behavior, scope, test results, and unrun checks with sufficient detail. Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 4 files. (2 skipped: 2 unsupported.)
- Fix all pre-merge checks with AI
✨ Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
Since #1002 a browser script that fails is run once more and a pass on the second run is reported as flaky. Two things were still missing for whoever maintains the lanes.
The report did not say what failed. "FLAKY scripts/ui-regression.ts" names a script with 65 checks. The lane now reads the failed run's log for the first error line and the first stack frame in
scripts/, and puts both in the console line, the warning annotation, the job summary andsummary.json:A script that fails twice gets an
::error title=Browser script failed::annotation with the same, which it did not have. A log with no recognisable error is reported as before, by script alone.Nothing counted failures across runs.
bun scripts/ci-flakes.ts [--hours 24] [--repo owner/name] [--json]reads recentci.ymlruns throughgh, every attempt of each, and prints each script and failing site with how often it failed and how often it passed only on a second run, most frequent first. It changes nothing. A failed job's log is read; a passed job's only when it carries the flaky warning, so 30 hours of this repository (242 runs, 516 browser jobs) take about 100 seconds instead of more than five minutes.What it shows today, first rows of
--hours 30:No change to which scripts run, to the second run, to exit codes or to what gates a merge.
Validation
bun run typecheck: pass.scripts/ci-flakes.test.ts(new),pages-gate.test.ts(new cases forfailureSite),conventions.test.ts,changelog-fold.test.ts: 0 fail. WithfailureSitemade to return no site, 4 of those cases fail.BROWSER_SCRIPTS=scripts/deep-link-demo-regression.ts GITHUB_ACTIONS=true bun scripts/ci-browser-parallel.tsand a throwaway line that made that script fail:::warning, the job summary andsummary.jsonall carryscripts/deep-link-demo-regression.ts:10andAssertionError: scratch forced failure;::error title=Browser script failed::with the same;process.exit(1)with nothing printed, first run only: exit 0, reported flaky by script alone, no "null" or "undefined".bun scripts/ci-flakes.ts --hours 30against this repository: exit 0 in 103 s; listsscripts/ui-regression.tsas flaky for run 38490957637 and the pre-retry failures.--jsonparses. WithGH_TOKENset to a bad token it printsgh: Bad credentials (HTTP 401)and exits 1.