fix(test): make test_browser_profile_coverage deterministic via DNS instrumentation#1183
Open
vringar wants to merge 2 commits into
Open
fix(test): make test_browser_profile_coverage deterministic via DNS instrumentation#1183vringar wants to merge 2 commits into
vringar wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes intermittent CI flakes in test_browser_profile_coverage by making the assertions depend on the crawl’s DNS instrumentation results (dns_responses) rather than on network-dependent domain set comparisons, so unexpected/failed resolutions no longer fail the test.
Changes:
- Enable
dns_instrumentfor the crawl used bytest_browser_profile_coverage. - Rework assertions to:
- require a
dns_responsesrow for each entered site (attempted resolution), - only require
http_requests+moz_placescoverage whendns_responses.addressesis present, - treat “only failures” (no addresses) as a valid/skip path.
- require a
- Add a sanity check that at least one entered site resolved successfully to ensure the test still exercises the positive path.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1183 +/- ##
==========================================
- Coverage 62.14% 62.12% -0.03%
==========================================
Files 40 40
Lines 3918 3918
==========================================
- Hits 2435 2434 -1
- Misses 1483 1484 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vringar
force-pushed
the
fix/profile-coverage-deterministic-dns
branch
4 times, most recently
from
9215b5b to
5646bee
Compare
…nstrumentation Reworks the profile-coverage check so it can no longer flake on network conditions and no longer passes vacuously. - Anchors each per-site profile assertion on the host Firefox actually resolved and received a response from (the dns_responses hostname), not the eTLD+1 of the URL we typed. This fixes the blogspot.com case, where the entered eTLD+1 is the degenerate '.blogspot.com' while the resolved host (www.blogspot.com) maps to a different registrable domain, and it tolerates redirects to a different eTLD+1. - Drops the tautological 'request was made' assertion. A dns_responses row with addresses comes from onHeadersReceived, which only fires after a response arrived, so the matching http_requests row is implied. The independent signal is profile coverage (moz_places), written by Firefox itself. - Strengthens the floor: requires a clear majority of TEST_SITES to both resolve and appear in the profile, so a broken crawl with a lost profile fails instead of degrading to 'at least one site resolved'. - Treats an empty addresses string as not-resolved (truthiness check). - Docstring/comments now describe what is actually guaranteed.
Also correct the docstring's description of the blogspot/public-suffix mechanism: the test does not rely on get_ps_plus_1 mapping www.blogspot.com to a real eTLD+1, but on applying the same get_ps_plus_1 to both the resolved host and the profile URLs.
vringar
force-pushed
the
fix/profile-coverage-deterministic-dns
branch
from
5646bee to
b47faa4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1163 —
test_browser_profile_coverageflaked intermittently when Firefox contacted an unexpected domain (e.g. a search partner such asbaidu.com) during startup. The old assertion compared the set of requested domains against the Firefox profile, which is inherently network-dependent.This restructures the test to be deterministic regardless of network conditions by keying every assertion off the DNS instrumentation (
dns_responsestable, merged via #1158/#1159).What changed
The test now enables
dns_instrumenton the crawl and, for each site inTEST_SITES:dns_responsesrow exists — proving the browser attempted to resolve it.addresses) → asserts the correspondinghttp_requestsandplaces.sqlite(moz_places) entries exist.errorand nulladdresses) → skips the HTTP/profile assertions for that domain. A timeout/NXDOMAIN is now a valid test path, not a failure.A final sanity check asserts that at least one entered site resolved successfully, guarding against the per-site loop silently doing nothing.
The previous network-dependent set-difference check (and its now-unused
json/crawl_historyhandling) is removed.Why this is deterministic
baidu.com(or any TEST_SITE) timing out no longer makes the test fail — it becomes a skipped branch. The only things asserted are invariants the crawl pipeline must uphold when DNS actually resolved: a resolved domain must have produced an HTTP request and must appear in the browser profile.Verification
Ran
test_browser_profile_coverageagainst Firefox 150 multiple times locally; all runs passed.