Repository navigation
Fix three flaky CI tests: demo dev-signer start, sensor event time, netserver log - #766
Merged
Merged
Conversation
…etserver log - record_demos --check starts every demo at once; on py3.9 Linux a dev signer's first start then outlasted the 10 s a client waits for one it starts (autospawn.Hung, pid None). The script now starts the dev signer of its demos and waits until it publishes endpoint.json. - test_sensor's event time was an hour after import; Windows CI reached the file more than an hour after collection, so the runs registered after their events. It is now a fixed 2099 time. - test_netserver slept 0.3 s for the server's traceback, which the connection's deadline can outrun; it now waits until the server is done with each connection. Each fix has a regression test that fails on main. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…running Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three tests failed CI at random on unrelated pull requests. Each is a test bug, now fixed at its cause, with a regression test that fails on main.
How often, and where
Read from the logs of the 165 failed test jobs among the last 300 CI runs (2026-11-06 07:10 to 2026-11-12 14:09 UTC):
test_demos.py::Demos::test_each_demo_verifies_and_runs_as_recordedtest_sensor.py, the same 4 tests each time (TestImportx3,TestFalcoImport)test_netserver.py::SilentClient::test_a_cut_connection_logs_no_traceback_but_a_handler_error_does1. The demo check called a dev signer that was still starting hung
All four failures are
tracekit.sdk.autospawn.Hung: a dev signer (pid None) holds .../signer.lock but does not answer. Three came from guardrails and one from completeness, the demos that run on the auto-started dev signer.pid Nonemeans the signer had taken its lock but had not yet publishedendpoint.json: it was still starting when the client's 10 s (autospawn.ensure) ran out.It was slow, not stuck. The failing
--checkruns took 18.1 to 19.0 s in all; passing ones on py3.9 Linux took 11 to 19 s. A signer that never served would have added up to 10 s more, becausetracekit downwaits that long for its pid.Why it was slow:
--checkruns every demo at once, in a thread pool of cpu+4 = 8 on a 4-vCPU runner. Each demo starts several Python processes (the demo, its signer, CLI calls), so about nine signers start together, and py3.9 is the slowest interpreter. The failures began 10 hours after #605 grewdev.yamlfrom 31 to 163 lines and the policy engine. That is timing only: a signer's start time wasn't measured here.Fix, in
scripts/record_demos.py: for the dev signer's demos, the script starts the signer itself, as autospawn would. It then waits until the signer publishesendpoint.json(written once it serves) or exits.story.Signeralready waits the same way for a configured signer'sserving online. The demo then finds the signer running. The 10 s in autospawn stays: a hook that gets no answer must still fail closed within its harness's timeout.tests/test_signer_dev.pystill covers autospawn from cold, end to end.Regression test: a
sitecustomizemakes the dev signer's start 11 s slower once it holdssigner.lock. On main the completeness demo then fails with CI'sHung(pid None); here it passes.2. The sensor tests' event time ran out an hour after import
TIME, the time of the tests' sensor events, was "an hour from now" at import, so at collection. The signer gives an event to the run registered by its time. A test that runs more than an hour after collection registers its run afterTIME, and its events go to no run:[] != [('exec', 'call_7', False), ...].The failing Windows jobs took 69 to 72 minutes, and all 6 reached
test_sensor59.8 to 61.1 minutes after collection. Of 135 sampled Windows jobs, the 129 that passed reached it at most 58.7 minutes after.Fix:
TIMEis a fixed timestamp in 2099. Regression test: a run registered two hours from now still takes the event (the signer's clock is patched, astests/factories.pydoes). On main it fails with0 != 1.3. The netserver test read the log before the server had written it
The connection's 0.3 s deadline keeps running while the server thread writes a handler's traceback. A client can therefore see the connection cut before the traceback is written, and the test then slept a fixed 0.3 s. CI captured only the traceback's first line.
Fix: the test waits until the server is done with each connection. Its
shutdown_requestruns afterhandle_error. With the traceback written 1 s late, main's test fails with CI's message, and this one passes.Stability
On macOS, with another agent's full test run and my own loops loading the machine:
test_sensor.pyandtest_netserver.py, 20 runs on Python 3.11: 20 of 20 passed (51 tests each).test_demos.py, 20 runs on 3.11: 20 of 20 passed. Every demo runs there, including all three on the dev signer.time.monotonic()starts near 0 in each process, and an import sends no heartbeat in its process's first 30 s. Sotest_sensor.pyrun alone on macOS py3.9 fails on main too; here it runs after the other two files. CI's long full run is never under 30 s by then.make test-ts lint build test: passed. test-ts: 162 tests, 0 failed. Lint and build clean. pytest: 4,973 passed, 394 skipped, 10 xfailed, in 29:52.Not checked
The Windows causes are argued from the code and CI logs; nothing ran on Windows. The demo script's wait relies on
endpoint.json, which Windows' TCP dev transport writes atomically after it binds, as the Unix socket path does after binding.