Skip to content

Tests: three speed checks miss on loaded macOS and Windows runners; make them measure without testing less #890

Open

Description

@LoopGlitch26

Three tests that check a speed bound miss it on the slower macOS and Windows CI runners, though the code under test is unchanged. They show up as red advisory jobs on unrelated PRs, which hides real failures. Each should keep checking what it means to (a decision under its deadline, linear time, a fast join) in a way a loaded runner can't break, without testing less.

Test Measures Missed on By
tests/test_graph.py::Records::test_graphify_join_indexes_five_thousand_file_nodes (L1521) wall clock (time.monotonic) of one _graphify over 5,000 nodes, < 1.0 s macOS py3.13 (#882's run) 1.026 s
tests/test_constraints.py::Grammar::test_hostile_text_is_read_in_linear_time (subtest "an except clause each", L488 onward) time.process_time, best of three, 4n < 8 × n Windows py3.13 (#878's run) [0.015625, 0.1875]
tests/test_grants.py::…::test_the_agent_own_grant_commands (L1008) each decide, a 64 KiB option flood included, within 1 s of wall clock macOS py3.13 (#878's run) 1.14 s (the same decide takes ~80 ms on a quiet machine, measured on #878)

A lead for the second one: both numbers are whole multiples of 1/64 s. On Windows, time.process_time ticks in 15.6 ms steps, so n measured one tick (0.015625), and the 8× ratio compares 12 ticks against 1.

Acceptance

  • Each test still fails on a real regression. Show it in the PR: slow the code under test on purpose (a sleep, or a quadratic loop in the linear-time case) and the test fails.
  • Each holds on a loaded runner. For example: measure enough work that the clock's resolution doesn't matter; compare against a baseline measured in the same test rather than a fixed number; use CPU time where the work is CPU-bound; or take the best of several runs. No bound is simply raised.
  • Two green runs of the macOS and Windows jobs on the PR.

Related: the Speed job's pattern budgets are #722's, so leave them out of this one.

Activity

  1. Siri-Durugapu commented on Nov 16, 2026

    @Siri-Durugapu
    Contributor

    Hi! I'd like to work on this issue as a contribution.

    I plan to investigate the three timing checks, make their measurements more robust across loaded CI runners without weakening their regression detection, and add/adjust tests to demonstrate that genuine performance regressions are still caught.

    I'll also verify the changes against the relevant macOS and Windows CI runs.

    Is this issue available for me to work on?

  2. LoopGlitch26 commented on Nov 16, 2026

    @LoopGlitch26
    MemberAuthor

    Thanks, @Siri-Durugapu, good to see you back after #703. I offered this one to @Seyamalam on #878 a few minutes before your comment, since they measured these misses there. If they haven't taken it by the end of 17 Nov (UTC), it's yours, and I'll assign it then.

  3. LoopGlitch26 commented on Nov 16, 2026

    @LoopGlitch26
    MemberAuthor

    @Siri-Durugapu, two you could start today if you'd like, either instead or while you wait:

  4. Siri-Durugapu commented on Nov 16, 2026

    @Siri-Durugapu
    Contributor

    Thanks for offering these alternatives! I'd like to take up #769 (Signer: measure the dev signer's cold start) while waiting for the availability of #890.

    I'll investigate the existing signer startup flow, understand the 10-second autospawn timeout, and implement the measurement and appropriate tests.

    Please assign #769 to me. Thanks!

  5. LoopGlitch26 commented on Nov 16, 2026

    @LoopGlitch26
    MemberAuthor

    Great, @Siri-Durugapu. GitHub only lets me assign you on an issue you've commented on, so please leave a short comment on #769 and I'll assign it there. Two at a time is fine: #769 now, and #696 once you've confirmed Pi runs for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions