Skip to content

Network-dependent tests (NuGetAssemblyResolver, MemexTemplateGenerator) present as phantom CI flakes; wall-clock deadlines misreport host/runner stalls #679

Closed

Description

@rbuergi

Found by a flake sweep across 60 of 62 test projects.

Two families of tests hard-fail when nuget.org is unreachable

  • MeshWeaver.Graph.Test.NuGetAssemblyResolverTest (3 tests) — FatalProtocolException: Unable to load the service index for https://api.nuget.org/v3/index.json
  • MemexTemplate.MemexTemplateGeneratorTest (7 tests) — Expected value to be 0 because generator failed → NU1900 … nuget.org

During the sweep these produced 10 failures that were purely environmental — a window where nuget.org was unreachable from this machine. On a clean network they pass.

Why file it

These will present as a phantom flake in CI whenever nuget.org has a bad minute: a red build, a failure message that points at NuGet internals rather than at "the network was down", and an engineer spending time attributing it. That is precisely the pattern this campaign has been unwinding — five "flakes" that were real defects, and the cost of the noise was that nobody trusted the signal.

Unlike the real defects, the right answer here is probably not a fix but a deliberate decision:

  1. Skip loudly when the feed is unreachable — a skipped test with "nuget.org unreachable" is honest; a red one is misleading.
  2. Pin to a local/offline feed so the tests exercise the resolver logic without live network.
  3. Accept and document — leave them network-dependent, but make the failure message say so, so triage takes seconds rather than minutes.

Whichever is chosen, the goal is that a network blip is never indistinguishable from a code defect.

Related hazard found in the same sweep

MonolithMeshTestBase's watchdog and several tests' polling deadlines (e.g. CommentDoneReactivityTest.WaitForReadOnlyMarkdown, 30 s wall-clock with 1 s polls) measure wall-clock. During the sweep the host machine slept (233 sleep events in pmset -g log), producing impossible durations — 5523 s and 7288 s against a 720 s cap — and a batch of spurious WATCHDOG-HARD failures. Every one passed on re-run under caffeinate.

On CI the equivalent is a runner stall: any suspension converts directly into a red build with a timeout message that describes the symptom and not the cause. Worth considering whether these deadlines should measure monotonic elapsed CPU/progress rather than wall-clock, or at least report suspected suspension when a deadline is exceeded by an implausible margin.

Activity

  1. added a commit that references this issue on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions