[chaofengw-nv]

Background

Between Oct 16 and Oct 29, 32 Dev Community CI runs (on ci/developer at 3a6a6562d) failed on Brev control-plane errors before any test ran. None of them was caused by the PR under test:

When (UTC)	Runs	Error
Oct 19 09:31 – Oct 21 01:11	21	Brev catalog must contain exactly one requested instance type
Oct 22 19:52 – 20:52	5	brev login returned 503 Service Unavailable
Oct 22 19:06, 19:14	2	SSH certificate service unavailable during the test
Oct 27 14:21, 23:39	2	Brev cleanup request is unavailable; organization check failed
Oct 20 16:04	1	brev login received a mismatched TLS certificate
Oct 17 04:54	1	Brev environment setup failed

brev login, the organization check and the catalog search each fail the run on their first error. The catalog error also does not say whether the type was missing or listed more than once, which need different fixes. Example runs: 39451003914 (catalog), 39847833005 (login 503), 39454247954 (TLS), 40603625522 (organization check).

Exit Criteria

• brev login in the GPU and cleanup jobs survives a few minutes of Brev unavailability.
• The organization check and catalog search, which run before the create request, are retried when Brev reports them unavailable. A retry never submits a second create.
• Deterministic failures fail immediately as before: a rejected credential (for example HTTP 401), a catalog type that cannot provide the requested disk, or a provider that does not match the profile.
• A persistent catalog mismatch reports how many offers Brev returned and from which provider.
• Non-goals: automatic re-dispatch of runs that exit 75 (coordination runs from main), a Nebius fallback when AWS has no capacity (it changes the GPU from L4 to L40S), and failures after the VM exists.

Implementation

• .github/workflows/community-ci.yml: both Log in to Brev steps retry five times, 30, 60, 90 and 120 seconds apart, then fail with an annotation.
• tools/brev_exec.py:
  • New PreflightUnavailable(ProvisionError) for read-only lookups that failed before the create request.
  • _allocation_organization raises it for HTTP 429 and 5xx. Other non-200 responses keep raising ProvisionError and are not retried.
  • _select_instance raises it when the search fails, returns invalid JSON, or lists the requested type zero or several times. The message now includes the number of matching offers, their providers and the catalog size, for example found 2 g6.4xlarge offers from aws, aws among 2.
  • New _preflight retries those lookups and InventoryError up to PREFLIGHT_ATTEMPTS = 4 times, 60, 120 and 180 seconds apart, within the provisioning deadline. provision() uses it for the organization check and the catalog search, which both run before create_started is set.
• .github/community-ci.md: documents the new retries next to the existing create-429 policy.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

All commands ran on CPU with CUDA_VISIBLE_DEVICES='' and PYTHONPATH=core/builder:apps/benchmark:..

• python -m pytest -q -n 8 tools/tests/test_community_gpu_ci.py tools/tests/test_community_ci.py tools/tests/test_community_gpu_lease.py: 1249 passed, 1 skipped.

• New tests in tools/tests/test_community_gpu_ci.py:

  • a catalog that is empty and then duplicated is retried, with the expected delays, before the only create;
  • a persistent empty or duplicated catalog reports its contents after four searches and never creates;
  • an unavailable organization check is retried before allocation;
  • HTTP 503 and 429 are retryable and 401 is not;
  • a disk mismatch is still searched only once.

  With tools/brev_exec.py reverted, 7 of these cases fail; only the disk-mismatch check passes.

• python -m pytest -q -n 8 tools/tests/test_architecture.py tools/tests/test_family_impact.py tools/tests/test_public_source_hygiene.py tools/tests/test_new_ci.py tools/tests/test_pr_metadata.py: 154 passed.

• ruff check and ruff format --check on the changed Python files: clean. python tools/legal_headers.py --check: 0 findings. git diff --check: clean.

• The Log in to Brev script, extracted from the workflow and run with a stub brev and sleep: two failures then success exits 0 after sleeping 30 and 60 seconds. Five failures exit 1 after sleeping 30, 60, 90 and 120 seconds.

Hardware, Environment, and Revisions

Base ci/developer at 3a6a6562d. Python 3.12 on a Linux x86_64 host; no GPU was used.

Not Run / Remaining Gaps

• No live Brev run. The retries are exercised against the existing FakeBrev and FakeCleanupAPI fixtures and a stub CLI.
• No actionlint run; the workflow file was parsed with PyYAML.
• The retries cover outages of minutes: up to about 5 minutes for login and 6 minutes for the lookups. The 40-hour catalog outage and the 1-hour login outage would still fail, now with a clearer error.

Contributor Self-Review

• I have completed a self-review of this change.

Notes For Future Readers

• Review order: _preflight and the two call sites in provision(), then _select_instance and _allocation_organization, then the workflow and the tests.
• --attempts 1 and the one-allocation rule are unchanged; the new retries stop before create_started is set.
• The same Oct 16–29 survey found two problems this PR does not address. 67 failures were Docker OOMKilled on the 64 GiB host profile. Dependency-layer failures for NeMo families came from virtualenv==21.14.6 in the shared GPU base lock, which needs packaging>=26.3, while NeMo's lightning<=2.4.0 needs packaging<25. Both need separate changes.

Risk level

• Low
• Medium
• High

Retries only repeat read-only lookups and an idempotent login, and only before any create request. The worst case is that a run reports the same infrastructure failure a few minutes later.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/TensorRT-Model-Connect
• Branch: ci/developer
• File: .coderabbit.yaml
• Commit: 3a6a656
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 663ee8ae-a657-417c-bdfc-baabe520a5b6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}