[FEATURE] Chaos test scenarios for transient failure recovery

[koydas]

🎯 Goal

Validate that the retry and error classification logic holds under simulated failure conditions — and prevent regressions as resilience features are added.

📍 Context

• Repo: autonomous-dev-loop
• Domain: Robustness validation
• Component: scripts/tests/anthropic_client.test.mjs, scripts/tests/groq_client.test.mjs, scripts/tests/llm_client.test.mjs, scripts/tests/smoke.test.mjs

🔗 Status (re-checked 2026-10-01 on main @ 4e499fb)

• Already done: network TypeError is retryable in both anthropic_client.mjs and groq_client.mjs, and both clients have network-error retry tests (retries when fetch throws a network error, rethrows the network error after exhausting retries). The scripts/lib/anthropic_client.mjs scope item and the matching criterion are checked below.
• Fallback tests depend on [BUG] callLLM provider fallback sends the primary provider's model, API key and URL to the fallback provider #162. Today callLLM passes the primary provider's apiKey, apiUrl and model to every provider, and the mocked fetch accepts any key and URL. A "503 then fallback" test written now would pass while the production fallback stays broken. Do the llm_client.test.mjs fallback cases after [BUG] callLLM provider fallback sends the primary provider's model, API key and URL to the fallback provider #162, and assert URL, auth header and model per provider.
• Overlap with [BUG] Groq 400 json_validate_failed causes silent fallback failure in code-generation #150 on scripts/tests/llm_client.test.mjs: do not generate both in parallel.
• Everything else in the scope below is still missing.

🚀 Description

The test suite is already comprehensive for happy-path and input validation scenarios. What is missing is targeted failure injection:

• groq_client.test.mjs covers 429 retry — good. No equivalent for 5xx.
• anthropic_client.test.mjs has no retry count verification tests (no check on exact number of fetch calls).
• llm_client.test.mjs does not verify that provider fallback is blocked on permanent 4xx errors.
• No test injects a network-level error (connection reset) to verify it is treated as retryable. (done)
• No test verifies timeout abort behaviour.
• smoke.test.mjs has no scenario for a run that fails transiently and recovers.

Dependencies resolved: #109 (error taxonomy) ✅ and #111 (retry utility) ✅ are both merged. scripts/lib/retry.mjs, scripts/lib/error_taxonomy.mjs, and scripts/lib/anthropic_client.mjs (with retryWithBackoff) are all in place.
Open dependency: #162 (per-provider config in the fallback), for the llm_client.test.mjs fallback cases.

🧩 Scope

In:

• scripts/tests/groq_client.test.mjs: add cases for HTTP 500, 502, 503, 504 — verify each is retried up to maxAttempts then throws.
• scripts/tests/anthropic_client.test.mjs: add cases for 500/503 retry (exact call count), 401 immediate throw (1 call only). network reset retried (done).
• scripts/tests/llm_client.test.mjs (after [BUG] callLLM provider fallback sends the primary provider's model, API key and URL to the fallback provider #162):
  • Verify fallback to secondary provider does NOT occur when primary returns 401.
  • Verify fallback DOES occur when primary returns 503 and exhausts retries, and that the secondary is called with its own URL, key and model.
• scripts/tests/retry.test.mjs:
  • Backoff sequence correctness (~200, ~400, ~800 ms before jitter).
  • Timeout abort (AbortError) counted as a retryable attempt.
  • Retry-After header respected.
• scripts/tests/smoke.test.mjs: add one scenario — first LLM call returns 503, second succeeds — verify output is correct and attempt counter is 2.
• scripts/lib/anthropic_client.mjs: fix network TypeError currently marked as retryable: false — should be retryable. (done: fetchErr.retryable = true)

Out:

• E2E tests requiring real network calls to Anthropic or Groq.
• Performance benchmarks.

🧪 Acceptance criteria

• When a 401 error occurs in callLLM, the secondary provider is never called and exactly one fetch call is made.
• When a network error (TypeError) occurs in callAnthropic, the retry count increases by 1 and the error is treated as retryable. (Covered on main.)
• When a timeout (AbortError) occurs, it is counted as a retryable attempt and the retry count increases by 1.
• When Retry-After header is present, the retry delay matches the header value (using fake timers — no real setTimeout).
• When all retry attempts are exhausted, the final error is thrown without additional fetch calls.
• A mocked 503 response on callAnthropic is retried and eventually returns the mocked success response (exact call count asserted).
• A mocked 503 response that persists across all attempts throws after exactly maxAttempts total calls.
• callLLM with a primary provider returning 503 and exhausting retries: secondary provider IS called, with its own URL, auth header and model. Requires [BUG] callLLM provider fallback sends the primary provider's model, API key and URL to the fallback provider #162.
• HTTP 500, 502, 503, 504 on callGroq are each retried up to maxAttempts then throw.
• Smoke: first LLM call returns 503, second succeeds — output is correct and attempt counter is 2.

⚙️ Constraints

• All tests must be deterministic — no flakiness from timing.
• No new test dependencies beyond what is already used in the suite.
• Tests must pass with node --test scripts/tests/*.test.mjs without any additional flags.

[github-actions]

🚫 Issue Validation Report

Field	Value
Status	INVALID — needs refinement
Score	85/100 [█████████░]

🚫 Blockers

The following issues must be resolved before this issue can proceed:

• Open dependency [BUG] callLLM provider fallback sends the primary provider's model, API key and URL to the fallback provider #162 required for llm_client fallback tests is not resolved or mocked

💡 Suggested Acceptance Criteria

Copy the block below and paste it into the issue description, then save. Saving will re-trigger validation automatically.

## Acceptance Criteria

- [ ] [ ] Verify that a 401 error in callLLM results in exactly one fetch call and no secondary provider invocation
- [ ] [ ] Ensure that a timeout AbortError is treated as retryable and increments the retry count by one
- [ ] [ ] Confirm that the Retry-After header value is used for the retry delay when present, using fake timers
- [ ] [ ] Validate that HTTP 500, 502, 503, and 504 responses in callGroq are each retried up to maxAttempts before throwing
- [ ] [ ] Test that a smoke scenario where the first LLM call returns 503 and the second succeeds results in correct output and exactly two attempts

---

Next step: Edit this issue, paste the acceptance criteria above (or write your own testable AC), and click Save. This will re-trigger the validation agent automatically.

[koydas]

📊 État actuel (audit du 2026-06-02)

Les dépendances bloquantes sont levées : #109 ✅ et #111 ✅ sont fermées.

✅ Déjà en place

• scripts/lib/retry.mjs — retryWithBackoff implémenté avec backoff exponentiel + jitter
• scripts/lib/error_taxonomy.mjs — classification TRANSIENT / PERMANENT / UNKNOWN
• scripts/lib/anthropic_client.mjs — utilise retryWithBackoff avec les codes 429/5xx retryables, 401 non-retryable
• scripts/lib/llm_client.mjs — interrompt le fallback provider si errorType === 'PERMANENT'
• scripts/tests/retry.test.mjs — couverture de base : succès, épuisement, retryable: false, waitMs, cap, jitter, propagation de l'erreur exacte

❌ Reste à faire

scripts/tests/groq_client.test.mjs

• HTTP 500, 502, 503, 504 : vérifier que chaque code est retenté jusqu'à maxAttempts, puis throw

scripts/tests/anthropic_client.test.mjs

• 503 retenté : vérifier le nombre exact d'appels fetch (pas seulement le throw final)
• 401 immédiat : vérifier que fetch n'est appelé qu'une seule fois (pas de retry)
• Network reset (TypeError) : vérifier qu'il est compté comme tentative retryable

scripts/tests/llm_client.test.mjs

• Primary = 401 (PERMANENT) → fallback provider non appelé
• Primary = 503 exhaustant les retries → fallback provider bien appelé

scripts/tests/retry.test.mjs

• Séquence de backoff : les délais approx. [200, 400, 800] ms (avant jitter)
• Timeout abort : une AbortError est comptée comme tentative retryable
• Retry-After header : un waitMs de 2000 sur une 429 avec Retry-After: 2

scripts/tests/smoke.test.mjs

• Scénario de récupération transient : premier appel LLM retourne 503, deuxième réussit — vérifier que la sortie est correcte et que le compteur d'appels est 2

Note : anthropic_client.mjs marque actuellement les TypeError réseau comme retryable: false, ce qui contredit le critère d'acceptance. À corriger en même temps que les tests.

---

Generated by Claude Code