Skip to content

Fix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readiness - #52994

Merged
pelikhan merged 6 commits into
mainfrom
copilot/aw-debug-daily-ambient-workflow
Aug 16, 2026
Merged

Fix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readiness#52994
pelikhan merged 6 commits into
mainfrom
copilot/aw-debug-daily-ambient-workflow

Conversation

Copilot AI commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Daily Ambient Context Optimizer (and other docker-sbx + Copilot SDK workflows) failed repeatedly because the harness sent its first chat-completion request to the api-proxy sidecar before its per-provider listener was actually accepting connections, exhausting all retries on ECONNREFUSED and ending the run with zero agent turns (driver_exit).

Root cause

  • api-proxy reports /reflect healthy and passes key validation before the per-provider listener (e.g. http://api-proxy:10002) is actually accepting TCP connections.
  • The Copilot SDK driver's 3 retry attempts all land inside this startup window and fail identically, so the run dies within ~1 minute with no agent turns recorded.
  • This exact defect was previously diagnosed and fixed in Harden Copilot api-proxy startup: verify listener accept readiness and absorb first-request ECONNREFUSED #52619, but that PR was closed without merging, leaving the underlying race unresolved.

Fix

  • awf_reflect.cjs: added waitForProviderListenerReady(), which probes a real TCP connect (TLS handshake for https:// baseUrls) against a provider listener with a bounded timeout/retry cadence.
  • copilot_harness.cjs: in SDK multi-provider mode, gates on listener readiness for every unique provider baseUrl before the first request, and classifies a first-attempt ECONNREFUSED as a one-shot short-backoff retry instead of burning the full retry budget on a known-transient race.
  • Re-applied the prior fix's test coverage in awf_reflect.test.cjs / copilot_harness.test.cjs, resolving conflicts with unrelated test additions merged since.
// copilot_harness.cjs (SDK mode): require real listener readiness before first request
const readiness = await waitForProviderListenerReady({
  baseUrl: providerBaseUrlToProbe,
  timeoutMs: AWF_PROVIDER_LISTENER_READY_TIMEOUT_MS,
  logger: log,
});
if (!readiness.ok) {
  emitInfrastructureIncomplete(
    `api-proxy provider listener was not ready at ${providerBaseUrlToProbe} before first Copilot SDK request (${readiness.error}).`
  );
  process.exit(1);
}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.49 AIC · ⌖ 5.67 AIC · ⊞ 6.3K ·
Comment /souschef to run again


Run: https://github.com/github/gh-aw/actions/runs/31921676473> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.82 AIC · ⌖ 5.65 AIC · ⊞ 8.7K ·

Comment /souschef to run again

Copilot AI linked an issue Aug 16, 2026 that may be closed by this pull request
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Triage

  • Category: bug (WIP self-debug)
  • Risk: low
  • Score: 20/100 (impact 5, urgency 10, quality 5)
  • Recommendation: defer

Zero-diff WIP PR auto-created by an agent to debug a failed Daily Ambient Context Optimizer run. No files changed yet. Revisit once a diff lands or close if stale.

Generated by 🔧 PR Triage Agent · auto · 40.8 AIC · ⌖ 2.37 AIC · ⊞ 8.1K ·

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Debug daily ambient context optimizer workflow failure Fix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readiness Aug 16, 2026
Copilot AI requested a review from pelikhan August 16, 2026 00:33
@pelikhan
pelikhan marked this pull request as ready for review August 16, 2026 00:35
Copilot AI balanced review requested due to automatic review settings August 16, 2026 00:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds provider-listener readiness checks to prevent Copilot SDK startup races.

Changes:

  • Probes provider TCP/TLS listeners before SDK requests.
  • Adds one-shot ECONNREFUSED retry handling.
  • Adds readiness and retry tests.
Show a summary per file
File Description
actions/setup/js/awf_reflect.cjs Implements listener readiness probing.
actions/setup/js/awf_reflect.test.cjs Tests probe behavior and timeouts.
actions/setup/js/copilot_harness.cjs Gates SDK startup and handles refused connections.
actions/setup/js/copilot_harness.test.cjs Adds detection and retry tests.

Review details

  • Files reviewed: 4/4 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment thread actions/setup/js/copilot_harness.cjs Outdated
// provider died in the narrow window between the probe and the first request — retry
// once as a fresh run. Later attempts (attempt > 0) fall through to the generic retry
// handling below instead of taking this one-shot path.
if (attempt === 0 && isConnectionRefused && maxRetries > 0) {
Comment on lines +495 to +509
socket.once(readyEvent, () => {
clear();
// Remove the error listener before tearing down the socket: a successful handshake has
// already settled this probe as ready, and a late/trailing error (e.g. an abrupt RST)
// must not overwrite lastError or otherwise affect the already-settled result.
socket.removeAllListeners("error");
socket.destroy();
settle(true);
});
socket.once("error", err => {
clear();
lastError = getErrorMessage(err);
socket.destroy();
settle(false);
});
Comment on lines +1882 to +1886
function decideConnectionRefusedRetry(isConnectionRefused, attempt, maxRetries) {
if (attempt === 0 && isConnectionRefused && maxRetries > 0) {
return { action: "retry", nextDelayMs: FIRST_CONNECTION_REFUSED_RETRY_DELAY_MS };
}
return { action: "continue" };
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot this PR looks ready for a finishing pass.

Please:

  • address any open inline review feedback from copilot-pull-request-reviewer
  • refresh the branch if needed
  • run the pr-finisher skill before returning it for maintainer review

Run: https://github.com/github/gh-aw/actions/runs/31918518969

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.49 AIC · ⌖ 5.67 AIC · ⊞ 6.3K ·
Comment /souschef to run again

…or listener, test production helper

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

@copilot this PR looks ready for a finishing pass.

Please:...

Finishing pass done in 047e2a8:

  • SDK-mode scoping: the one-shot ECONNREFUSED retry now goes through a new exported shouldRetryFirstConnectionRefused({ copilotSDKMode, attempt, isConnectionRefused, maxRetries }), so CLI failures keep the generic retry policy (including --continue) instead of being forced into a fresh run.
  • Socket error listener: waitForProviderListenerReady() no longer calls removeAllListeners("error"). The probe settles as ready before destroy(), the error listener stays installed, and the handler ignores already-settled probes — so a late RST can't surface as an uncaught EventEmitter error. The timeout path settles before destroy() for the same reason.
  • Tests: the inlined retry-decision helper in copilot_harness.test.cjs was replaced with tests that call the exported production helper, including a CLI-mode guard case; awf_reflect.test.cjs now asserts the error listener is still installed at destroy() time.

Branch is up to date with main (0 commits behind). Prior CI on this branch had no failures — the completed CJS run on 839b739 was green, and the remaining runs are action_required (awaiting maintainer approval). Local validation (make agent-report-progress, incl. format, JS lint, typecheck, impacted tests, and make recompile lock-file sync) passes. CI on the new HEAD is unverified until a maintainer re-triggers it.

Copilot AI requested a review from gh-aw-bot August 16, 2026 01:20
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot this PR looks ready for a finishing pass.

Please:

Run: https://github.com/github/gh-aw/actions/runs/31921676473

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.82 AIC · ⌖ 5.65 AIC · ⊞ 8.7K ·
Comment /souschef to run again

@pelikhan
pelikhan merged commit 1f1a46a into main Aug 16, 2026
1 check failed
@pelikhan
pelikhan deleted the copilot/aw-debug-daily-ambient-workflow branch August 16, 2026 02:26
Copilot stopped work on behalf of gh-aw-bot due to an error August 16, 2026 02:26
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.87.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] Daily Ambient Context Optimizer failed

4 participants