[Due for payment 2026-09-17] Client drops writes rejected with `jsonCode 503` instead of retrying with backoff

[melvin-bot]

Problem

When the backend sheds writes during instability, it answers with HTTP 200 and an app-level jsonCode: 503 — "This action is unavailable right now. Please try again shortly."

The client treats that as a final failure: failureData is applied, the request is removed from the persisted queue, and the user gets the red "couldn't be delivered" state on their message. No retry, no backoff. This is the one failure mode that explicitly asks the client to try again later, and it's the one that bypasses the retry path.

Why it happens

processHTTPRequest is what decides retry vs. final — SequentialQueue.process() only retries when the promise rejects. It throws for:

• non-ok HTTP statuses (500 / 502 / 504 / unknown)
• HTTP 429 (throttled)
• jsonCode 400 + duplicate record
• jsonCode 666 + already-created message
• jsonCode 666 + socket title (Auth down/timed out)

A jsonCode 503 in a 200 body matches none of these, so the promise resolves, the queue removes the request as processed, and failureData lands on the message. There is no 503 handling anywhere in src/libs/Network.

• src/libs/HttpUtils.ts
• src/libs/Network/SequentialQueue.ts

Observed in production

2026-08-31, ~16:30–16:37 UTC. Two comments were queued in a Chronos chat during backend instability.

1. The first hit a backend timeout (jsonCode 666). That rejects, so the client rolled back, backed off, retried once, and got a 200. Self-healed correctly.
2. The second sat behind it in the SequentialQueue for ~7 minutes. When it finally went out it came back jsonCode 503 and was dropped, with the error stamped on the message. The user had no retry affordance — only the X to clear it.

Proposed fix

Treat jsonCode 503 like the other service-interruption cases: throw from processHTTPRequest so SequentialQueue rolls back the optimistic data and retries through RequestThrottle with exponential backoff. If the retry cap is exhausted, fall through to today's behavior and apply failureData.

Worth confirming with the backend team that 503 is only ever returned for load shedding. If it can also mean "permanently unavailable for this action", the client needs to distinguish the two before retrying blindly.

Reported in Slack.

Issue Owner

Current Issue Owner: @mallenexpensify

[abdullahitijani]

Proposal

Problem

When the backend sheds writes during instability, it responds with HTTP 200 + app-level jsonCode: 503 ("This action is unavailable right now. Please try again shortly."). The client treats this as a final failure: failureData is applied, the request is removed from the persisted queue, and the message is stamped with the red "couldn't be delivered" state — no retry, no backoff. This is the one failure mode that explicitly tells the client to try later, and it's the only one that bypasses the retry path.

Root Cause

processHTTPRequest (src/libs/HttpUtils.ts) is what decides retry vs. final — SequentialQueue.process() only retries when the promise rejects. It throws only for:

• non-ok HTTP statuses (500 / 502 / 504 / 520)
• HTTP 429 (throttled)
• jsonCode 400 + duplicate record
• jsonCode 666 + already-created message
• jsonCode 666 + socket (Auth down/timed out)

A jsonCode: 503 in a 200 body matches none of these, so the promise resolves. The response flows through the SaveResponseInOnyx middleware (src/libs/Middleware/SaveResponseInOnyx.ts), where applyHTTPSOnyxUpdates (src/libs/actions/OnyxUpdates.ts:72) sees jsonCode !== 200 and applies failureData. SequentialQueue.process()'s success branch (src/libs/Network/SequentialQueue.ts:326) then calls endPersistedRequestAndRemoveFromQueue, dropping the request as "processed successfully." There is no 503 handling anywhere in src/libs/Network.

Fix

Treat jsonCode: 503 like the other service-interruption cases: throw from processHTTPRequest so the queue rolls back optimistic data and retries via RequestThrottle with exponential backoff (random 10–100 ms initial wait, doubling up to a 10 s cap, max 10 retries). If the retry cap is exhausted, the existing .catch() path already falls back to applying failureData.

Add a constant and a guard in src/libs/HttpUtils.ts:

// in ~/src/CONST/index.ts, JSON_CODE:
SERVICE_UNAVAILABLE: 503,

// in processHTTPRequest, after the existing jsonCode guards:
if (response.jsonCode === CONST.JSON_CODE.SERVICE_UNAVAILABLE) {
    throw new HttpsError({
        message: CONST.ERROR.EXPENSIFY_SERVICE_INTERRUPTED,
        status: CONST.JSON_CODE.SERVICE_UNAVAILABLE.toString(),
        title: 'Service unavailable',
        requestID: response.requestID,
    });
}

Using the same EXPENSIFY_SERVICE_INTERRUPTED message already thrown for 500/502/504/520 means the retry path (and FailureTracker) treats 503 identically to those cases with no extra wiring.

Result

• 503 load-shed responses are rolled back and retried through the existing exponential-backoff throttle — matching the self-healing already proven for 666 timeouts (the first comment in the linked report).
• If the retry cap is exhausted, behavior falls back to today's final-failure path — no regression.
• failureData/successData semantics are unchanged.

[melvin-bot]

@dukenv0307 Eep! 4 days overdue now. Issues have feelings too...

[dukenv0307]

still in review

[mountiny]

Merged