[Due for payment 2026-07-28] [Onyx] Investigate `UnknownError: Connection to Indexed Database server lost.` storage error

[fabioh8010]

Coming from #87844.

Summary

Let's investigate the following Onyx storage error: UnknownError: Connection to Indexed Database server lost., which accounts for 637,821 (19.0%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.

Platforms	Top Methods
Desktop Safari (558K), iOS Safari (72K), iOS Chrome (8K)	setWithRetry (578K), multiSetWithRetry (34K)

Retries Exhausted	Evictions	No Eviction	Quota Checks	>= 95% Full	Mean % Full	Quota Check Failed
106,394	0	0	0	—	—	0

• Safari/WebKit-exclusive — the IDB connection drops unexpectedly
• No eviction or quota involvement — not recognized as a capacity error, so Onyx retries immediately without eviction
• 106K exhausted all retries (16.7% of this error) — immediate retries don't help because the connection itself is gone

Action Items

• Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — is this caused by tab backgrounding, memory pressure, or Safari's aggressive IDB cleanup?
• Consider adding retry backoff for this error. Immediate retries are ineffective when the connection is gone — a delay could allow the browser to re-establish it. (being handled here)
• Consider detecting when the IDB connection is lost and reopening it before retrying
• Consider adding a visibilitychange listener to proactively reconnect when the tab returns to foreground

---

Issue Owner

Current Issue Owner: @fabioh8010

[leshniak]

I'll investigate this too.

[leshniak]

Investigation Report

Note: This is an AI-assisted analysis based on Sentry, VictoriaLogs and codebase data. The hypothesis (tab backgrounding as root cause) and recommendations below are preliminary and will be confirmed/refined in the next steps with deeper Fullstory session analysis and targeted VL queries.

Error Overview

Metric	Value
Error	UnknownError: Connection to Indexed Database server lost. Refresh the page to try again
VL retry-level logs (7d)	121,645
Unique sessions (7d)	21,609
Retries exhausted (7d)	21,071 (97.5%)
Sentry events (30d / 90d)	275 / 884
Sentry unique users (30d)	179
Sentry issue	APP-EY (open since 2025-11-04)
Platforms	100% WebKit (Safari, Mobile Safari, Chrome iOS, Edge iOS, DuckDuckGo, WKWebView)

The 2,300x gap between Sentry (275) and VL (121K) is expected — Sentry captures only the unhandled rejection that bubbles to window.onunhandledrejection, while VL captures every Failed to save to storage retry attempt (up to 5 per operation).

Hypothesis: Safari Kills IDB Connections for Backgrounded Tabs

Sentry breadcrumbs show a consistent pattern across sampled events:

1. App loads normally, IDB works, user is active
2. Multi-hour gap in breadcrumbs — tab was backgrounded or device was sleeping
3. Tab returns to foreground → reconnection burst fires (version.json, Ping, PusherPing, ReconnectApp)
4. ReconnectApp response triggers Onyx writes → IDB connection is already dead → UnknownError

For example, in session #1 (Sentry event), the breadcrumbs show a 5h22m gap between last activity (05:30 UTC) and the error (10:52 UTC). After the gap: version.json check, Ping, PusherPing, ReconnectApp — all API fetches succeed (status 200), but the subsequent Onyx IDB writes fail because the connection was severed during the background period.

Safari/WebKit aggressively terminates IDB connections for backgrounded tabs to conserve resources. When the tab comes back, the JavaScript context is still alive but the underlying IDB connection to the browser's database process has been severed. All subsequent IDB operations fail immediately.

Sampled Sessions

#	Date (UTC)	Browser	Page	Sentry	Fullstory
1	2026-05-15 10:52	Edge Mobile 148.0 / iOS 26.4	/home	f8a3570f	Session
2	2026-05-15 02:57	Safari 18.7.5 / macOS	/search	13893db5	Session
3	2026-05-13 18:26	Safari 26.4 / macOS	/e/{reportID}	599726b9	Session
4	2026-05-13 16:25	Safari 18.3.1 / macOS	/onboarding/purpose	6ef2cf74	N/A
5	2026-05-13 15:11	Safari 26.1 / macOS	/home	adb622cc	Session

All sampled events show lowMemoryThreat: false and handled: no (unhandled rejection via onunhandledrejection). No stacktrace is available — the error originates from IDB internals and surfaces as a promise rejection.

Evidence

1. Retries Are Completely Futile

Retry attempt distribution (VL, 7d):

Attempt	Count
1/5	19,979
2/5	20,097
3/5	20,152
4/5	20,312
5/5	21,035

Almost perfectly flat. Every operation that fails attempt 1 also fails 2, 3, 4, and 5. The connection is permanently dead within the retry window. Immediate retries (current behavior) cannot help because there is no delay or reconnection logic — the same dead connection is reused.

97.5% of sessions that encounter this error exhaust all 5 retries, resulting in silent data loss (retryOperation resolves with Promise.resolve() — the caller never knows).

2. Bursty — Few Sessions Generate Massive Log Volume

Top hourly spikes (VL, 7d, UTC):

Time (UTC)	Count	Likely scenario
May 14 18:00	3,992	Single user's tab returns, triggers ReconnectApp write storm
May 14 19:00	2,708	Continuation of same burst
May 12 22:00	2,278	Another large session
May 11 20:00	2,055	Another large session

A single backgrounded tab returning can generate thousands of failed retry attempts as ReconnectApp triggers a cascade of Onyx writes (setWithRetry and multiSetWithRetry) against a dead IDB connection — consistent with the reconnection burst pattern observed in session #1's breadcrumbs.

Average: ~5.6 retry entries per session, but distribution is heavily skewed — some sessions produce hundreds of entries.

3. 100% WebKit — No Other Browser Engine Affected

Sentry breakdown (30d):

Browser	Events	Unique Users
Mobile Safari	126	78
Safari (desktop)	93	63
Chrome iOS	17	10
Edge iOS	14	10
DuckDuckGo (iOS + macOS)	14	9
Other (Google app, OktaMobile, WKWebView)	11	9

All browsers in the list use WebKit under the hood (Chrome/Edge/DuckDuckGo on iOS are required to use WebKit by Apple's App Store policy). Zero events from Chromium or Gecko engines. This is visible across the sampled sessions — ranging from Edge Mobile on iOS (session #1) to desktop Safari 18.x-26.x (sessions #2-#5).

OS distribution: macOS (360 events), iOS (297+ events). Both desktop and mobile Safari are affected.

4. Affects Landing Pages During Reconnection

Top pages (Sentry):

Transaction	Events
/search	175
/home	156
/inbox	36
/settings	11

These are the pages a user lands on when returning to a backgrounded tab — exactly where ReconnectApp fires. The sampled sessions confirm this: /home (#1, #5), /search (#2), /e/{reportID} (#3), /onboarding/purpose (#4).

5. Not Memory or Storage Related

• lowMemoryThreat: false in all 5 sampled Sentry events
• Zero eviction events, zero quota checks associated with this error (from #87844 cross-reference)
• Error is not classified as a capacity error in Onyx, so the eviction flow is never triggered

6. Broadly Distributed Across Users and Releases

• Sentry: 179 unique users in 30d, top user has 9 events, most have 1-3
• Spread evenly across many app releases — not a regression from a specific version
• Safari versions affected: 18.x through 26.x — this is a long-standing WebKit behavior, not version-specific

7. Operations That Fail

VL breakdown by onyxMethod (7d):

Method	Count	%
setWithRetry	87,389	74%
multiSetWithRetry	30,637	26%

Current Handling in Onyx

• Error classification: This error is NOT in STORAGE_ERRORS (which only contains quotaexceedederror and database or disk is full), so it is treated as a generic error
• Retry behavior: retryOperation retries immediately 5 times with zero delay, reusing the same dead connection
• Failure mode: After 5 retries, retryOperation resolves with Promise.resolve() — silent data loss
• No reconnection logic: Unlike the backing store healing mechanism (PR #780) which drops the cached dbp and reopens, this error does not trigger any reconnection attempt
• onclose handler exists (createStore.ts:30-33): clears dbp when the browser fires onclose on the IDB connection — but Safari may not always fire this event before the connection is dead, leaving a stale dbp cached

Preliminary Recommendations

These are initial suggestions based on the analysis above. They will be validated/refined after confirming the hypothesis with deeper Fullstory session reviews and targeted VL queries.

1. Detect this error class — add "connection to indexed database server lost" to a new IDB_CONNECTION_ERRORS constant in OnyxUtils.ts (already specced in the retry-backoff design doc)
2. Reconnect before retry — when this error is caught in executeTransaction (similar to backing store heal at createStore.ts:133-142), drop the cached dbp and open a fresh IDB connection before retrying. This is the critical missing piece — without it, retries and backoff are both useless.
3. Retry backoff — handled separately in #87782, but this error specifically needs the reconnect step before any backoff delay matters
4. visibilitychange listener — proactively detect tab foregrounding and test/refresh the IDB connection before Onyx writes land, preventing the error from occurring at all
5. Consider NON_RETRIABLE_ERRORS applicability — if the reconnect-and-retry strategy doesn't work (i.e., the IDB server is truly gone and cannot be reopened), this error should be added to NON_RETRIABLE_ERRORS to avoid generating 5x futile retry logs per operation

Open Questions (next steps)

• Session recovery: After the IDB connection drops, does it EVER recover within the same session? Or is every subsequent write also failing until page refresh?
• Browser version correlation: Is there any Safari version that handles this better?
• Fullstory confirmation: Review the sampled Fullstory sessions above to visually confirm the tab backgrounding pattern

[leshniak]

Final Investigation Report

Error Overview

Metric	Value
Error	UnknownError: Connection to Indexed Database server lost. Refresh the page to try again
VL retry-level logs (7d)	121,645
Unique sessions (7d)	21,609
Retries exhausted (7d)	21,071 (97.5%)
Sentry events (30d / 90d)	275 / 884
Sentry unique users (30d)	179
Sentry issue	APP-EY (open since 2025-11-04)
Platforms	100% WebKit (Safari, Mobile Safari, Chrome iOS, Edge iOS, DuckDuckGo, WKWebView)

The gap between Sentry (275) and VL (121K) is expected — Sentry captures unhandled rejections that bubble to window.onunhandledrejection (plus logger integration and sampling), while VL captures every Failed to save to storage retry attempt (up to 5 per operation). The Sentry events likely originate from initialization paths (getAll, multiGet, getAllKeys) and logger integration rather than from the retry chain itself, which resolves with Promise.resolve().

Root Cause: Safari Terminates IDB for Backgrounded Tabs

Safari/WebKit aggressively terminates IDB connections for backgrounded tabs. When the tab returns, the JavaScript context is still alive but the underlying IDB connection is severed. This is a documented WebKit behavior (WebKit #197050, WebKit #201483).

Sentry breadcrumbs and Fullstory session replays confirm the pattern:

1. App loads normally, IDB works, user is active
2. Multi-minute to multi-hour gap — tab backgrounded or device sleeping
3. Tab returns to foreground → reconnection burst fires (version.json, Ping, PusherPing, ReconnectApp)
4. ReconnectApp response triggers Onyx writes → IDB connection is dead → UnknownError

Caveat: Tab backgrounding is the primary trigger, but not the only one — WebKit networking process crashes and memory pressure can also sever the IDB connection without a clean background/foreground cycle.

Sampled Sessions

Sentry Events

#	Date (UTC)	Browser	Page	Sentry	Fullstory
S1	2026-05-15 10:52	Edge Mobile 148.0 / iOS 26.4	/home	f8a3570f	Session
S2	2026-05-15 02:57	Safari 18.7.5 / macOS	/search	13893db5	Session
S3	2026-05-13 18:26	Safari 26.4 / macOS	/e/{reportID}	599726b9	Session
S4	2026-05-13 16:25	Safari 18.3.1 / macOS	/onboarding/purpose	6ef2cf74	N/A
S5	2026-05-13 15:11	Safari 26.1 / macOS	/home	adb622cc	Session

S1 breadcrumb analysis: 5h22m gap between last activity (05:30 UTC) and error (10:52 UTC). After the gap: version.json, Ping, PusherPing, ReconnectApp — all API fetches succeed (status 200), but subsequent Onyx IDB writes fail.

All sampled events show lowMemoryThreat: false and handled: no.

Fullstory Sessions

#	Location	Browser	Session	Key Finding
F1	Los Angeles	OS X, Safari	Replay	Error from session start, repeats every ~100s for 22+ bursts. Tab already dead on arrival — reconnection timer generates continuous errors.
F2	Woodmont, CT	iOS, Safari	Replay	Active browsing → 29min gap → error. Classic backgrounding.
F3	Madrid	OS X, Safari	Replay	Error fires immediately on /search. Connection already dead at session start.
F4	Seven Points, TX	iOS, Safari	Replay	Signup → settings → 25min gap → InvalidStateError heal fires → fresh connection also fails with connection lost.
F5	New York, NY	iOS, Safari	Replay	Error at 00:01 → user keeps working 40+ min with broken IDB. Writes silently fail. Multiple error-click events.
F6	New York, NY	iOS, Safari	Replay	14min gap → InvalidStateError → connection lost → user works 40 min with errors recurring at 15:32, 16:05, 16:30, 17:17. Connection permanently dead.

Evidence

1. Retries Are Futile (with current code)

Retry attempt distribution (VL, 7d):

Attempt	Count
1/5	19,979
2/5	20,097
3/5	20,152
4/5	20,312
5/5	21,035

Almost perfectly flat. The connection is dead within the retry window. But critically: the current code never clears the cached dbp for this error — all 5 retries reuse the same stale connection. This doesn't prove that a fresh indexedDB.open() would also fail (see "Important nuance" below).

97.5% of sessions exhaust all 5 retries, resulting in silent data loss (retryOperation resolves with Promise.resolve()).

2. Bursty — Few Sessions Generate Massive Log Volume

Top hourly spikes (VL, 7d, UTC):

Time (UTC)	Count
May 14 18:00	3,992
May 14 19:00	2,708
May 12 22:00	2,278
May 11 20:00	2,055

A single backgrounded tab returning triggers a cascade of Onyx writes against a dead IDB connection — consistent with S1 breadcrumbs and F1 replay.

3. 100% WebKit

Sentry breakdown (30d):

Browser	Events	Unique Users
Mobile Safari	126	78
Safari (desktop)	93	63
Chrome iOS	17	10
Edge iOS	14	10
DuckDuckGo (iOS + macOS)	14	9
Other (Google app, OktaMobile, WKWebView)	11	9

Safari version breakdown (VL, 7d):

Safari Version	Count	%
26.x	17,680	87.3%
18.x	2,449	12.1%
17.x	115	0.6%

Reflects market share — no version is immune. Long-standing WebKit behavior.

4. IDB Never Recovers (without intervention)

Fullstory F5 and F6: users worked 30-40+ minutes after the first error. Errors recurred at every navigation point. The connection is permanently dead until page refresh.

The app remains usable from in-memory cache — users navigate, browse, submit expenses. But all writes silently fail.

5. Important Nuance: Reconnection Was Never Tested for 99% of Cases

"Connection to Indexed Database server lost" is an UnknownError on IDBRequest — it hits the async rejection path in executeTransaction. The existing heal mechanism in createStore.ts only triggers for:

• InvalidStateError (sync throw on db.transaction())
• Backing store corruption (UnknownError with "Internal error opening backing store")

For this error, the cached dbp is never cleared and indexedDB.open() is never re-attempted. The flat retry counts prove the stale connection is reused, not that a fresh connection would also fail.

In the ~1% of cases where InvalidStateError heal did fire first (F4, F6), the fresh connection also failed — but WebKit #197050 comment #5 suggests this is expected when the IDB server process was terminated. Whether a delayed reopen (e.g., after visibilitychange + short wait) could succeed is untested.

6. Top Pages = Landing Pages

Transaction	Events
/search	175
/home	156
/inbox	36

Exactly where ReconnectApp fires on tab return.

7. Not Memory or Storage Related

• lowMemoryThreat: false in all sampled events
• Zero evictions, zero quota checks (from #87844)

8. Operations That Fail

Method	Count	%
setWithRetry	87,389	74%
multiSetWithRetry	30,637	26%

Current Handling in Onyx

• Error classification: Not in STORAGE_ERRORS, treated as generic error
• Retry behavior: retryOperation retries immediately 5 times with zero delay, reusing the same stale cached dbp
• Heal path: createStore.ts heal only triggers for InvalidStateError and backing store errors — this error bypasses it entirely
• Failure mode: After 5 retries, resolves with Promise.resolve() — silent data loss
• onclose handler: Exists in createStore.ts — clears dbp when browser fires onclose. Safari may not fire onclose reliably when it kills the IDB server, leaving stale dbp cached.

Proposed Fix

Layer 1: Extend createStore heal + fail-fast (immediate win)

In createStore.ts: Extend executeTransaction error handling to detect "connection to indexed database server lost" (and "connection is closing"). On match:

1. Drop cached dbp (set to undefined)
2. Attempt one reopen (indexedDB.open()) + retry the transaction
3. If the reopen also fails → set a module-level storagePermanentlyDegraded flag

In OnyxUtils.ts: When retryOperation encounters this error after the createStore heal has already failed, stop retrying immediately (fail-fast). Emit one logAlert per session instead of 5× logInfo per operation.

Impact: Eliminates ~120K VL entries/week. Tests whether a fresh indexedDB.open() can actually recover some sessions (currently unknown — we've never tried for 99% of cases).

Layer 2: Proactive detection via visibilitychange

Add a visibilitychange listener in the IDB provider. When tab returns to foreground:

1. Run a lightweight IDB probe (readonly transaction or sentinel key read)
2. Optional short delay to give Safari time to restore the IDB server
3. If probe fails → set storagePermanentlyDegraded before ReconnectApp triggers the Onyx write storm

This prevents the entire error cascade. Note: document.onfreeze/document.onresume (Page Lifecycle API) are not supported in Safari — visibilitychange is the correct event.

Layer 3: User notification

When storagePermanentlyDegraded is set, expose a callback (e.g., Onyx.onStorageDegraded) so the App can show a persistent banner prompting the user to refresh. A full page reload is the documented recovery path per WebKit #197050.

Alternative: Call degradePerformance() to switch to MemoryOnlyProvider for the rest of the session — this makes the silent failure explicit and matches existing degradation infrastructure in storage/index.ts. However, this doesn't solve the data loss on next refresh (IDB is stale).

Scoping

Layer	Effort	Impact
1: Heal + fail-fast	Small	Eliminates log noise, tests recoverability, stops silent data loss chain
2: visibilitychange probe	Medium	Prevents error cascade entirely
3: User notification	Small (Onyx side)	Fixes silent data loss UX

Layers are independent and can ship separately. Layer 1 should go first — it both fixes the immediate noise problem and answers the open question of whether indexedDB.open() can recover these sessions.

Note on #87782 (retry backoff)

The retry backoff proposal in #87782 targets connection errors broadly with exponential delay. For this specific Safari error class, backoff alone won't help — the connection is permanently dead, not transiently unavailable. The fix here (heal + fail-fast) should be treated as a separate path from general backoff, which remains useful for other error classes.

[leshniak]

Proposed Solution

The core problem

When Safari kills the IDB connection, Onyx keeps using the stale cached connection (dbp in createStore.ts). Every retry hits the same dead connection — indexedDB.open() is never re-attempted. This is why retries are 100% futile (flat distribution across all 5 attempts).

The fix: detect the error and reopen the connection

In createStore.ts, the .catch handler around executeTransaction currently only handles InvalidStateError. We extend it to also detect "Connection to Indexed Database server lost" and "Connection is closing". On match:

1. Drop the cached dbp (forcing a fresh indexedDB.open() on the next call)
2. Retry the transaction once with the new connection
3. Bounded by a shared heal budget (3 attempts, reset on every successful transaction) to prevent infinite loops

Note: this is the same heal pattern being built for the sibling issue #87862 (Chromium backing store corruption) in react-native-onyx#780. Both errors have the same structural problem (stale dbp, never cleared) and the same fix shape — we're handling them together in one mechanism.

What this tells us

The investigation found that 99% of connection lost cases never attempted a fresh indexedDB.open() — the heal path was bypassed entirely. Only ~1% triggered InvalidStateError first (where the existing heal fired, but the fresh connection also failed).

This fix directly tests what happens for the other 99%:

• If a fresh connection succeeds → session recovers silently, heal budget resets. Problem solved for that session.
• If a fresh connection also fails → heal budget drains, error propagates. We now have data to confirm that recovery truly requires a page reload, and can fail-fast instead of retrying futilely (via #90633).

Either outcome is a win — we either recover sessions or stop wasting 120K+ log entries/week on retries that can never succeed.

Follow-up layers (separate PRs, independent)

Layer	What	Why
Fail-fast (#90633)	After heal fails, add to NON_RETRIABLE_ERRORS to stop 5× futile retries in retryOperation	Eliminates log noise, stops silent data loss chain
Proactive detection	visibilitychange listener + lightweight IDB probe when tab returns to foreground	Detects dead connection before ReconnectApp triggers the write storm
User notification	Onyx.onStorageDegraded callback so the App can show a "please refresh" banner	Fixes silent data loss — users currently work 30-40 min from cache without knowing writes are lost

[fabioh8010]

The Proactive detection idea sounds interesting to me and I think it can help in these cases.

However I don't think we should do User notification – I don't think it's good UX to notify users about it, mainly because

1. As we saw before users can still use the app normally even with the DB broken
2. It can create frustration or feeling of broken product if users face these kind of issues frequently, and at the end we are not really helping too much

I think we should focus in efforts of healing and preventing the DB gets to this state, while keeping the failing situations invisible to the user.

[leshniak]

@fabioh8010 extended Expensify/react-native-onyx#780 to handle this kind of error.

[leshniak]

Extracted proactive detection to Expensify/react-native-onyx#788