Skip to content

fix(mobile): resend dictation audio lost when the connection drops - #35777

Merged
AmoabaKelvin merged 3 commits into
mainfrom
mobile-dictation-dying-socket
Oct 29, 2026
Merged

AmoabaKelvin merged 3 commits into
mainfrom
mobile-dictation-dying-socket

Conversation

@AmoabaKelvin

@AmoabaKelvin AmoabaKelvin commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator
Files Added Deleted Net
Test 2 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​115 $\color{#cf222e}{\Huge{\mathbf{−}}}$​2 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​113
Prod 10 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​258 $\color{#cf222e}{\Huge{\mathbf{−}}}$​151 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​107

ELI5

When the phone's connection to the desktop drops while you're dictating, the audio it had just sent was lost, so a word or a second of speech went missing from the transcript. Now the phone sends that audio again once it reconnects, and the desktop makes sure each piece is heard once, in order.

What Changed

Before: since #35338, dictation keeps recording through a reconnect, but every chunk in flight on the dying connection was rejected as "delivery unknown" and dropped. That was about a second of speech on a hard drop. The phone couldn't resend because the host had no way to tell a resend from new audio, so a resend could have repeated words.

After:

  • Host (runtime-mobile-dictation-controller.ts, new mobile-dictation-chunk-order.ts): chunks carry a seq, and the host feeds them in seq order, each seq once. A chunk that arrives ahead of a missing resend is held until the resend fills the gap. The hold is capped at what the phone itself holds through a reconnect: 30s of 16 kHz PCM (the shared MOBILE_DICTATION_MAX_PENDING_AUDIO_BYTES) and 1,000 chunks (30s of 32ms native events). Past either cap, the oldest gaps are given up on until both hold again. Finish feeds whatever is still held. The speech.dictation.start reply now says dedupesResentChunks: true.
  • Phone (mobile-dictation-audio-chunk.ts, mobile-dictation-desktop-start.ts): every chunk carries a seq. Only when the start reply has the flag does the phone resend a delivery-unknown chunk, after each reconnect, while that recording is still the active one. It extends the resend fix(mobile): keep dictation recording through a reconnect and keep what was already said #35338 added for chunks stranded in a direct→Relay switch (sendAcrossConnectionMove) rather than adding a second path. A request that timed out on a connection still up is not resent there.

Why

Ordering has to live on the host. On a plain direct reconnect, chunks captured during the outage are parked in the transport's own connect wait and go out before any resend. On the OTA page they also cross the bridge. The phone's dictation code can't put a resend ahead of them, and an earlier version of this branch that tried was beaten by both routes in review. The host is the one place that sees every chunk, so it orders them there. Feeding a resend as soon as it lands, out of order, would garble the words around the drop.

The capability rides on the start reply, not a status-probe capability. It's free, it's answered by the exact host holding the session, and the phone already waits for that reply before it sends any audio.

Mixed versions behave as today:

  • Old phone, new host: no seq, so every chunk is fed directly.
  • New phone, old host: no flag, so nothing is resent. The old host's DictationChunk is a plain z.object, so it strips seq.

Not in this PR: noticing a silent drop quickly. On direct the liveness watchdog takes about 44s (20s idle, then 3 × 8s probes), which is past the host's 30s grace, and relay has no idle probe at all. So this PR doesn't change the silent-drop case. A stacked follow-up makes an overdue chunk reply fire a decisive liveness probe, so the drop is noticed within the grace on both the native app and the OTA page. The 30s grace is unchanged on purpose: raising it would hold the speech model and block other devices' starts for longer, only to cover a gap the follow-up closes.

Linked Issue

#35213: this fixes the lost speech on a hard drop or connection switch. Fast detection of a silent drop follows in a stacked PR.

Visual Proof

N/A: no UI changes. The change is in which audio reaches the host after a drop, and the tests below prove it through the real dictation hook and the real host controller.

Testing

  • Host (runtime-mobile-dictation-controller.test.ts): a resent chunk is fed once and in spoken order, and finish feeds what's still held. A gap that never fills is given up on at the cap instead of stalling the transcript. Both tests fail on the pre-fix code.

  • Phone (mobile-dictation-reconnect.test.tsx): through useMobileDictation, a chunk lost to a drop is resent to a host with the flag, and never to a host without it. A recording cancelled during the outage is not resent. Each fails with its guard removed.

  • Mobile RPC recording goldens for the dictation families replay clean. The two chunk goldens and the scenario were re-recorded for the new seq param.

  • Focused dictation suites (mobile hooks, platform capture, session dictation; host controller, orcad speech services, speech worker lifetime) pass, as do the changed-file code-quality and React Doctor gates. Full lint, typecheck and test are left to CI.

  • Platforms: logic is platform-independent (phone transport + Node host); not hand-tested on a device.

  • I manually tested these changes locally

  • Automated tests added/updated, or explained why not below

AI Disclosure

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Ensure no issues in: Security, Cross-platform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance

  • Security: the host hold is bounded by bytes (about 1.3 MB of base64) and by count (1,000 chunks), so a hostile seq or a flood of tiny chunks can't make it loop, throw or grow past that.
  • Remote / headless: orca serve and orcad hosts use the same controller.
  • Folder workspaces: not affected.

For blessing (new behaviour: a host capability, and what the phone resends):

  • Try: dictate on the phone over LAN, toggle the desktop's Wi-Fi off and on mid-sentence (a hard drop), keep talking, then stop. The transcript should include the words spoken right before and during the drop, with nothing repeated.
  • Mixed versions: an older desktop gets no resends (no flag in the start reply), and an older phone is fed as today (no seq).
  • Worktree for a dev build: /Users/kelvinamoaba/orca/workspaces/orca/dictation-dying-socket-audio (branch mobile-dictation-dying-socket).

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

The host feeds chunks in seq order, each seq once, holding audio that
arrives ahead of a resend for up to the reconnect grace, and says so in
its start reply. To such a host the phone resends chunks rejected as
delivery-unknown after each reconnect, so speech sent on a dying
connection is no longer lost.
@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was established.

Summary

The phone resends audio left unanswered by a dropped connection when the host supports it. The host feeds numbered chunks once, in order.

  • Phones resend lost dictation audio after reconnecting to a capable desktop.
  • The desktop feeds resent dictation audio once and in spoken order.

Diagram

sequenceDiagram
    participant Phone
    participant Host
    participant Speech
    Phone->>Host: Start recording
    Host-->>Phone: "dedupesResentChunks = true"
    Phone->>Host: Chunk 0
    Host->>Speech: Feed chunk 0
    Note over Phone,Host: Connection drops during chunk 1
    Phone->>Host: Chunk 2 after reconnect
    Note over Host: Hold chunk 2
    Phone->>Host: Resend chunk 1
    Host->>Speech: Feed chunks 1 and 2 in order
    Phone->>Host: Finish
    Host->>Speech: Feed remaining held audio and finish
Loading

Reviews (5) · Last reviewed commit: "fix(mobile): skip every gap needed to ke..." · Reviewed by Greptile

@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

Dictation chunk requests now include sequence numbers. The mobile client reads whether the host deduplicates resent chunks and retries eligible requests after reconnect while dictation remains active. The runtime buffers sequenced chunks, forwards them in sequence order, and drains remaining chunks before finishing. Updated scenarios, goldens, and tests cover sequence numbers, resend behavior, and chunk ordering.

Priority: Normal

Merge Risk: Low · up to 164bc

Resending lost dictation audio after a reconnect now works without an age cutoff. Two edge cases remain: in rare near-limit cases, a recoverable audio gap can be abandoned early, and if the connection never returns, stopping dictation can wait noticeably longer before failing. Neither is likely to cause broad breakage, but both are worth tightening.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title "fix(mobile): resend dictation audio lost when the connection drops" matches the main change. The PR adds sequence numbers to dictation chunks, has the phone resend chunks lost during a drop…
Description check Passed The description follows the repository template. It has ELI5, What Changed, Why, Linked Issue (#35213), Visual Proof (N/A with reason), Testing, AI Disclosure, Review, Agent skill upstream boundary, N…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title "fix(mobile): resend dictation audio lost when the connection drops" matches the main change. The PR adds sequence numbers to dictation chunks, has the phone resend chunks lost during a drop, and has the host reorder and deduplicate them. The title is specific and follows a conventional-commit style.

Full details: Description check

Explanation

The description follows the repository template. It has ELI5, What Changed, Why, Linked Issue (#35213), Visual Proof (N/A with reason), Testing, AI Disclosure, Review, Agent skill upstream boundary, Notes, and Checklist sections. It explains the before and after, the mechanism, mixed-version behavior, and the known limitation for silent drops. Some items are unchecked, including the manual testing box and the final lint, typecheck, test, and build checkbox, and the AI Disclosure and Review sections are empty. The linked issue is referenced as a plain issue number rather than a closing keyword, which the template expects as "Fixes #". These gaps are minor, so the description is mostly complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 4 functions across 9 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e6c28706-c4fb-4aff-97d4-e02fd1d71f5d
Commits

Reviewing files that changed from the base of the PR and between 478a008 and a622241.

Files selected for processing: 12
  • mobile/rpc-foundation/goldens/matrix-speech.dictation-chunk-speech.dictation.chunk-1.json
  • mobile/rpc-foundation/goldens/speech-audio-chunk-acknowledged.json
  • mobile/rpc-foundation/pilot-scenarios.json
  • mobile/src/hooks/mobile-dictation-audio-chunk.ts
  • mobile/src/hooks/mobile-dictation-desktop-start.ts
  • mobile/src/hooks/mobile-dictation-pending-audio-budget.ts
  • mobile/src/hooks/mobile-dictation-reconnect.test.tsx
  • mobile/src/hooks/use-mobile-dictation.ts
  • src/main/runtime/mobile-dictation-chunk-order.ts
  • src/main/runtime/runtime-mobile-dictation-controller.test.ts
  • src/main/runtime/runtime-mobile-dictation-controller.ts
  • src/shared/rpc-contract/speech-params.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to GitHub limitations.

⚠️ Outside diff range comments (1)

Minor · Count decoded bytes for the holding limit. · mobile-dictation-chunk-order.ts:6

src/main/runtime/mobile-dictation-chunk-order.ts:6
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Count decoded bytes for the holding limit.

Base64 padding makes the sum of chunk lengths exceed this limit before the decoded audio exceeds 960,000 bytes. (raw.githubusercontent.com)

In the new test, after releasing the ten small chunks, six 160,000-byte chunks remain. They fit exactly, but the new loop releases another chunk and skips the preceding gap. A later resend for that gap is then discarded.

Track each chunk’s decoded byte length instead of comparing the sum of encoded lengths.


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 762da479-29f5-4f08-8cfc-971c16105e21
Commits

Reviewing files that changed from the base of the PR and between 06428c0 and 164bcc3.

Files selected for processing: 2
  • src/main/runtime/mobile-dictation-chunk-order.ts
  • src/main/runtime/runtime-mobile-dictation-controller.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 831ccb5b-b93a-431e-b991-8c9c21658b44
Commits

Reviewing files that changed from the base of the PR and between 164bcc3 and d873917.

Files selected for processing: 1
  • mobile/src/hooks/mobile-dictation-audio-chunk.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

@AmoabaKelvin
AmoabaKelvin force-pushed the mobile-dictation-dying-socket branch from d873917 to 164bcc3 Compare October 29, 2026 08:35

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e8508a7b-b225-481e-b663-992dbbf40a54
Commits

Reviewing files that changed from the base of the PR and between d873917 and 164bcc3.

Files selected for processing: 1
  • mobile/src/hooks/mobile-dictation-audio-chunk.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

timeoutMs: MOBILE_DICTATION_RECONNECT_GRACE_MS
}),
// A recording cancelled during the outage must not reach the host when it comes back.
queue.pendingAudioBudget.hostDedupesResends ? () => queue.isActive(dictationId) : undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

Supported by static analysis

Script executed:

sed -n '1,220p' mobile/src/hooks/mobile-dictation-audio-chunk.ts
rg -n 'failActiveDictation|finish|75_000|30_000|DELIVERY_UNKNOWN|delivery' mobile/src/hooks/use-mobile-dictation.ts

Repository: stablyai/orca

Length of output: 7907


Script executed:

sed -n '186,250p' mobile/src/hooks/use-mobile-dictation.ts
echo '--- speech-params ---'
sed -n '1,200p' src/shared/rpc-contract/speech-params.ts
echo '--- finish operation timeouts ---'
rg -n -F -e 'dictationSessionFinish' -e 'timeoutMs' mobile/src/dictation/mobile-dictation-operations.ts || test "$?" -eq 1
echo '--- 75s constants ---'
rg -n -e '75_000' -e '75000' mobile/src src/shared/rpc-contract || test "$?" -eq 1

Repository: stablyai/orca

Length of output: 5206


Fail the dictation when the chunk reconnect wait expires.

When a chunk's reconnect wait expires, sendAcrossConnectionMove rethrows the original delivery-unknown error. The chunk catch suppresses that error, so the dictation is not failed. finishDictation then waits for pending chunks, which adds up to 30 seconds. It then sends the finish request with a 75-second timeout, even though the host has already dropped the session.

Return a distinct reconnect-expired error from sendAcrossConnectionMove. Handle it in the chunk catch by calling failActiveDictation. The dictation can then end before finishDictation sends its request.

@AmoabaKelvin
AmoabaKelvin merged commit 558be99 into main Oct 29, 2026
98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant