Skip to content

feat(live): wire LeChef proposals and bounded session privacy into the live gateway - #161

Draft
obvious-autobuild[bot] wants to merge 1 commit into
obv/burst5-fridgeluck-lechef-toolsfrom
obv/burst5-fridgeluck-lechef-live-wiring
Draft

obvious-autobuild[bot] wants to merge 1 commit into
obv/burst5-fridgeluck-lechef-toolsfrom
obv/burst5-fridgeluck-lechef-live-wiring

Conversation

@obvious-autobuild

Copy link
Copy Markdown

What changed and why

Wires the LeChef tool layer into the Gemini Live session, per the live-wiring slice of docs/spec/lechef/SPEC.md (§8.3, §9, §15).

Tool surface. The live gateway now binds the four legacy read tools plus the 15 registered LeChef tools (six reads, nine propose* tools). mutate_inventory is gone: the backend never writes to the phone's truth — it proposes, the phone approves and executes, and the receipt is what the model sees. The system prompt is rewritten accordingly (confidence policy, proposal rules, meal-log always explicit, camera looks can never yield a safety verdict).

Proposal routing. Each connection gets its own LivePhoneExecutor (the phone mirror) and LeChefRuntime. Reads answer from the phone's once-per-session bounded snapshot; proposals are routed to the phone over the socket and the tool call stays open until the phone's receipt resolves it (executed / rejected / expired) — the model can never claim an execution the phone did not confirm. Receipts resolve at most once; an unknown-proposal receipt is a no-op.

Session privacy (SPEC §9).

  • Camera frames live in an in-process cache (2-min TTL from server receipt time, 500-session LRU); only their metadata (mime, size, receipt time) reaches the store — never the bytes.
  • No transcript is persisted; the user's words are relayed to the model and dropped.
  • Client-supplied latestConfidence is rejected — the exact-mode gate is computed server-side only.
  • confirmedIngredients are trimmed to 50 lines with grams coarsened to 10 g.
  • Session ids are minted server-side; a client-supplied id is logged and ignored. Client tool_response frames are rejected at the protocol layer.
  • Sessions are deleted on close and swept on a TTL timer; expiry enforcement stays in reads.

How I checked it

  • bunx tsc --noEmit -p tsconfig.json — clean.
  • bun test src/__tests__ — 321 pass / 0 fail (was 318; stale mutate_inventory and 5-tool-count expectations updated to the new surface).
  • New src/__tests__/liveSessionGateway.test.ts (6 end-to-end tests over a real WebSocket with a fake Live session): server-side session-id minting, client-authority stripping, metadata-only frame storage + delete-on-close, snapshot trimming/confidence rejection, and the full proposal → receipt → resolved-tool-response round trip. All pass.
  • bun src/run/lechef-suite.ts — 65/65 job successes, 0 unsafe answers, 0 unauthorized actions.

What I could not check

  • No Gemini Live credentials here: the gateway was tested with a fake ai.live.connect, so real model sessions (audio modality, interruptibility, real tool-call shapes) are unverified.
  • No phone client on the other end: the socket proposal/receipt protocol is exercised by tests only, not by the iOS app.
  • iOS changes are out of scope for this PR, so nothing was checked against hosted macOS CI.

Assumptions

  • The legacy read tools stay alongside the LeChef reads (the spec supersedes rather than deletes them); easy to drop in a follow-up if unwanted.
  • Client session_context ingredient lines are trimmed (name + coarse grams) rather than rejected outright — the store keeps a hint, not a measurement.
  • PR base is obv/burst5-fridgeluck-lechef-tools (the tools packet), continuing that slice.

🤖 Generated with Obvious

…e live gateway

The live WebSocket gateway now runs the LeChef tool surface: the four
legacy read tools plus the 15 registered LeChef tools (six reads, nine
proposal tools), with mutate_inventory removed — the backend never
writes to the phone's truth; it proposes and the phone executes.

- per-connection LeChefRuntime + LivePhoneExecutor: reads answer from
  the phone's once-per-session bounded snapshot; proposals route over
  the socket and the tool call stays open until the phone's receipt
  resolves it (executed/rejected/expired), never a claimed success
- protocol: session ids minted server-side (client-supplied ids are
  logged and ignored); client tool_response frames are rejected;
  proposal receipts resolve pending calls at most once
- privacy: camera frames live in an in-process TTL+LRU cache and only
  their metadata reaches the store; no transcript is persisted;
  client-supplied latestConfidence is dropped (the exact-mode gate is
  server-computed); confirmed ingredient lines are trimmed to 50 and
  coarse grams; sessions are deleted on close and TTL-swept
- system prompt rewritten for LeChef (confidence policy, proposal
  rules, no safety verdicts from camera looks)

Verified: bunx tsc --noEmit clean; bun test src/__tests__ 321 pass /
0 fail (new liveSessionGateway.test.ts covers session-id minting,
client-authority stripping, metadata-only frame storage, snapshot
trimming, and the proposal->receipt round trip); LeChef suite
65/65 job successes, 0 unsafe answers, 0 unauthorized actions.

Co-authored-by: obvious-autobuild[bot] <262744130+obvious-autobuild[bot]@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

This PR was authored by a bot without an assigned CodeRabbit review seat. Ask an organization administrator to assign a seat to the bot. See the seat-assignment guide. After the seat is assigned, request a review again.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c9f783be-a4c0-4158-abbb-dfecb24acd2e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SamGu-NRX

Copy link
Copy Markdown
Owner

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
⚠️ Action not completed

Review skipped.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@obvious-autobuild

Copy link
Copy Markdown
Author

Independent review (Obvious, Autobuild + Max)

Verdict: ready (as a draft) — the PR does what its body claims, every runnable claim reproduced exactly, and the new tests demonstrably bite. No blocking defect found. One decision worth making before merge is finding 1.

Findings (ranked by severity)

1. Medium — get_restock_plan now reads a ledger nothing can write, while the rewritten prompt tells the model to use it.

  • Where: backend/gemini-agent/src/agent/toolRegistry.ts:125 (deps.ledger.snapshot()); src/agent/systemPrompt.ts:44 and :98–100.
  • Evidence: with mutate_inventory removed, InventoryLedger.addItems/decrementItems have zero remaining production callers (grep over src, non-test: only inventoryLedger.ts itself; server.ts:118 and api/webhooks.ts:17 only read snapshot()). The ledger is in-memory and starts empty every boot, so the tool can only ever return an empty plan — yet the prompt explicitly instructs "Use get_restock_plan … when the user asks about expiring food or what to buy."
  • Smallest fix: drop get_restock_plan from TOOL_DECLARATIONS and the prompt line (SPEC §8.3's registry table doesn't include the legacy four), or derive the plan from the phone's bounded pantry snapshot. The body flags keeping the legacy reads as an assumption; this is the concrete cost of that assumption.

2. Low — proposal_receipt is resolved with no shape check against the routed proposal.

  • Where: src/services/livePhoneExecutor.ts:136–145 (resolveProposal), fed verbatim from src/services/liveSessionGateway.ts:379–390.
  • Evidence: any receipt with a truthy proposalID is stored and wakes the waiter. payloadHash is never compared to the routed proposal (which the executor already holds in proposals), and state is not validated against the receipt enum — a buggy phone sending state: "approved" resolves the open tool call as status: "resolved" carrying a malformed receipt.
  • Smallest fix: in resolveProposal, compare receipt.payloadHash to the routed proposal's hash (ignore on mismatch) and treat a state outside executed/rejected/expired as a client_error. The submit path already re-validates hash and TTL (:104–121); the resolve path is the asymmetric half.

3. Low — a pending proposal tool call can block for up to 300 s.

  • Where: src/agent/toolRegistry.ts:194–196 — waitForReceipt(..., PROPOSAL_TTL_SECONDS * 1000); PROPOSAL_TTL_SECONDS = 300 (src/lechef/types.ts:373).
  • Evidence: a phone that stays connected but never receipts keeps the tool open five minutes; function calls queued behind it in the same toolCall message dispatch sequentially in the gateway's loop, so the model gets nothing until expiry. Socket close resolves the waiters (close() → resolve(null)), so this needs a live-but-silent phone — but a crashed app with an open socket is exactly that.
  • Smallest fix: cap the wait at a shorter budget (30–60 s) or return the pending outcome immediately and treat a later receipt as log-only. The body declares the open-tool-call design; the 5-minute tail deserves a conscious choice.

4. Low — the once-per-session bounded snapshot is silently dropped if it arrives before the Gemini session opens.

  • Where: src/services/liveSessionGateway.ts:401 — the if (!session) gate; only proposal_receipt is handled ahead of it.
  • Evidence: ai.live.connect is async; a phone sending session_context immediately on connect gets client_error "Live session not initialized yet.", and the protocol has no ack or retry for session_context — LeChef reads would then return blank defaults for the whole session.
  • Smallest fix: handle session_context before the gate like receipts (it never touches the model session), or buffer the last one per connection and apply it on open. The gate itself is pre-existing, but the snapshot upload is new in this PR and is once-per-session.

5. Low — frame bytes moved into process memory with no per-frame size bound.

  • Where: src/services/liveSessionGateway.ts:527–541 (recordLatestFrame), src/services/frameCache.ts (InMemoryFrameHolder.record).
  • Evidence: bounds are TTL (2 min) and 500 sessions; frame size is unbounded. Previously the bytes went to the store (Firestore caps a doc at 1 MiB); now a 100 MB base64 "frame" sits in RAM for up to 2 min across up to 500 sessions — an OOM surface for an authenticated client.
  • Smallest fix: reject frames above a size cap (e.g. 1 MiB base64) before frames.record, restoring the old effective bound.

What I verified (commands and results)

All on PR head 347ed52 (fresh worktree of pull/161/head), in backend/gemini-agent:

  • bunx tsc --noEmit -p tsconfig.json → exit 0.
  • bun test src/__tests__ → 321 pass / 0 fail across 22 files — matches the body's claim.
  • bun test src/__tests__/liveSessionGateway.test.ts → 6 pass. The new tests run a real WebSocket server, the real gateway, the real (memory) store, and a fake Live session; the proposal round-trip exercises the real dispatch path (runtime.dispatch → LivePhoneExecutor.submitProposal → socket envelope → client receipt → resolved sendToolResponse).
  • bun src/run/lechef-suite.ts → 65/65 job successes, 0 unsafe answers, 0 unauthorized actions, all gates true — matches the body.
  • Full diff read; body claims cross-checked against the code: 4 legacy + 15 LeChef tools (REGISTERED_TOOLS, runtime.ts:25), mutate_inventory gone, server-minted session ids (liveSessionGateway.ts:142–163), client tool_response rejected at the protocol layer (CLIENT_ENVELOPE_TYPES + :365–376), metadata-only frame storage (writeEnvelope), no transcript (recordClientText removed), client latestConfidence rejected (applySessionContext omits it), 50-line/10 g trim (BOUNDED_SNAPSHOT_MAX_LINES, trimBoundedIngredients), delete-on-close (close envelope → socket.close → cleanup; also on socket error), 5-minute TTL sweep, receipts resolve at most once and unknown-proposal receipts are a no-op (resolveProposal receipts-map guard — covered by a test), legacy latestCameraFrame bytes deleted on the next write (FieldValue.delete()). Every tool declaration's arg names match the nine proposal builders in lechef/proposals.ts.
  • Revert experiments (each change reverted immediately after the run; worktree left clean, git status --porcelain empty):
    • Re-added "tool_response" to CLIENT_ENVELOPE_TYPES (the old behavior) → rejects a client tool_response frame… fails (5 pass / 1 fail). Restored.
    • Leaked dataBase64 back into the store write (the old byte-persistence behavior) → caches frame bytes in-process, stores only metadata… fails (5 pass / 1 fail). Restored.

What I could not verify

  • Real model sessions: no Gemini Live credentials here — everything ran against the fake ai.live.connect, so audio modality, interruptibility, and real tool-call shapes are unverified (the body discloses this).
  • The phone side: no iOS client exercises the proposal/receipt protocol yet; it is test-only (the body discloses this).
  • Firestore adapter paths: all tests run the memory store. The Firestore expiry enforcement, sweepExpired query, and legacy-bytes delete typecheck but never executed; the owner-run Firestore TTL policy on expireAt (SPEC §9.5) remains owner-run.
  • macOS CI: iOS is out of scope for this PR, so there was nothing to check there.

Scope is right: nine files, all in the live-wiring slice the body names (liveSessionGateway.ts + liveSessionStore.ts are exactly what SPEC §15 assigns to it), no drive-by changes, no CI/workflow edits.

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants