Follow-up from AI review on #56 (gemini-code-assist, cubic-dev-ai, greptile).
Problem
MappingStore (crates/honmoon-core/src/secret_tokenizer/mod.rs) accumulates every placeholder→secret mapping for the lifetime of the gateway process and never evicts. Two consequences:
- Unbounded memory growth — a long-running or high-traffic gateway with
--redact-secrets keeps every historical secret mapping in memory, risking OOM (gemini secret_tokenizer:386, cubic secret_tokenizer:381).
- Per-response full clone — once any mapping exists,
handle_response calls snapshot() which clones the entire map on every response, including clean responses with no placeholders (greptile mitm.rs:695). This is a deliberate tradeoff (the clone avoids holding the mutex across body streaming, see the comment at mitm.rs handle_response), but it scales with the unbounded store.
Why not a naive fix
A plain LRU/TTL eviction would silently drop mappings still needed for detokenization, leaking unresolved placeholders or breaking the cache-stable determinism guarantee (#20). As gemini notes, retention must be tied to a session/request lifecycle, not a blind cache policy.
Proposed direction
- Design a retention policy scoped to conversation/session lifetime (or an explicit capacity + correctness-preserving eviction) that keeps detokenization correct.
- Once retention is bounded, the per-response
snapshot() clone cost is bounded too; optionally revisit whether a cheaper reference/Arc sharing scheme can replace the clone without holding the lock during streaming.
Deferred from #56 as out of scope for that PR; the current behavior is correct for typical (small, stable secret-set) sessions.
Follow-up from AI review on #56 (gemini-code-assist, cubic-dev-ai, greptile).
Problem
MappingStore(crates/honmoon-core/src/secret_tokenizer/mod.rs) accumulates every placeholder→secret mapping for the lifetime of the gateway process and never evicts. Two consequences:--redact-secretskeeps every historical secret mapping in memory, risking OOM (geminisecret_tokenizer:386, cubicsecret_tokenizer:381).handle_responsecallssnapshot()which clones the entire map on every response, including clean responses with no placeholders (greptilemitm.rs:695). This is a deliberate tradeoff (the clone avoids holding the mutex across body streaming, see the comment atmitm.rshandle_response), but it scales with the unbounded store.Why not a naive fix
A plain LRU/TTL eviction would silently drop mappings still needed for detokenization, leaking unresolved placeholders or breaking the cache-stable determinism guarantee (#20). As gemini notes, retention must be tied to a session/request lifecycle, not a blind cache policy.
Proposed direction
snapshot()clone cost is bounded too; optionally revisit whether a cheaper reference/Arcsharing scheme can replace the clone without holding the lock during streaming.Deferred from #56 as out of scope for that PR; the current behavior is correct for typical (small, stable secret-set) sessions.