Skip to content

fix: preserve the auto-capture cursor after successful history-flow extractions - #932

Merged
rwmjhb merged 1 commit into
CortexReach:masterfrom
gorkem2020:fix/autocapture-watermark-reset
Jul 15, 2026
Merged

rwmjhb merged 1 commit into
CortexReach:masterfrom
gorkem2020:fix/autocapture-watermark-reset

Conversation

@gorkem2020

Copy link
Copy Markdown
Contributor

Problem

After every successful auto-capture write, the per-session cursor is reset to zero (autoCaptureSeenTextCount.set(sessionKey, 0)). In history-carrying sessions, where agent_end delivers the whole session each turn, the cursor doubles as the slice offset, so the turn after any successful extraction re-reads the entire session history as new. Observed live: a later extraction's input re-contained turns already extracted minutes earlier. Costs grow with session length, and re-rolling the same content through extraction and admission multiplies the chances of a low-quality item eventually slipping through.

The reset is not simply a bug: the commit trail and a pinning test show it is intentional for ingress-fed sessions (per-message channels feeding pendingIngressTexts), where the counter acts as a pure new-text accumulator for extractMinMessages windowing and a length-based reset would wedge the window closed.

Fix

Flow-aware reset: pendingIngressTexts.length > 0 ? 0 : eligibleTexts.length. Ingress sessions keep the accumulator semantics; history sessions record the consumed length so subsequent turns slice only the delta.

Tests

Red-proven regression (turn 2 re-read turn 1's content before the fix; only the delta after) plus the pre-existing ingress counter-reset scenario passing unchanged.

Verification on a live gateway

A four-turn session produced extraction inputs containing each turn exactly once; the previous whole-history re-read signature is gone.

Copy link
Copy Markdown

After #919 was merged, this PR now has merge conflicts with the latest main. Please rebase the branch onto the current main, resolve all conflicts, and push the updated head. We will review the updated revision once it is clean and mergeable.

@gorkem2020
gorkem2020 force-pushed the fix/autocapture-watermark-reset branch from 67ef600 to 137ff6e Compare July 13, 2026 17:08

@app3apps app3apps left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 137ff6e. The targeted tests and full suite pass, and the patch fixes the reported cursor reset for the covered monotonically growing history flow.

Two non-blocking follow-ups are worth tightening:

  • Auto-capture work is fire-and-forget. An older, shorter capture can finish after a newer one and overwrite the newer watermark, causing already-consumed messages to be submitted again.
  • When the next eligible history is equal in length or shorter, the strict growth check leaves the entire history in newTexts; unchanged retries or rolling windows can therefore be reprocessed wholesale.

A per-session generation/serialization guard plus equal-length and out-of-order completion regressions would make the cursor robust. These edge cases do not outweigh the concrete improvement in this PR, so approving.

Copy link
Copy Markdown

This PR has developed merge conflicts again after the latest changes landed on main. The approval above applies to head 137ff6e, but GitHub now reports the branch as conflicting.

Please rebase onto the current main, resolve the conflicts, and rerun the test suite. Once the updated head is pushed, we will re-check the conflict resolution before merge.

…ction

Two auto-capture flows share the autoCaptureSeenTextCount counter. In
the ingress flow (message_received feeds pendingIngressTexts and each
agent_end carries only the newest message) the counter is a pure
accumulator of new texts toward extractMinMessages, and resetting it to
0 after a successful extraction is the intended windowing from issue
message history each turn) the same counter is also the slice cursor
into that history; resetting it to 0 made the very next capture re-read
and re-extract the entire session: repeated LLM cost and repeated
admission and dedup evaluations over already-extracted content.

Make the post-success reset flow-aware: ingress-fed turns keep the
reset to 0 (the existing counter-reset scenario in
test/smart-extractor-branches.mjs still passes unchanged), while
history-carrying turns record the consumed history length so the next
capture only sees the delta. Red-proved: the new regression test fails
against the previous code with turn 1 content re-read in turn 2, and
passes after the change. Wire the suite into the local npm test chain
and the core-regression CI group.
@gorkem2020
gorkem2020 force-pushed the fix/autocapture-watermark-reset branch from 137ff6e to eadeaf2 Compare July 14, 2026 15:55

@app3apps app3apps left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed rebased head eadeaf2. The targeted tests and full suite pass, and the monotonic-history cursor fix remains sound for the primary flow.

Non-blocking follow-ups: equal or shorter history redelivery can still be treated as entirely new, and cursor updates around extraction/concurrent completion could inflate or regress the saved position. Overlap, redelivery, and concurrency tests would tighten those edges.

Approving.

@rwmjhb
rwmjhb merged commit 7791bd3 into CortexReach:master Jul 15, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants