Repository navigation
feat(export): paginate the non-session collections the frame guard's hint calls out - #1143
Conversation
|
@dmazhukov is attempting to deploy a commit to the rohitg00's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 Walkthrough
Pre-merge checks ·
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/mcp/server.ts (1)
365-381: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winForward
memory_exportpagination args tomem::export.
memory_exportalways callssdk.trigger({ function_id: "mem::export", payload: {} }), somem::exportignores pagination and never returnscollectionPagination. The REST export handler already reads and forwardsmaxSessions,offset,collectionLimit, andcollectionOffset, while the MCP tool schema still declares an emptyproperties. Add argument validation and pass the same fields into the payload.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/mcp/server.ts` around lines 365 - 381, Update the memory_export case and its MCP tool schema to validate and accept maxSessions, offset, collectionLimit, and collectionOffset, then forward those values in the payload of sdk.trigger for mem::export. Preserve the existing isExportTooLarge handling and response formatting while ensuring pagination arguments reach the export function.
🧹 Nitpick comments (1)
src/functions/export-import.ts (1)
176-191: 🚀 Performance & Scalability | 🔵 TrivialCollection pagination still requires a full
kv.list()read per collection.
sliceCollectionpagesmemories,graphNodes, and the other collections only afterkv.list()has already fetched every row for that scope. For very large collections (the issue mentions 8K+ memories, 34K observations per session),collectionLimit/collectionOffsetshrink the response size but do not reduce the KV read cost behind each collection. This matches the PR's stated goal (avoid oversized WebSocket responses), so it is not a blocking concern, but if a collection ever grows large enough that the full-list read itself becomes slow, pagination here will not help.Consider tracking this as a follow-up if
StateKV.listgains offset/limit support later.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/functions/export-import.ts` around lines 176 - 191, Track this as a follow-up rather than changing the current export flow: collection pagination in sliceCollection occurs after each kv.list call and cannot reduce the full KV read. If StateKV.list later supports offset/limit parameters, update the collection reads in the export function to pass collectionOffset and collectionLimit directly while preserving the existing collection mappings and response slicing behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/mcp/server.ts`:
- Around line 365-381: Update the memory_export case and its MCP tool schema to
validate and accept maxSessions, offset, collectionLimit, and collectionOffset,
then forward those values in the payload of sdk.trigger for mem::export.
Preserve the existing isExportTooLarge handling and response formatting while
ensuring pagination arguments reach the export function.
---
Nitpick comments:
In `@src/functions/export-import.ts`:
- Around line 176-191: Track this as a follow-up rather than changing the
current export flow: collection pagination in sliceCollection occurs after each
kv.list call and cannot reduce the full KV read. If StateKV.list later supports
offset/limit parameters, update the collection reads in the export function to
pass collectionOffset and collectionLimit directly while preserving the existing
collection mappings and response slicing behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 562fdced-a35f-4cde-bb2d-b2080bd80994
📒 Files selected for processing (7)
src/functions/export-import.tssrc/mcp/rest-proxy.tssrc/mcp/server.tssrc/triggers/api.tssrc/types.tstest/export-import.test.tstest/mcp-standalone-proxy.test.ts
|
Heads-up on the red CI here: both failures reproduce on a clean checkout of
An endpoint landed in #1132 / #1136 without the generated docs moving with it. I checked that this branch adds no endpoints: #1144 refreshes the four files that carry the count. Once it lands I'll rebase this branch so the matrix reflects the actual change. |
|
Good catch on The empty
|
|
Correction to my note above: I've folded the same 4-line refresh into this branch (3966cb8, identical to #1144's commit) rather than waiting.
#1144 still stands on its own if you'd rather take the housekeeping separately — it is the same commit, so whichever merges first makes the other a no-op. |
c473c5b to
6bd1868
Compare
6bd1868 to
f1613dc
Compare
|
Rebased onto v0.9.29 and reduced. The size guard this PR originally carried is gone — Title and description rewritten to match. 1614 tests pass, |
f2b4c51 to
c07aadc
Compare
c07aadc to
42ffa7f
Compare
42ffa7f to
d756f9d
Compare
87f2e6e to
007a26e
Compare
The export refuses a store that does not fit one transport frame, and its hint said the non-session collections "are not yet paginated". They are now: ?collectionLimit / ?collectionOffset window memories, summaries and the sixteen top-level collections, and ?collections= returns only the named ones. Sessions and their observations come back only when the list names sessions, so a client after lessons does not pull every session. Access logs follow the memories on the page, because import keeps a log only next to its memory. A paged or selected export carries collectionPagination (totals for every collection, the selection, hasMore over the selected ones), and import refuses strategy replace for such a file instead of wiping everything the file does not hold. Without these parameters the response is unchanged. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
The tool declared no properties and always exported with an empty payload, both through the server's /mcp/call and through the @agentmemory/mcp shim the plugin launches, so the paging the REST endpoint offers was out of reach from MCP. Both now forward maxSessions, offset, collectionLimit, collectionOffset and collections; the shim passes them as the query string of /agentmemory/export. collections is forwarded raw, empty value included: an empty selection means no collections, and that only stays distinguishable from an absent argument if this layer does not drop it. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
007a26e to
83ba43c
Compare
|
Confirming this is still the live blocker on v0.9.30, with a measurement from a real store. Environment: Windows 11,
Two things this data point adds to the motivation:
So on v0.9.30 the endpoint is not merely bounded, it is terminally unavailable for any store this size, and Thanks for keeping this rebased. |
mem::export now pins the graph generation before it reads, and the standalone proxy carries both the export query and the session query. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
…edges A store whose summaries alone exceed the frame limit is refused whole and exported in pages with collections=summaries and collectionLimit. Unknown collection names stay out of the echoed selection, access logs stay out unless memories are selected, an access-log selection reports no further page, and memory_export pages a real export end to end. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
replace already refused a collection page or selection, but an export taken with maxSessions that stops short of the last session, or starts past the first, went through and wiped the store down to that page. A window that covers every session is still accepted. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
collectionLimit leaves sessions and their observations whole unless collections leaves them out, and access logs come only with memories. Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
Pages the export's top-level collections.
?collectionLimit/?collectionOffsetwindow memories, summaries, the graph and the other collections;?collections=returns only the named ones, sessions only when named.memory_exporttakes the same arguments through/mcp/calland the@agentmemory/mcpshim in proxy mode.On v0.9.30 a store whose
summariesalone pass the frame limit gets 413 withstoppedAt: "summaries"at any?maxSessions/?offset, as measured in the comment below;export-incremental-budget.test.tsreproduces it and pages it out withcollections=summaries.Access logs travel with their memories. Import refuses
replacefor any partial export, a session page included. Without the new parameters the response is unchanged.