Skip to content

feat(export): paginate the non-session collections the frame guard's hint calls out - #1143

Merged
rohitg00 merged 6 commits into
rohitg00:mainfrom
dmazhukov:fix/1142-export-transport-limit
Oct 20, 2026
Merged

rohitg00 merged 6 commits into
rohitg00:mainfrom
dmazhukov:fix/1142-export-transport-limit

Conversation

@dmazhukov

@dmazhukov dmazhukov commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Pages the export's top-level collections. ?collectionLimit / ?collectionOffset window memories, summaries, the graph and the other collections; ?collections= returns only the named ones, sessions only when named. memory_export takes the same arguments through /mcp/call and the @agentmemory/mcp shim in proxy mode.

On v0.9.30 a store whose summaries alone pass the frame limit gets 413 with stoppedAt: "summaries" at any ?maxSessions / ?offset, as measured in the comment below; export-incremental-budget.test.ts reproduces it and pages it out with collections=summaries.

Access logs travel with their memories. Import refuses replace for any partial export, a session page included. Without the new parameters the response is unchanged.

@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown

@dmazhukov is attempting to deploy a commit to the rohitg00's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

The export flow now supports collection selection and pagination, reports collection totals, and enforces a configurable serialized-size limit. REST and MCP responses return HTTP 413 for oversized exports. Proxy errors include truncated response details.

Changes: Export protection

Layer / File(s) Summary
Export contracts and assembly
src/types.ts, src/functions/export-import.ts, test/export-import.test.ts
Exports support collection selection, offsets, and limits. Results include totals and hasMore metadata. Oversized serialized results return ExportTooLarge refusals.
HTTP and MCP export responses
src/triggers/api.ts, src/mcp/tools-registry.ts, src/mcp/server.ts, test/mcp-export-tool.test.ts, test/memories-pagination.test.ts, plugin/skills/agentmemory-mcp-tools/REFERENCE.md
REST and MCP handlers validate and forward export parameters. Oversized results return HTTP 413. Tool documentation describes paging, collection selection, and refusal behavior.
Proxy error detail reporting
src/mcp/rest-proxy.ts, test/mcp-standalone-proxy.test.ts
Proxy errors include up to 500 characters from the response body. The regression test checks the export-size hint.
Endpoint count updates
AGENTS.md, README.md, plugin/skills/agentmemory-rest-api/REFERENCE.md, src/index.ts
Documentation and startup status text report the updated REST endpoint counts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant api_export as api::export
  participant mem_export as mem::export
  participant collections as Collection reads
  Client->>api_export: Request export with collection selection and pagination
  api_export->>mem_export: Forward validated export parameters
  mem_export->>collections: Read selected collections and totals
  collections-->>mem_export: Paginated data and totals
  mem_export-->>api_export: ExportData or ExportTooLarge
  api_export-->>Client: HTTP 200 or HTTP 413
Loading

Possibly related PRs

  • rohitg00/agentmemory#849: Both PRs modify export behavior in src/functions/export-import.ts, src/types.ts, and test/export-import.test.ts.
  • rohitg00/agentmemory#1144: Both PRs update REST endpoint counts in project documentation and startup status text.
Pre-merge checks · Passed 4 · Failed 1

Failed Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed The changes address issue #1142 by refusing oversized exports, preserving worker availability, and adding pagination for large collections.
Out of Scope Changes check Passed The implementation, tests, MCP updates, and documentation changes support export refusal and pagination; no unrelated code changes are present.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the main change: pagination for non-session export collections to address the transport frame-size limit.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/mcp/server.ts (1)

365-381: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Forward memory_export pagination args to mem::export.

memory_export always calls sdk.trigger({ function_id: "mem::export", payload: {} }), so mem::export ignores pagination and never returns collectionPagination. The REST export handler already reads and forwards maxSessions, offset, collectionLimit, and collectionOffset, while the MCP tool schema still declares an empty properties. Add argument validation and pass the same fields into the payload.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp/server.ts` around lines 365 - 381, Update the memory_export case and
its MCP tool schema to validate and accept maxSessions, offset, collectionLimit,
and collectionOffset, then forward those values in the payload of sdk.trigger
for mem::export. Preserve the existing isExportTooLarge handling and response
formatting while ensuring pagination arguments reach the export function.
🧹 Nitpick comments (1)
src/functions/export-import.ts (1)

176-191: 🚀 Performance & Scalability | 🔵 Trivial

Collection pagination still requires a full kv.list() read per collection.

sliceCollection pages memories, graphNodes, and the other collections only after kv.list() has already fetched every row for that scope. For very large collections (the issue mentions 8K+ memories, 34K observations per session), collectionLimit/collectionOffset shrink the response size but do not reduce the KV read cost behind each collection. This matches the PR's stated goal (avoid oversized WebSocket responses), so it is not a blocking concern, but if a collection ever grows large enough that the full-list read itself becomes slow, pagination here will not help.

Consider tracking this as a follow-up if StateKV.list gains offset/limit support later.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/functions/export-import.ts` around lines 176 - 191, Track this as a
follow-up rather than changing the current export flow: collection pagination in
sliceCollection occurs after each kv.list call and cannot reduce the full KV
read. If StateKV.list later supports offset/limit parameters, update the
collection reads in the export function to pass collectionOffset and
collectionLimit directly while preserving the existing collection mappings and
response slicing behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/mcp/server.ts`:
- Around line 365-381: Update the memory_export case and its MCP tool schema to
validate and accept maxSessions, offset, collectionLimit, and collectionOffset,
then forward those values in the payload of sdk.trigger for mem::export.
Preserve the existing isExportTooLarge handling and response formatting while
ensuring pagination arguments reach the export function.

---

Nitpick comments:
In `@src/functions/export-import.ts`:
- Around line 176-191: Track this as a follow-up rather than changing the
current export flow: collection pagination in sliceCollection occurs after each
kv.list call and cannot reduce the full KV read. If StateKV.list later supports
offset/limit parameters, update the collection reads in the export function to
pass collectionOffset and collectionLimit directly while preserving the existing
collection mappings and response slicing behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 562fdced-a35f-4cde-bb2d-b2080bd80994

📥 Commits

Reviewing files that changed from the base of the PR and between 5023cf3 and 58b2b42.

📒 Files selected for processing (7)
  • src/functions/export-import.ts
  • src/mcp/rest-proxy.ts
  • src/mcp/server.ts
  • src/triggers/api.ts
  • src/types.ts
  • test/export-import.test.ts
  • test/mcp-standalone-proxy.test.ts

@dmazhukov

Copy link
Copy Markdown
Contributor Author

Heads-up on the red CI here: both failures reproduce on a clean checkout of main and are not from this branch.

  • npm run skills:check → DRIFT: plugin/skills/agentmemory-rest-api/REFERENCE.md (records 118, the generator produces 119)
  • test/consistency.test.ts → README says 129 endpoints on port, 130 are registered

An endpoint landed in #1132 / #1136 without the generated docs moving with it. I checked that this branch adds no endpoints: npm run skills:gen produces the same 119 on main and here.

#1144 refreshes the four files that carry the count. Once it lands I'll rebase this branch so the matrix reflects the actual change.

@dmazhukov

dmazhukov commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Good catch on memory_export — fixed in 712d9ba.

The empty payload: {} predates this PR, but the refusal I added tells the caller to page, and that advice was not actionable from the MCP tool because the schema declared no properties. src/mcp/tools-registry.ts now declares maxSessions, offset, collectionLimit and collectionOffset, and src/mcp/server.ts validates and forwards them with the same integer bounds the REST handler uses. plugin/skills/agentmemory-mcp-tools/REFERENCE.md is regenerated via npm run skills:gen.

test/mcp-export-tool.test.ts covers the four cases: arguments forwarded, unusable bounds dropped, a refusal reported as 413, and a normal export still 200.

@dmazhukov

dmazhukov commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Correction to my note above: I've folded the same 4-line refresh into this branch (3966cb8, identical to #1144's commit) rather than waiting.

npm run skills:check runs ahead of the test suite, so leaving it out meant the whole matrix went red at the first step and showed nothing about the change itself. With it in, this branch is green locally: skills:check passes and npx vitest run --exclude test/integration.test.ts gives 1560 passed, 0 failed.

#1144 still stands on its own if you'd rather take the housekeeping separately — it is the same commit, so whichever merges first makes the other a no-op.

@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from c473c5b to 6bd1868 Compare August 3, 2026 03:27
@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from 6bd1868 to f1613dc Compare August 9, 2026 15:17
@dmazhukov dmazhukov changed the title fix(export): refuse dumps past the transport limit instead of dropping the worker feat(export): paginate the non-session collections the frame guard's hint calls out Aug 9, 2026
@dmazhukov

dmazhukov commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto v0.9.29 and reduced.

The size guard this PR originally carried is gone — frame-guard.ts landed first and does the same job, so keeping mine would have been two guards racing in one function. What is left builds on checkPayloadFrameSize instead: the collection pagination its hint asks for, plus ?collections=, plus the MCP tool being able to reach either. The hint itself is updated to name the parameters rather than describe the gap.

Title and description rewritten to match. 1614 tests pass, tsc output is byte-identical to main.

@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch 2 times, most recently from f2b4c51 to c07aadc Compare August 17, 2026 04:30
@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from c07aadc to 42ffa7f Compare September 12, 2026 05:20
@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from 42ffa7f to d756f9d Compare September 25, 2026 05:17
@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from 87f2e6e to 007a26e Compare October 3, 2026 04:52
The export refuses a store that does not fit one transport frame, and its
hint said the non-session collections "are not yet paginated". They are
now: ?collectionLimit / ?collectionOffset window memories, summaries and
the sixteen top-level collections, and ?collections= returns only the
named ones. Sessions and their observations come back only when the list
names sessions, so a client after lessons does not pull every session.

Access logs follow the memories on the page, because import keeps a log
only next to its memory. A paged or selected export carries
collectionPagination (totals for every collection, the selection, hasMore
over the selected ones), and import refuses strategy replace for such a
file instead of wiping everything the file does not hold.

Without these parameters the response is unchanged.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
The tool declared no properties and always exported with an empty payload,
both through the server's /mcp/call and through the @agentmemory/mcp shim
the plugin launches, so the paging the REST endpoint offers was out of
reach from MCP. Both now forward maxSessions, offset, collectionLimit,
collectionOffset and collections; the shim passes them as the query string
of /agentmemory/export.

collections is forwarded raw, empty value included: an empty selection
means no collections, and that only stays distinguishable from an absent
argument if this layer does not drop it.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
@dmazhukov
dmazhukov force-pushed the fix/1142-export-transport-limit branch from 007a26e to 83ba43c Compare October 3, 2026 06:13
@7889545

7889545 commented Oct 12, 2026

Copy link
Copy Markdown

Confirming this is still the live blocker on v0.9.30, with a measurement from a real store.

Environment: Windows 11, @agentmemory/agentmemory@0.9.30, file_based KV, single user, ~1,500 sessions / ~280k graph edges.

GET /agentmemory/export  ->  HTTP 413
{
  "bytes": 15729297,
  "limitBytes": 15728640,
  "oversized": true,
  "stoppedAt": "summaries",
  "success": false
}

bytes - limitBytes = 657 bytes. The guard trips on summaries alone, so it never even reaches memories or the sixteen top-level collections -- the refusal cannot tell you which of the remaining ones would also overflow.

Two things this data point adds to the motivation:

  1. It only gets worse. The overshoot was 399 B when the guard first tripped on this store; it is 657 B now. Every summarized session enlarges summaries, and that collection is read whole regardless of ?maxSessions / ?offset, which only slice sessions. There is no parameter combination that exports this store.
  2. The refusal's own advice is not actionable on v0.9.30. EXPORT_OVERSIZE_HINT ends with or export fewer collections, but no such parameter exists on the released version -- which is precisely what this PR adds.

So on v0.9.30 the endpoint is not merely bounded, it is terminally unavailable for any store this size, and GET /agentmemory/export is the only whole-store export there is. Restoring it at all (even partially) is the difference between having a backup path and not having one.

Thanks for keeping this rebased.

mem::export now pins the graph generation before it reads, and the standalone proxy carries both the export query and the session query.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
…edges

A store whose summaries alone exceed the frame limit is refused whole and
exported in pages with collections=summaries and collectionLimit. Unknown
collection names stay out of the echoed selection, access logs stay out
unless memories are selected, an access-log selection reports no further
page, and memory_export pages a real export end to end.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
replace already refused a collection page or selection, but an export
taken with maxSessions that stops short of the last session, or starts
past the first, went through and wiped the store down to that page. A
window that covers every session is still accepted.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
collectionLimit leaves sessions and their observations whole unless
collections leaves them out, and access logs come only with memories.

Signed-off-by: Dmitrii Zhukov <dmitry0983@gmail.com>
dmazhukov added a commit to dmazhukov/agentmemory that referenced this pull request Oct 17, 2026
@rohitg00
rohitg00 merged commit 4b95a62 into rohitg00:main Oct 20, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants