Skip to content

perf(catalog): use row-major encoding for manifest record fields - #798

Merged
azimafroozeh merged 3 commits into
ModernRelay:mainfrom
azimafroozeh:manifest-packed-record
Sep 28, 2026
Merged

azimafroozeh merged 3 commits into
ModernRelay:mainfrom
azimafroozeh:manifest-packed-record

Conversation

@azimafroozeh

Copy link
Copy Markdown
Collaborator

What & why

A branch's __manifest currently uses a columnar layout: record fields are stored in separate columns, each with its own pages. Publish and reopen scans read these fields together, requiring data from several columns.

This PR groups the eight record fields and their null bitmask into one Lance packed record column. Within it, each row's encoded field values are stored consecutively in row-major order. object_id and object_type remain separate columns. Scans read the packed column's pages and expand the records into logical columns in memory, reducing storage read requests.

Backing issue / RFC

Follow-up to the copy-on-write publisher.

  • Accepted issue or RFC: not linked.

Checklist

  • Change is focused on the catalog's stored layout.
  • Tests added for nulls, conversion, historical reads, and mixed-format upgrades.
  • Public upgrade docs and API descriptions updated.
  • Reviewed against docs/dev/invariants.md.

Local verification

Recorded runs, not rerun for this draft:

  • cargo test -p omnigraph-catalog --locked: 56 passed.
  • cargo test -p omnigraph-gqt --locked: passed.
  • cargo fmt --all --check: passed.
  • cargo clippy --workspace --all-targets --locked -- -D warnings -W clippy::dbg_macro: passed.

Four sandbox failures and the final DST rerun remain outstanding.

Notes for reviewers

A physical column can span multiple pages, and Lance can combine or split their byte ranges into read requests. The catalog reads the whole packed struct; encoded size and history growth remain separate concerns.

New graphs use format v12. Existing v11 branches convert on their next publish; the new binary reads both formats and historical versions. Once main converts, v11-only binaries cannot open the graph.

The offline route to convert all branches is required before the release shipping v12. History retention remains separate work.

@azimafroozeh
azimafroozeh added this pull request to the merge queue Sep 28, 2026
Merged via the queue into ModernRelay:main with commit c22c3ab Sep 28, 2026
30 checks passed
@azimafroozeh
azimafroozeh deleted the manifest-packed-record branch September 28, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant