Repository navigation
perf(catalog): use row-major encoding for manifest record fields - #798
Merged
azimafroozeh merged 3 commits intoSep 28, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
A branch's
__manifestcurrently uses a columnar layout: record fields are stored in separate columns, each with its own pages. Publish and reopen scans read these fields together, requiring data from several columns.This PR groups the eight record fields and their null bitmask into one Lance packed
recordcolumn. Within it, each row's encoded field values are stored consecutively in row-major order.object_idandobject_typeremain separate columns. Scans read the packed column's pages and expand the records into logical columns in memory, reducing storage read requests.Backing issue / RFC
Follow-up to the copy-on-write publisher.
Checklist
docs/dev/invariants.md.Local verification
Recorded runs, not rerun for this draft:
cargo test -p omnigraph-catalog --locked: 56 passed.cargo test -p omnigraph-gqt --locked: passed.cargo fmt --all --check: passed.cargo clippy --workspace --all-targets --locked -- -D warnings -W clippy::dbg_macro: passed.Four sandbox failures and the final DST rerun remain outstanding.
Notes for reviewers
A physical column can span multiple pages, and Lance can combine or split their byte ranges into read requests. The catalog reads the whole packed struct; encoded size and history growth remain separate concerns.
New graphs use format v12. Existing v11 branches convert on their next publish; the new binary reads both formats and historical versions. Once main converts, v11-only binaries cannot open the graph.
The offline route to convert all branches is required before the release shipping v12. History retention remains separate work.