Skip to content

observe(storage): attribute the lazy adjacency rebuild and property authentication - #1516

Merged
DecisionNerd merged 2 commits into
mainfrom
observe/1449-attribute-rebuild-and-property-auth
Sep 21, 2026
Merged

DecisionNerd merged 2 commits into
mainfrom
observe/1449-attribute-rebuild-and-property-auth

Conversation

@DecisionNerd

@DecisionNerd DecisionNerd commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes the two attribution gaps #1449 named. The lifecycle phase counters silently under-reported two real workloads:

Gap Before After
Lazy adjacency rebuild (query-time fallback for appends and missing/stale published indexes) 0 bytes attributed — raw File I/O in the builder Reads and writes land in read_path_scan: projected edge-table reads (ReadPathFile), spill-run writes and reads, CSR shard writes and first-touch authenticated reads, shard-manifest read, manifest write
Property-role authentication (authenticate_inventory_file) Bytes counted into the inventory's own metrics only record_read + record_blocks under hydration_verification, bytes/calls/block-equivalents

Why publish/import do not move

Construction publish (encode_adjacency) and clean v2 import scope their index builds to EncodeWritePostwriteAuthentication with a PhaseScope, so read_path_scan stays zero for construction (preserving the documented contract and the closed construction-evidence accounting, which is artifact-granular by design) while the lifecycle ledger now carries the build in the row that owns it.

Acceptance against #1449

  • Adjacency rebuild reads and writes attributed to a phase
  • Property-role authentication attributed
  • Re-measure the S20 one-hop on the bench host and show attributed bytes against measured disk bytes — not done here: durable projects cannot open in this environment, so the issue stays open for that measurement. Note the post-perf(storage): publish the adjacency CSR with the generation instead of rebuilding it per process #1453 steady state serves the published CSR presence-only, so the re-measure should also record the new fallback-free baseline.

Tests

Five new storage tests, all passing:

  • the_lazy_rebuilds_reads_and_writes_reach_the_lifecycle_counters
  • spill_runs_are_counted_in_both_directions_of_their_lifetime (tiny chunk_rows forces spill runs)
  • a_publish_style_scope_moves_the_build_out_of_the_read_path_row
  • serving_a_row_attributes_its_shard_payload_read
  • property_authentication_reaches_the_lifecycle_counters

Validation: cargo clippy --workspace -- -D warnings clean, cargo fmt --all -- --check clean, graphforge-exec adjacency tests 25/25, and the storage lib failure set is byte-identical to the pre-change base (314 durable-filesystem environment failures in both; this VM cannot open durable projects).

Related #1388, #1422, #1446.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…uthentication

The lifecycle phase counters under-reported two real workloads (#1449):

- The lazy adjacency rebuild read every projected edge table and wrote its
  spill runs, CSR shards and manifests with raw File I/O, so a query-time
  rebuild (the fallback for appends and missing or stale published indexes)
  attributed nothing. read_path_scan reported 579 KB of a 6.4 GB hop query.
- authenticate_inventory_file computed the streamed SHA-256 over every
  property payload at admission and reported the totals in its own metrics
  without ever reaching the phase counters, so property-bearing opens
  under-reported the hydration/verification row.

The rebuild's reads and writes now flow through the read-path row (edge
tables via ReadPathFile, spill-run and shard reads and writes, the CSR
manifest read and write, and serving-time shard authentication on first row
touch). Property authentication records bytes, calls and block equivalents
under the hydration/verification row. Construction publish and clean import
scope their index builds to the encoding row with a PhaseScope, so
construction never records a read-path scan and the closed construction
evidence contract is untouched.

The S20 re-measure on the bench host is the remaining acceptance evidence;
durable projects cannot open in this environment.
@coderabbitai

coderabbitai Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: CurateLabs/graphforge/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 16984989-9fba-4b7f-a2cc-48253ca72c51

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the core Core source code changes label Sep 20, 2026
write_csr_shard reads every shard back in full to hash it; that read is
rebuild I/O of the whole shard and was the one builder read still invisible
to the lifecycle counters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@DecisionNerd
DecisionNerd force-pushed the observe/1449-attribute-rebuild-and-property-auth branch from df939b2 to d85aa47 Compare September 21, 2026 17:21
@github-actions github-actions Bot added documentation Improvements or additions to documentation ci-cd CI/CD configuration changes labels Sep 21, 2026
@DecisionNerd
DecisionNerd added this pull request to the merge queue Sep 21, 2026
Merged via the queue into main with commit f8f8667 Sep 21, 2026
21 checks passed
@DecisionNerd
DecisionNerd deleted the observe/1449-attribute-rebuild-and-property-auth branch September 21, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd CI/CD configuration changes core Core source code changes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant