observe(storage): attribute the lazy adjacency rebuild and property authentication - #1516
Conversation
…uthentication The lifecycle phase counters under-reported two real workloads (#1449): - The lazy adjacency rebuild read every projected edge table and wrote its spill runs, CSR shards and manifests with raw File I/O, so a query-time rebuild (the fallback for appends and missing or stale published indexes) attributed nothing. read_path_scan reported 579 KB of a 6.4 GB hop query. - authenticate_inventory_file computed the streamed SHA-256 over every property payload at admission and reported the totals in its own metrics without ever reaching the phase counters, so property-bearing opens under-reported the hydration/verification row. The rebuild's reads and writes now flow through the read-path row (edge tables via ReadPathFile, spill-run and shard reads and writes, the CSR manifest read and write, and serving-time shard authentication on first row touch). Property authentication records bytes, calls and block equivalents under the hydration/verification row. Construction publish and clean import scope their index builds to the encoding row with a PhaseScope, so construction never records a read-path scan and the closed construction evidence contract is untouched. The S20 re-measure on the bench host is the remaining acceptance evidence; durable projects cannot open in this environment.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: CurateLabs/graphforge/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
write_csr_shard reads every shard back in full to hash it; that read is rebuild I/O of the whole shard and was the one builder read still invisible to the lifecycle counters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
df939b2 to
d85aa47
Compare
Summary
Closes the two attribution gaps #1449 named. The lifecycle phase counters silently under-reported two real workloads:
FileI/O in the builderread_path_scan: projected edge-table reads (ReadPathFile), spill-run writes and reads, CSR shard writes and first-touch authenticated reads, shard-manifest read, manifest writeauthenticate_inventory_file)record_read+record_blocksunderhydration_verification, bytes/calls/block-equivalentsWhy publish/import do not move
Construction publish (
encode_adjacency) and clean v2 import scope their index builds toEncodeWritePostwriteAuthenticationwith aPhaseScope, soread_path_scanstays zero for construction (preserving the documented contract and the closed construction-evidence accounting, which is artifact-granular by design) while the lifecycle ledger now carries the build in the row that owns it.Acceptance against #1449
Tests
Five new storage tests, all passing:
the_lazy_rebuilds_reads_and_writes_reach_the_lifecycle_countersspill_runs_are_counted_in_both_directions_of_their_lifetime(tinychunk_rowsforces spill runs)a_publish_style_scope_moves_the_build_out_of_the_read_path_rowserving_a_row_attributes_its_shard_payload_readproperty_authentication_reaches_the_lifecycle_countersValidation:
cargo clippy --workspace -- -D warningsclean,cargo fmt --all -- --checkclean, graphforge-exec adjacency tests 25/25, and the storage lib failure set is byte-identical to the pre-change base (314 durable-filesystem environment failures in both; this VM cannot open durable projects).Related #1388, #1422, #1446.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.