research(storage): attribute remaining ingestion cost after shaping repair #1282

Problem and objective

Bulk ingestion remains a large part of the canonical lifecycle. Determine the dominant remaining ingestion costs after #1279, and produce a measured recommendation for reducing time to a durably published, queryable project. This is a bounded attribution study, not authorization for speculative product changes or a blanket optimization campaign.

Evidence reviewed

• Historical accepted S20/S22 receipts: ingestion 217.267 / 1,104.101 s, whole lifecycle 359.113 / 1,682.601 s. S22 ingestion accounts for approximately 65.6% of lifecycle wall time; seal alone is 899.920035 s. These nested times must not be added.
• The completed S20 receipt from the integrated fix(scale): resolve S20–S22 process RSS growth blocking S24 admission #1278 campaign observes ingestion 197.749 s and lifecycle 339.061594 s (58.3% ingestion), using frozen source 6cbfde81494f864a3a61e4d887138aaaa256c92e. Its S22 receipt was not yet published when this issue was prepared. These single observations are not a controlled isolated speedup measurement; the integrated source includes both perf(storage): diagnose and resolve lifecycle runtime scaling threatening S25/S26 #1279 and the RSS repair.
• perf(storage): diagnose and resolve lifecycle runtime scaling threatening S25/S26 #1279 / PR perf(storage): share decoded sources in shaping row merges #1280 already removed per-selected-row Arrow source-descriptor duplication. Its controlled shaping fixture reduced median CPU 4.69 → 3.90 s, but deliberately retained the external-merge passes. Do not repeat that repair or extrapolate its fixture gain to the whole lifecycle.
• Existing production-accumulator tests model 272 / 1,088 input runs with 544 / 3,264 records read and written, respectively. The additional merge level is real deterministic work. This one-record-per-input model does not establish every artifact family's pass count or its share of ingestion time.
• Earlier S18 profiles identify fixed-record merging, Parquet row merging, canonical encoding, hashing, and synchronization as investigation targets. Their inclusive/traced timings and whole-process CPU samples do not establish the post-repair ingestion bottleneck.

Sources: shared evidence ledger, merged repair and validation, current RSS repair/evidence campaign. Reuse the final validated campaign receipts when available; retain source/executable identities and distinguish provisional observations from accepted evidence.

Study

1. Reconcile the completed integrated S18/S19/S20/S22 prefix and existing construction timings before scheduling measurements. Preserve every accepted receipt and all preselected observations. Do not interfere with the current host campaign or duplicate qualification merely for this study.
2. Split ingestion into append/input preparation, seal/shaping, canonical encoding, validation/authentication, synchronization, recovery and publication using existing timing scopes and counters. Attribute CPU samples and logical/physical I/O to actual commands and call stacks. Keep inclusive time, disjoint time, residual time, CPU, I/O wait and sync latency separate.
3. Quantify merge passes and bytes per artifact family, especially actual run-count boundaries. Distinguish necessary external sorting from avoidable decode/encode, hashing, copying or rereading. Inspect graph_construction.rs, graph_construction/shaping_merge.rs, graph_construction_encoding.rs and existing construction evidence before adding instrumentation.
4. Design fixed 1×/2×/4× real Rust-facade fixtures and boundary cases around the relevant fan-in transitions. Keep generator semantics, batches, source/toolchain identities and resource limits fixed. Preselect at least three uninstrumented observations per comparable case; record host activity/cache conditions without system-wide cache changes. Run profiling separately and report its scope and overhead limitations.
5. Rank measured opportunities by attributable ingestion time/work, expected benefit, and memory/disk/durability costs. Consider merge scheduling, bounded buffering, encoding or concurrency only when profiles justify them. Do not infer that more threads help from elapsed time alone. Set a baseline-derived minimum useful benefit and resource budget before any later optimization experiment.

Acceptance and completion evidence

• Given the completed integrated receipts, the report reproduces ingestion/whole-lifecycle fractions and construction phase totals, identifies source differences and uncertainty, and makes no claim from an unfinished S22 run.
• Given comparable facade fixtures, all preselected runs and exact commands are retained; measured attribution explains the dominant remaining costs and how they scale, with deterministic pass/byte/row counters supporting timing claims.
• Given a candidate recommendation, identify affected ownership/code paths, expected eliminated work, resource tradeoffs and a deterministic regression oracle. Preserve exact node/edge counts, ordered one-/two-hop results, reopen, export/full verification and clean-import correctness. Include corruption, cancellation and recovery coverage appropriate to any proposed change.
• Publish a sanitized study in the existing scale evidence ledger, with raw-artifact hashes and measurement identities. Conclude with ranked, bounded repair recommendations or an evidence-backed no-change disposition. No speculative implementation is required to close this study.

Boundaries and relationships

Related: completed #1279 (reuse its attribution and repair), active #1278 / PR #1281 (reuse final combined receipts; RSS remains separately owned), #1256 (existing operation timings), #1194 (remaining storage costs), and #900/#745 (canonical scale outcomes).

This study creates no new blocker or acceptance requirement for those canonical issues. A verified blocker discovered by the study must be scoped under its existing canonical owner rather than silently expanding that owner's scope.

No S24+ execution, timeout/reserve increases, weakened verification/authentication/fsync, hardware substitution, public API/durable-format changes, query-runtime optimization, or concurrent host benchmark campaign. Any new host measurements must wait for an idle host and remain within the existing controller's separately authorized scale cap, resource limits, reserve and first-failure stopping. Instrumented measurements are diagnostic, not admission evidence.

This issue was prepared by read-only receipt, source and GitHub review; no new profiling or workload execution occurred. Keep graph contents, credentials, UUID inventories and private host paths out of published evidence.