C2 · carve re-encode — slice into a fresh small dictionary

[styk-tv]

Carve chain (tracking #6; SPEC.pgRDF.CARVE 4b.5 / 5.B). Carve into a FRESH small dictionary by reusing the staged DICT phase on reload, so the slice owns its term space (not the 1.80 B source terms) and fits small hardware. The file mode: parallel decode-to-file then load_turtle_staged_run re-encodes. Follows C1 (#10). Track per adopted method.

[styk-tv]

C2 mechanism proven at scale — set-based re-encode, no text decode (E160 trial)

Empirical run carving the full unpruned SE 2-hop (120,874,307 quads, supernodes included) off the live 8.2 B Wikidata-truthy on E160. This proves the C2 re-encode path and corrects the §14 design (which used text decode → .nt → reload, whose decode was the ~74 s / 140 K bottleneck = hours at 120 M).

The decode bottleneck is avoidable. Two changes:

1. EXTRACT — index-only split UNION (see the separate index-only issue): 120.8 M quads in 42.7 s.
2. RE-ENCODE without text — the standalone slice does NOT need a text round-trip or an id remap:
   • distinct_terms = 120,724,737 (the slice references 6.5 % of the 1.84 B source terms — this is the small dict).
   • Build the dict subset with a set-based hash semi-join — one Parallel Seq Scan of the dict + a hash of the 120.7 M term-ids — not 120 M random pkey probes (the thing that made §14 decode slow).
   • Keep the original (sparse) dict ids — no renumber. Sparse bigint ids are fine; quads still reference them; unique_term stays valid; lexical_md5 (STORED generated) auto-recomputes on insert. No FK on datatype_iri_id, so insert order is free.
   • Move to a fresh DB by cross-DB COPY of the dict subset + the quad slice (defer-index: drop hexastore + unique_term, ALTER id SET GENERATED BY DEFAULT, COPY, rebuild). No serialise/parse.

So C2 = id-space EXTRACT → set-based dict-subset → COPY into a fresh small-dict DB → rebuild hexastore (+ trigram, see #20). The only unavoidable cost is one sequential pass over the source dict heap; everything else is local.

This is the mechanism carve_graph(..., format => 'inplace'/'newdb') should codify. Updating the v0.6.18 design here; will fold the measured numbers into SPEC.pgRDF.CARVE.md.