Import research: recipes as materialized views, update modes measured against real source shapes

[phas02]

What it buys

A decided import design before the storage layer moves again. The direction: a recipe is a materialized view over a source; its result lands as parquet; versions belong to the record and the cube, not to the data.

What stands

• Sources are one of three kinds: files (landed by recipe, the one path today), a lake (parquet in an object store, readable in place by the engine), a database (read through ADBC with the recipe's SQL run at the source; hard deletes cannot be seen from outside the source).
• IMPORT <table> appends files not yet landed and refuses changed or gone files; the file list rides a snapshot property (LANDING_FILES_PROP); the schema is inferred under a blocking wait (The file reader infers its schema under a blocking wait #58); a changed recipe's re-land drops the table before the new landing stands (A changed recipe's re-land drops the table before the new landing stands #59); the pin cannot overwrite (Strike and aspect re-declare on the record #5).
• The record holds glossary, aspects, witnesses, functions, sources, relationships, measurements. Landings and recipes still ride Iceberg snapshots and table properties (The imports relation is composed from data-table snapshots; storing it keeps the landing record through snapshot expiry #45).

The research

Shapes, each with what an update means for it: a file drop that grows; a file rewritten whole on every export; a partitioned lake where one partition moves; a database with an update key; a database with hard deletes, fenced by its owner into a view or a replica.

Modes to measure against them: full replace; append of new files; partition replace; increment by update key; merge on key. For each: cost at three sizes, what it requires of storage, what the record must remember to run it again.

The deliverable is that table. The storage decision reads off it: what remains of the data plane after landings, recipes and versions are record rows is a file list per recipe version and the engine's parquet scan.

Done when

The table exists with measured numbers; the recipe's update mode is one word on the declaration, proposed as a SPEC.md diff; the storage decision is a ruling with its grounds.

[phas02]

Research result

1. What a source kind can give

kind	what the engine sees	what an update can be told from outside
files (parquet, csv, json under a location)	a listing: path, size, modified (crates/import/src/lib.rs, list_source)	a new path; a changed size or modified time; a gone path. Never which rows changed inside a rewritten file.
lake (parquet in an object store, read in place)	the same listing; the engine's own parquet scan over it	the same; a partition is a path prefix
database (relational_db, the recipe's SQL run at the source)	the recipe's result set and nothing else	nothing: no listing, no change marks; only what the SQL selects. A hard delete is invisible unless the source is fenced into a view or a replica its owner refreshes whole.

2. Shapes against modes

Shapes are what real exports look like; modes are what a refresh does to the landed table. Each cell: what storage must do, what the record must hold to run it again, and what stands today.

shape → / mode ↓	file drop that grows (new files, old ones never change)	file rewritten whole on every export	partitioned lake, one partition moves	database with an update key	database with hard deletes, fenced
full replace	works, wasteful	the one mode that fits	works, wasteful	works when the view is small	the one mode that fits
append new files	the one mode that fits	wrong: doubles the rows	wrong for the moved partition	not applicable	not applicable
partition replace	not needed	not needed	the one mode that fits	not applicable	not applicable
increment by update key	not needed	not needed	not needed	the one mode that fits; misses deletes	misses deletes
merge on key	not needed	not needed	not needed	works; needs the key and a delete or rewrite in storage	works for updates, still misses deletes

What storage must do, per mode: full replace and partition replace need a set of files to stop being the table's and another set to start, in one commit; append needs only an add; increment by key needs an add plus a high-water mark in the record; merge on key needs a delete file or a rewrite of the files holding the old rows.

What the record must hold, per mode: the files each landing read with their size and modified time (held today as a snapshot property, LANDING_FILES_PROP); for partition replace the partition-to-files map; for the keyed modes the key columns and the last key value seen. All are rows once landings are record rows (#45).

What stands today (crates/session/src/session.rs, the Declaration::Recipe arm and import; crates/import/src/lib.rs, open_import):

• full replace exists as a changed recipe's re-land: the new result is collected whole, the table is dropped, created and appended (A changed recipe's re-land drops the table before the new landing stands #59). Measured on the trial with a 12,290-row parquet table over a remote store: 1.6 s and 2.3 s for two runs, end to end.
• append of new files exists as IMPORT, and only when the recipe maps rows one for one over a single scan and every landed file still stands as it landed; a changed or gone file is refused by name because the lake cannot replace rows.
• the other three modes do not exist. Nothing in the test data produces their shapes either: the finance generator exports one file per table and rewrites it whole; the RelBench tables are one parquet file each. A growing drop and a partitioned lake need a generator that writes them before those cells can be measured.

Every landing moves the dataset's whole pin: re-landing constructor_results, a table no metric reads, set bands and checks back to "run again" on the trial (#71 is the same coupling on the cube's key).

3. What the pinned substrate can do

iceberg-rust at 5c3d18f offers fast_append and no overwrite. Upstream, every write beyond an append is open: the overwrite action (apache/iceberg-rust#2185), row delta with delete files (#2203), the merging snapshot producer and file rewrite (#3046), the copy-on-write primitive (#2752). The RFC that fixes the transaction architecture they all need (#2620) merges as a five-milestone plan with none of the milestones shipped. Replace in place on an Iceberg table in this pin therefore has no date. Two more standing costs of the format here: a catalog walk loads and parses every table's metadata file per statement (the pin and load spans, about 40 ms per table over a remote store), and the DataFusion scan takes no dynamic filter (#46) and no shared cache (#29).

4. The storage fork

After #45 and the two steps behind it (dataset settings and recipes as record rows, the table's version in the record instead of the snapshot id), what the data plane must provide is: a table, its versions, and each version's parquet files. Two ways to provide it.

Stay on Iceberg. Keep the format for the landed tables; wait for the overwrite and row-delta actions for the replace and merge modes; keep the walk. Cost: the one mode a reporting refresh needs on every run has no date, and until it lands a replace is a drop and a gap; the per-statement metadata loads stay; #46, #29, #30 stay open.

A DuckLake-shaped catalog in our database. The DuckLake 1.0 specification keeps a table's versions as rows: ducklake_snapshot (snapshot_id), ducklake_table, ducklake_column, ducklake_data_file with begin_snapshot and end_snapshot delimiting the snapshots a file is valid for, ducklake_delete_file for row deletes, and ducklake_inlined_data_tables for small inserts. A full replace is one transaction on those rows: the old files get an end_snapshot, the new ones a begin_snapshot; a partition replace is the same over a subset; append is an insert; merge on key is a delete file or a rewrite of the affected files. Reads are DataFusion's own parquet scan over the version's file list, the one scan the engine pushes dynamic filters into, so #46 closes by construction, and #59, #29 and #30 with it; the walk goes, because the file list is a query on the catalog we already hold. The data keeps no history: files no version references are deleted, which is what ducklake_files_scheduled_for_deletion is for. Cost: a writer of our own, about the size of write_files today, and leaving iceberg-rust.

Pick: the DuckLake-shaped catalog. The deciding case is the update story: replace in place is the one mode that always works for every shape and is what a refresh does every time, and the pinned format cannot do it with no date in sight. The subset needed is small, sits in the SQL database the record already lives in, and is readable by any DuckLake client without us owning a format.

5. The recipe's update mode

A language question, for SPEC.md §9, not decided here. The forms a corpus test would compare, each against the two real artifacts we have (a table exported whole every run; a drop that grows):

• no new word: IMPORT t means the recipe's result as the source stands now, appended when that is provably the same thing (one scan, rows one for one, landed files unchanged), replaced whole otherwise. This is the rule open_import already applies, with the refusal replaced by a replace.
• one word on the act: IMPORT t appends, IMPORT t REPLACE replaces.
• one word on the declaration: DECLARE RECIPE … [REPLACE | APPEND] AS $$…$$.

The keyed modes need a key on the recipe in every form, and no source that needs them exists yet.

6. Order of the work that follows

1. The imports relation is composed from data-table snapshots; storing it keeps the landing record through snapshot expiry #45, extended: landings, recipes and dataset settings as record rows; the table's version in the record. Right in both worlds.
2. The storage swap, as one issue after the ruling: the catalog tables, the writer, the scan; closes A changed recipe's re-land drops the table before the new landing stands #59, A join's dynamic filter never reaches the Iceberg scan #46, iceberg-rust: table loading takes no shared ObjectCacheProvider #29, iceberg-datafusion: task_writer is pub(crate); the write path re-derives it #30 by construction.
3. A gloss rebuilds only the cubes it can change #71: with a version per table and per relation in the record, the cube's key can name what it reads.
4. The cube as a table in the lake, materialized per landing, so a fresh instance serves it instead of building it.
5. Strike and aspect re-declare on the record #5 is independent and waits on its own ruling.

[phas02]

Ruled: no new word. IMPORT t is the recipe's result as the source stands now — appended when that is provably the same thing, replaced whole otherwise — and every landing stays a row of the record. The forms in §5 above are closed.

[phas02]

Ruled: an import's meaning is the recipe's result as the source stands now, in one commit; whether the engine appends or replaces to get there is its own economy, and a recipe that names its key merges by it. The work is #81, #82 and #83; the measured table this issue asked for is replaced by the equality test in #83, since every mode must equal a replace. The storage decision was ruled in #75.