Databox

A local-first ingestion and data-product platform with dlt-managed Iceberg tables on AWS S3, an Apache Polaris catalog, and local Trino analytics. Databox transforms data with SQLMesh, validates it with Soda, and orchestrates the workflow with Dagster—without always-on infrastructure.

Platform boundary

Databox owns reusable source ingestion, Polaris/Iceberg raw authority, generic environmental models, platform health, and reusable source-quality controls. Its canonical registry contains seven sources: six routine refresh sources plus the explicit pinned AVONET snapshot. The public databox_sources.usfws interface is provider-only and requires caller-owned targets.

flowchart LR
    sources[Public sources] --> dlt[dlt]
    dlt --> iceberg[Iceberg tables on S3]
    dlt -->|commits metadata| polaris[Apache Polaris]
    polaris -->|catalog discovery| trino[(Local Trino)]
    iceberg -->|raw table data| trino
    sqlmesh[SQLMesh models] -->|executes| trino
    trino -->|writes| analytics[Analytics Iceberg tables]
    soda[Soda] -. validates .-> trino
    dagster[Dagster] -. orchestrates .-> dlt
    dagster -. orchestrates .-> sqlmesh
    dagster -. asset checks .-> soda

Loading

The active deployment selects DATABOX_SQLMESH_GATEWAY=trino and materializes SQLMesh production outputs in a separate Iceberg analytics catalog. The local DuckDB gateway and its existing outputs remain available for rollback. See Trino migration for setup, validation, and rollback.

From source to model

New dlt sources move through a reviewable, agent-guided modeling workflow:

flowchart LR
    schema[dlt schema] --> annotate["annotate-sources<br/>annotations + taxonomy"]
    annotate --> ontology[create-ontology]
    ontology --> cdm["generate-cdm<br/>Kimball CDM"]
    cdm --> transform["create-transformation<br/>SQLMesh models"]

Loading

The project skills—annotate-sources, create-ontology, generate-cdm, and create-transformation—turn raw schemas into business-aware warehouse models before transformation SQL is written. See the workflow.

Quickstart

Prerequisites: Python 3.12+, uv, and Task.

Evaluate without live providers

git clone https://github.com/Doctacon/databox.git
cd databox
task install   # creates .env from .env.example when absent
task ci

After the initial dependency install, source tests replay recorded responses, so task ci needs neither provider credentials nor a populated warehouse.

Build the local warehouse

After task install, configure the Polaris client, AWS S3 bucket, temporary AWS writer credentials (access key, secret key, and session token), EBIRD_API_TOKEN, NOAA_API_TOKEN, and XENO_CANTO_API_KEY values documented in .env.example. The configured databox_lake must already be provisioned with s3://<bucket>/warehouse and the bucket-scoped role before publication. Start the local catalog, bootstrap the pinned AVONET snapshot once, then refresh the routine sources. For Trino outputs, set DATABOX_SQLMESH_GATEWAY=trino in .env and provision the separate analytics catalog as shown below:

$EDITOR .env
mkdir -p data .dagster
docker compose --env-file .env -f compose.iceberg.yml up -d
curl --fail --silent http://127.0.0.1:8182/q/health/ready
uv run python scripts/platform/provision_trino_catalog.py
DAGSTER_HOME="$PWD/.dagster" PYTHONPATH="$PWD" \
  uv run dg launch --target-path packages/databox --job avonet_ingest
task full-refresh   # ingest Iceberg, run project-wide SQLMesh, then verify Soda contracts

AVONET is intentionally excluded from routine refreshes. See the operations runbook.

The Compose stack runs localhost-only Polaris with a PostgreSQL metadata backend; the authoritative Iceberg data and metadata files remain in AWS S3. It also includes the official open-source Polaris Console at http://127.0.0.1:8080, built from a pinned Apache polaris-tools revision and connected to the Polaris API on port 8181.

Protected integration verification

Ordinary CI is credential-free and never contacts live providers or publishes to S3. Maintainers can manually dispatch Polaris Iceberg integration through its protected GitHub environment. It runs each of the six routine sources independently with GitHub OIDC, disposable Polaris credentials, and a source-scoped integration/<run>/<attempt>/<source>/warehouse prefix. The first complete passing run on 2026-09-03 verified all six real provider-to-Iceberg paths without targeting the normal warehouse/ prefix. Historical evidence was retired with the former planning workspace; current runs are recorded by the protected workflow.

Learn more

• Documentation and data dictionary
• Architecture decisions
• Configuration
• Commands
• Adding a source
• Forking and rebranding Databox

License

MIT