Dofek (דופק)

Provider-agnostic fitness and health data ingestion, analytics, and visualization. Dofek connects fitness, sleep, nutrition, body-composition, and training sources to a shared data model served by web and iOS clients.

Start Here

• Documentation index: architecture, product flows, provider research, and operational runbooks.
• Database model: canonical storage rules and schema.
• Provider guide: provider interfaces, authentication, ingestion, and tests.
• Testing guide: test tiers, Docker dependencies, and local recovery.
• Development environment: pinned tools, cloud initialization, Dev Containers, CodeGraph, and RTK.
• Deployment guide: production topology, release flow, secrets, and diagnostics.
• Analytics guide: dbt-owned ClickHouse models and bounded builds.
• Processing-status runbook: end-to-end ingestion and analytics evidence.
• Database backup runbook: Databasus health, R2 freshness, and isolated restore verification.

Architecture

API and import providers ──> sync/import workers ──> Postgres (fitness.*)
                                  │                       │
                                  │                       └─ PeerDB CDC
                                  │                              │
High-volume sensor samples ──────┴─> Redpanda ────────────────> ClickHouse
                                        │                         │
                                        └─> Cloudflare R2 archive │
                                                                  │
                          Web app and iOS app <── Express + tRPC ──┘
                                              │
                                              └─ Redis + BullMQ

• Postgres owns relational application state and lower-volume raw health data.
• High-volume metric_stream events bypass Postgres. Redpanda is the hot ingest log, Cloudflare R2 is the durable archive, and ClickHouse is the serving copy.
• PeerDB mirrors lower-volume fitness.* tables into postgres_fitness.* in ClickHouse.
• Incremental dbt models under analytics/models/ produce expensive derived analytics outside request paths. The serialized analytics worker also calls the pure @dofek/cycling-power algorithms through its bounded projector; dbt inputs precede projection and dependent healthspan models follow it. Derived sensor analytics are never written back to Postgres.
• Express and tRPC serve both clients. BullMQ workers use Redis for scheduled syncs, imports, exports, post-sync work, and analytics cleanup.

See the metric-stream architecture, the schema guide, and the deployment architecture for the detailed contracts.

Prerequisites

• mise at the minimum version declared in mise.toml. mise installs Node, pnpm, Python, uv, Infisical, CMake, Ninja, CodeGraph, RTK, and the remaining portable CLI tools from the reviewed checksums in mise.lock.
• Docker with Compose.
• Outside the Dev Container: Git, a C/C++ compiler, and a bootstrapped vcpkg checkout exposed through VCPKG_ROOT. The container supplies these from its pinned Dockerfile; vcpkg documents its bootstrap process.
• Infisical CLI access to this repository's linked project. Local commands that need application secrets fail if infisical export --env=prod cannot run; see Infisical CLI documentation.

Quick Start

MISE_LOCKED=1 mise install --locked
mise exec -- infisical login
mise run cloud:init

mise run cloud:init installs workspace dependencies, initializes CodeGraph, verifies RTK, starts Postgres, ClickHouse, Redis, and Redpanda on workspace-specific local ports and writes those ports to .env.local. It then loads the checked-in environment plus Infisical secrets, applies Postgres migrations, and applies ClickHouse migrations. See the development-environment guide for cloud prebuilds, non-interactive authentication, and the macOS/iOS profile.

For the complete local CDC topology, start the PeerDB Compose overlay and configure its mirrors:

pnpm compose -- -f docker-compose.yml -f docker-compose.peerdb.yml up -d
pnpm clickhouse-cdc

Always invoke Compose through pnpm compose --; the wrapper pins the physical workspace, project name, and project directory so concurrent Conductor workspaces remain isolated. Docker documents the same isolation controls in project names and --project-directory.

Development

Run the main processes in separate terminals:

# Provider sync runner
pnpm dev

# Express + tRPC API
pnpm --dir packages/server dev

# Vite web app
pnpm --dir packages/web dev

# Expo mobile app
pnpm --dir packages/mobile start

Useful repository commands:

pnpm lint
pnpm typecheck
pnpm test
pnpm test:integration
pnpm test:all
pnpm test:changed
pnpm test:changed:all
pnpm storybook:web
pnpm storybook:mobile

Before pnpm lint, start the workspace's ClickHouse service with pnpm compose -- up -d --wait clickhouse. For a new workspace, initialize .env.local first with pnpm compose:env --write. Full lint includes dbt-templated analytics SQL checks; see integration dependencies.

pnpm test and pnpm test:changed are Docker-free unit/mobile tiers. Database-backed tests run through the integration or :all commands, which start the workspace's Compose dependencies. See docs/testing.md before running or changing integration tests.

Repository Map

.
├── src/                    Root package: providers, sync, queues, and DB access
├── packages/server/        Express + tRPC API
├── packages/web/           Vite + React web client
├── packages/mobile/        Expo + React Native iOS client and native modules
├── packages/               Shared domain packages and provider clients
├── analytics/              Incremental dbt models for ClickHouse
├── drizzle/                Forward-only Postgres migrations
├── native/fit-decoder/     C++ Garmin FIT streaming decoder
├── scripts/                TypeScript repository and operational tooling
├── cypress/                Browser end-to-end tests
├── deploy/                 Terraform and Docker Swarm production configuration
└── docs/                   Architecture, research, runbooks, and history

pnpm-workspace.yaml is the canonical package list. Package-local READMEs describe public responsibilities and usage; adjacent AGENTS.md files contain agent-only implementation guidance.

Shared cross-client logic belongs in domain packages such as @dofek/format, @dofek/scoring, @dofek/nutrition, @dofek/training, @dofek/stats, @dofek/onboarding, and @dofek/providers. Web and mobile clients render server-computed metric values rather than recomputing them.

@dofek/climbing-catalogue is an independent public library for climbing route discovery and identity evidence. Dofek's catalogue service supplies application cache policy; source composition supplies credentials and rate-limited transport.

Data and Provider Contracts

• Providers implement the interfaces in src/providers/types.ts and register through the provider registry.
• API, credential, OAuth, config, and import-only providers are cataloged in src/providers/README.md; this code-adjacent catalog is authoritative instead of a duplicated count in this file.
• Apple Health upload/import and native WHOOP BLE capture enter through web/mobile ingestion paths outside the scheduled provider registry.
• Raw records retain provider and source attribution. Deduplication happens in queries and read models, never by discarding records during ingestion.
• Nutrients use row-based canonical storage in fitness.food_entry_nutrient. fitness.v_nutrition_provider_daily retains raw per-provider totals; serving totals derive from fitness.v_nutrition_daily, which resolves one contribution set at query time and reports ambiguous overlaps explicitly. See the canonical nutrition schema.

Authentication and Secrets

The clients support email/password authentication, configured Google or Apple identity providers, and eligible data-provider login flows. The server advertises the available methods through /api/auth/providers; see the app-password guide and src/auth/README.md.

Garmin connects through the approved application's OAuth flow and receives new data through official push delivery. Existing history and account-export ZIP imports remain available; users with legacy private tokens must reconnect. See the Garmin guide for Starter limits, required application configuration, and the deployment checks before releasing portal delivery (provider implementation, authorization migration).

Committed .env files contain non-secret configuration. Infisical contains secrets such as client secrets, tokens, private keys, and encryption keys. Production CI exports those values to a short-lived runner file; it does not store them on the server. Follow the production-secrets runbook when adding or rotating a secret.

Schema and Migrations

• drizzle/0000_baseline.sql is the fresh-Postgres baseline; later migrations are forward-only and tracked by drizzle/meta/_journal.json.
• ClickHouse migrations are ordered TypeScript modules in src/db/clickhouse-migrations/.
• pnpm setup-db is the canonical fresh-database bootstrap.
• pnpm generate creates a Drizzle migration and regenerates schema diagrams.
• Deploy migrations must remain schema-only. Large backfills and read-model refreshes require bounded, resumable operational tooling and a separate runbook.

See docs/schema.md and the ClickHouse read-model deploy runbook.

Deployment

The production image uses Node 26 and contains the Express server, built web assets, migrations, workers, sync entrypoints, and the native FIT decoder. GitHub Actions deploys the image and infrastructure as a single Docker Swarm release to the OCI host.

Do not deploy production manually from this overview. Follow deploy/README.md for migration ordering, Infisical injection, health gates, rollback boundaries, and diagnostics.

To check when production last deployed or why merged changes have not released, follow Checking the Latest Production Deployment. Inspect Deploy Web's server job and correlate it with the running image; the procedure also checks whether automatic deployment is disabled.

Documentation Policy

docs/README.md separates current reference documentation from dated design and incident records. Update active docs whenever behavior, commands, or operations change. Preserve historical plans and incident evidence as records; do not treat them as current instructions.