[Improve] Harden schema: host-aware uniqueness, composite webhook keys, retention, and classification constraints - #136
Conversation
…s, retention, and classification constraints - repositories: scope uniqueness to (provider, host, fullName/externalRepoId) with NULL hosts coalesced to '' so un-backfilled rows keep colliding - webhooks: delivery ids are now unique per (provider, delivery_id) instead of globally; drop the redundant provider index - webhooks: redact clearly sensitive payload fields before storage and add a daily WebhookCleanup BullMQ job honoring WEBHOOK_RETENTION_DAYS (default 30) - tasks/task_runs: CHECK constraints for workflow, surface, trigger, visibility, state, harness, requested-work, commit-author kind, run kind - tasks: rename ambiguous provider column to model_provider - migration 0006 + real-database tests covering every vocabulary value and the new uniqueness semantics
|
No code issues found. See task Reviewed the schema-hardening changes: host-aware repository uniqueness (NULL-host Notably solid: all webhook insert paths (GitHub/GitLab/Gitea/ADO share One deployment-time caveat (not a code defect): the |
Finishes the schema-hardening leftovers from finding 9 of the pre-release architecture audit (P1). Follows up on #100, #102, and #103.
What changed
1. Host-aware repository uniqueness
repositorieswas unique on(source_control_provider, full_name)and(source_control_provider, external_repo_id), so identical names/ids on different self-managed GitLab/Gitea/ADO hosts collided. Both unique indexes now include the host:repositories_provider_host_full_name_uniqueon(provider, coalesce(host, ''), full_name)repositories_provider_host_external_repo_uniqueon(provider, coalesce(host, ''), external_repo_id)hostis nullable until historical rows are backfilled, so NULLs are coalesced to''inside the index. That keeps un-backfilled rows colliding with each other exactly as strictly as before (a plain column index would treat NULLs as distinct and silently weaken the constraint), which also means the new indexes are strictly weaker than the old ones and cannot fail on existing data. All four production sync paths already stamp a non-null normalized host (github.comliteral;new URL(baseUrl).hostfor GitLab/Gitea/ADO); the repository factory now stampshosttoo so tests exercise the same shape. NoonConflictanywhere targets these indexes (verified: all repo upserts are find-then-update).2. Composite webhook delivery key
webhooks.delivery_idwas globally unique; delivery ids are only unique per provider (GitHub GUIDs, Linear delivery ids, sha256-of-body fallbacks for GitLab/Gitea/ADO). Now unique on(provider, delivery_id). BothrecordWebhookhelpers dedupe via untargetedON CONFLICT DO NOTHING, so no code change was needed there; the redundantwebhooks_provider_idxis dropped because the new unique index leads withprovider.3. Webhook payload redaction + retention
apps/api/src/handlers/webhook-payload-redaction.tsdeep-masks clearly credential-bearing keys (secret,token,password,credentials,authorization,*_secret,*_token,api_key,private_key, ... in snake/camel/kebab case) before the payload row is stored. Handlers still receive the original payload; only the stored audit copy is redacted. Free-text fields (comment bodies, PR descriptions) are intentionally kept.WebhookCleanupjob on the existingscheduled-jobsBullMQ queue deletes rows older thanWEBHOOK_RETENTION_DAYS(new env var, default 30). The batched delete lives inpackages/db/src/lib/webhook-retention.ts(deleteExpiredWebhooks) and is covered by a real-database test; the first run on a long-lived deployment drains the backlog in bounded batches.4. Database-level CHECKs for durable task classifications
Following the initiator-stamp CHECK pattern:
tasks:workflow,surface,trigger,visibility,state,harness,requested_work_kind,requested_work_kind_source,commit_author_kind(NULL-tolerant)task_runs:kind,harnessThe vocabulary source of truth stays in
@roomote/types; a real-database test (schema-constraints.test.ts) iterates every exported constant value and asserts Postgres accepts it, so extending a vocabulary without updating the CHECK (schema + migration) fails CI.run statuswas deliberately left unconstrained (mutable runtime lifecycle, not a durable classification).5.
tasks.providerrenamed totasks.model_providerConfirmed ambiguous: it is the inference-provider slot paired with
tasks.model(only production writer stamps the'opencode'constant; only reader is the instance telemetry report), while source-control/communication/compute providers all have their own typed columns. Renamed column + Drizzle property (modelProvider), plusDEFAULT_STANDARD_TASK_PROVIDER->DEFAULT_STANDARD_TASK_MODEL_PROVIDER. Not exposed in any API/frontend surface.Migration
0006_schema_hardening(+ snapshot, journal). Verified three ways against real Postgres 17:db:push:testparity for the test suite.Validation
packages/db: full suite (238 tests) including new real-DBschema-constraints.test.tsandwebhook-retention.test.tsapps/api: all webhook handler suites (223 tests) including new redaction coverageapps/bullmq(91),packages/cloud-agents(433),packages/env(65), targetedpackages/sdktitle-refreshpnpm lint:fast,pnpm check-types:fastclean.pnpm knipverified clean via a baseline diff and--no-gitignore(knip misbehaves inside.agents/worktrees/checkouts because the parent repo gitignores that path; it passes in a normal checkout).Notes for reviewers
host: 'github.com'(no GHES support today), so host-aware uniqueness currently only has teeth for GitLab/Gitea/ADO.