Skip to content

fix: connect metric-stream sinks to Postgres - #3250

Merged
Asherlc merged 37 commits into
mainfrom
account-erasure-hardening
Nov 10, 2026
Merged

Asherlc merged 37 commits into
mainfrom
account-erasure-hardening

Conversation

@Asherlc

@Asherlc Asherlc commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

The live metric-stream consumer fails owned publication with Postgres ECONNREFUSED after #3234: its production service lacks DATABASE_URL and startup loads the loopback development default. The existing deployment job passed while the consumer was idle, so its successful stability check did not prove the new Postgres dependency.

Add one DATABASE_URL entry to the shared sink environment anchor using the existing required POSTGRES_PASSWORD and db:5432 endpoint. Legacy, live and history sinks inherit it. No runtime code, new secret, timeout, retry, health threshold, offset or manual production change.

Validation: stock workspace Compose rendering reproduces the omission in the original production stack; the corrected production and maintenance overlays resolve the exact synthetic connection for all three sinks; missing POSTGRES_PASSWORD fails interpolation. The CI yamllint command passed on the changed file, and git diff --check passed. Per repository policy, declarative configuration is verified by stock schema-aware tooling instead of static-config unit tests. Final CI and the canonical release must still pass; actual owned consumption and post-release task stability remain recovery gates.

Evidence: original maintenance release, incident record. A names-only live probe confirmed DATABASE_URL and POSTGRES_PASSWORD absent, and a bounded log window traced the first observed batch failure to withOwnedQueries. No production data or settings were changed during diagnosis.

Exact head 6879eec passed CI 42747662690: 75 successful jobs, 22 intentional skips, zero failures, including all four integration shards and full unit coverage. No unresolved review threads. Current main 127fb11 adds independent health-sensor models and leaves the sink environment, owner and callback unchanged; normal main CI remains the combined-tree deployment gate.

Retain Redpanda routes and marker contracts. Add typed activity aliases and real physical-erasure regressions. Record the proven publication race as a diagnostic patch; writer integration remains a release blocker.
Copilot AI balanced review requested due to automatic review settings November 10, 2026 03:10
@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Sorry @Asherlc, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 3 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 127fb11 and 6879eec.

Files selected for processing: 2

  • deploy/stack.yml
  • docs/production-incident-baseline.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Configuration

Add the existing required Postgres connection to the shared sink environment so all metric-stream sinks can reach the database.

  • deploy/stack.yml L415

Documentation

Record the incident’s confirmed root cause, the deployment fix and validation, and the remaining recovery gates.

  • docs/production-incident-baseline.md L36303-36309

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@mergestorm-vortex

Copy link
Copy Markdown

Mergestorm reviews this repository's pull requests

Every PR gets AI code review that flags real bugs before they merge. This account's 30 free credits for this monthly window are used, so reviews are paused at 6879eec and resume automatically on Nov 29 (UTC).

The Starter plan includes 120 credits a month.

Upgrade for more reviews →

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.40%. Comparing base (d21ce8f) to head (6879eec).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3250   +/-   ##
=======================================
  Coverage   97.40%   97.40%           
=======================================
  Files        1144     1144           
  Lines       63146    63146           
  Branches    19617    19617           
=======================================
  Hits        61505    61505           
  Misses       1595     1595           
  Partials       46       46           
Flag Coverage Δ
integration 56.78% <ø> (-0.01%) ⬇️
unit 91.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codereviewbot-ai

codereviewbot-ai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Review Summary (LGTM)

The changes provide the missing DATABASE_URL environment variable to the metric-stream ClickHouse sink services in deploy/stack.yml and record the incident baseline per repository guidelines.

Key Verifications

  1. Targeted fix: DATABASE_URL: postgres://health:${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required}@db:5432/health is added to the &metric-stream-clickhouse-sink-environment anchor, propagating cleanly to metric-stream-clickhouse-sink, metric-stream-live-clickhouse-sink, and metric-stream-history-clickhouse-sink.
  2. Fail-loud validation: Uses ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is required} to ensure services hard-fail immediately if configuration prerequisites are absent.
  3. Consistency: The user, host (db:5432), database (health), and credential interpolation align with the database container and other production services in deploy/stack.yml.
  4. Documentation: The entry added to docs/production-incident-baseline.md thoroughly documents the incident date, symptoms, root cause, validation, and retrospective without introducing ad-hoc resilience knobs.

🤖 Reviewed by codereviewbot.ai - Catch bugs before your team does.

@Asherlc
Asherlc merged commit 3b0b961 into main Nov 10, 2026
110 checks passed
@Asherlc
Asherlc deleted the account-erasure-hardening branch November 10, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants