Skip to content

docs: record verified analytics owner recovery - #3252

Merged
Asherlc merged 3 commits into
mainfrom
investigate-sentry-issue-7838431803
Nov 10, 2026
Merged

Asherlc merged 3 commits into
mainfrom
investigate-sentry-issue-7838431803

Conversation

@Asherlc

@Asherlc Asherlc commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Records DOFEK-SERVER-83, where an abandoned non-quiescent publication owner blocked analytics admission across worker replacements, and the subsequently approved production recovery.

Recovery stopped cooperating writers and verified executor/request termination before releasing only the recorded owner under its advisory lock. The pending intent and ingestion reservation were preserved for replay. The normal entrypoint then passed all 46 dbt models, recorded eight datasets with zero failures, refreshed 11 caches and released the owner normally. Original replica counts and images were restored; web returned HTTP 200 and analytics readiness reported healthy.

The original executor termination path remains unconfirmed, with deployment correlation documented. The separate sink database-connection failure still requires deployment of merged PR #3250. No runtime code, image, timeout, retry or healthcheck changed in this PR.

Validation: git diff --check passed. Runtime tests were skipped locally for the Markdown-only diff; hosted checks run normally. An explicit whole-file CSpell scan reported 457 historical findings outside the new entry; that documentation file is outside the configured CI spellcheck scope.

Summary by Sourcery

Document the approved, evidence-backed recovery and replay of a blocked analytics publication owner without changing runtime behavior.

Enhancements:

  • Document the verified production recovery of an analytics publication owner that remained blocked across worker replacements, including preservation and successful replay of the pending intent.
  • Record deployment correlation, validation evidence, unresolved termination diagnosis, and follow-up guidance for future owner recovery incidents.

Documentation:

  • Add a production incident baseline entry covering the analytics admission failure, controlled recovery, replay validation, service restoration, and remaining sink-connection follow-up.

@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Documentation

Records the analytics-owner incident, approved recovery and replay evidence, while keeping the unresolved executor termination cause and separate sink-connection follow-up distinct.

  • docs/production-incident-baseline.md L36446-36458

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 25 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3b0b961 and 3236db2.

Files selected for processing: 1

  • docs/production-incident-baseline.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.40%. Comparing base (3b0b961) to head (3236db2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3252      +/-   ##
==========================================
+ Coverage   96.96%   97.40%   +0.44%     
==========================================
  Files        1146     1146              
  Lines       63139    63139              
  Branches    19623    19623              
==========================================
+ Hits        61223    61501     +278     
+ Misses       1866     1592     -274     
+ Partials       50       46       -4     
Flag Coverage Δ
integration 56.93% <ø> (+9.05%) ⬆️
unit 91.23% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Asherlc Asherlc changed the title docs: record blocked analytics owner and recovery docs: record verified analytics owner recovery Nov 10, 2026
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

2 similar comments
@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@Asherlc
Asherlc marked this pull request as ready for review November 10, 2026 04:45
Copilot AI balanced review requested due to automatic review settings November 10, 2026 04:45
@sourcery-ai

sourcery-ai Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Sorry @Asherlc, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 2 days and 2 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@mergestorm-vortex

Copy link
Copy Markdown

Mergestorm reviews this repository's pull requests

Every PR gets AI code review that flags real bugs before they merge. This account's 30 free credits for this monthly window are used, so reviews are paused at 3236db2 and resume automatically on Nov 29 (UTC).

The Starter plan includes 120 credits a month.

Upgrade for more reviews →

@Asherlc
Asherlc merged commit fcf84da into main Nov 10, 2026
111 checks passed
@Asherlc
Asherlc deleted the investigate-sentry-issue-7838431803 branch November 10, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants