Skip to content

fix: precompute overview health sensor inputs - #3247

Merged
Asherlc merged 6 commits into
mainfrom
fix-never-loads
Nov 10, 2026
Merged

Asherlc merged 6 commits into
mainfrom
fix-never-loads

Conversation

@Asherlc

@Asherlc Asherlc commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

The Overview leaves health metrics and insights loading while request-time oxygen and WHOOP temperature queries scan 139–200 million raw rows; insights.compute times out after 120 seconds.

Move decoding and raw current-state selection into one bounded incremental dbt model. Shared API reads retain caller timezones, delayed measurement dates, replay deduplication and source priorities for web and mobile. Reuse the existing projection to find complete replacement keys before native FINAL; add a schema-only target migration and current-schema test fixtures.

Validation: 23,966 unit/mobile tests, 26 focused real-engine tests (including native dbt initial/incremental execution), lint, root/server/web typechecks and canonical spellcheck pass. After integrating main's account-erasure changes, all 26 focused database tests and root/server/web typechecks passed again. The raw-read regression changes from 100,063 rows to below 50,000 with 100,000 unrelated sensor rows. All checks on final head 90f19975aaa69324e914befe4f6ba427b73279b8 passed, including complete CI run 42718035207.

Production preparation completed on November 9 before merge: the channel projection covers all 23 current active parts; the schema-only target exists; 45 missing historical oxygen day markers were bootstrapped and bounded raw-key checks now find zero missing health markers. Candidate image revision f82d4ce44fcef0a7a593493bb65e6452c5883f22 ran 140 successful single-threaded dbt invocations, at most 32 days each. The final target contains 4,474 days and 240,898 typed samples, with 0 missing days and 0 dirty days. Production model queries peaked at 2.115 seconds, 907,671 rows read and 186 MiB query memory. No application rollout, timeout change or optimizer forcing was performed. See the cutover procedure and incident evidence.

The production outer FINAL plan pruned through all five replacement-key columns, selecting 48 of 25,130 granules for a representative temperature day. A serving-table smoke query over both typed channels completed in 51 ms. The fresh image scan passed after an earlier GHCR cached-layer 403 interruption, without changing permissions or weakening CI; its underlying registry cause remains unconfirmed. Future imports spanning more than one batch still require source-to-target watermark checks rather than treating one successful dbt invocation as complete freshness.

Immediately before merge, eight days had changed during CI. Candidate revision 90f19975aaa69324e914befe4f6ba427b73279b8, digest sha256:182e05e689caa67244c3348eccf250eb24ac2e4803f367bf948ad020f1cfb8cd, refreshed the model through the newly deployed shared analytics owner and stock owned-dbt artifact verification. The invocation succeeded in 0.41 seconds; the protected source-to-target check returned 4,474 days, 0 missing and 0 dirty. The projection check also returned zero missing parts. No application rollout was performed as an operator action; normal main CI/deployment follows the merge.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d21ce8f and 90f1997.

Files selected for processing: 24

  • analytics/README.md
  • analytics/models/read_models/daily_health_sensor.sql
  • analytics/models/read_models/read_model_microbatch.sql.test.ts
  • docs/clickhouse-read-model-deploy-runbook.md
  • docs/production-incident-baseline.md
  • docs/superpowers/plans/2026-11-09-overview-loading.md
  • docs/superpowers/specs/2026-11-09-overview-loading-design.md
  • entrypoint.sh
  • packages/server/src/repositories/daily-health-sensor-test-helpers.ts
  • packages/server/src/repositories/daily-health-sensor.integration.test.ts
  • packages/server/src/repositories/daily-metrics-repository.integration.test.ts
  • packages/server/src/repositories/daily-metrics-spo2-query.ts
  • packages/server/src/repositories/health-sensor-query.ts
  • packages/server/src/repositories/whoop-sensor-repository.integration.test.ts
  • packages/server/src/repositories/whoop-sensor-repository.ts
  • packages/server/src/routers/clickhouse-integration-test-helpers.ts
  • packages/server/src/routers/clickhouse-integration-test-models.ts
  • packages/server/src/routers/clickhouse-integration-test-read-models-b.integration.test.ts
  • packages/server/src/routers/clickhouse-integration-test-read-models-b.ts
  • packages/server/src/routers/correlation-range.test.ts
  • src/db/clickhouse-migrations/0111_daily_health_sensor.ts
  • src/db/clickhouse-migrations/registry.test.ts
  • src/db/clickhouse-migrations/registry.ts
  • src/processing/dataset-contracts.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@sourcery-ai

sourcery-ai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Schema and migrations

Add the persisted daily health-sensor model and its schema-only ClickHouse migration, defining the bounded refresh and typed sample representation that the API will read.

  • analytics/models/read_models/daily_health_sensor.sql L1-149
  • src/db/clickhouse-migrations/0111_daily_health_sensor.ts L1-22

Implementation

Switch oxygen and WHOOP temperature reads to the prepared samples while retaining the existing query-time grouping and selection behavior.

  • packages/server/src/repositories/daily-metrics-spo2-query.ts L14, L35-37, L40 (removed lines)
  • packages/server/src/repositories/health-sensor-query.ts L1-8
  • packages/server/src/repositories/whoop-sensor-repository.ts L7-15

Configuration

Register the new model with production analytics processing and the sleep-dashboard dbt build so it is maintained by the existing pipeline.

  • entrypoint.sh L22
  • src/db/clickhouse-migrations/registry.test.ts L19 (removed lines), L23
  • src/db/clickhouse-migrations/registry.ts L110, L223
  • src/processing/dataset-contracts.ts L41, L243-248

Documentation

Document the model, safe production cutover gates, preparation evidence, and implementation plan for future reviewers and operators.

  • analytics/README.md L3-12
  • docs/clickhouse-read-model-deploy-runbook.md L583-644
  • docs/production-incident-baseline.md L36294-36413
  • docs/superpowers/plans/2026-11-09-overview-loading.md L1-71
  • docs/superpowers/specs/2026-11-09-overview-loading-design.md L1-29

Tests

Exercise the model against ClickHouse, including decoding, corrections, deletion, bounded reads, native dbt builds, and API reads after removing access to the raw log.

  • packages/server/src/repositories/daily-health-sensor-test-helpers.ts L1-39
  • packages/server/src/repositories/daily-health-sensor.integration.test.ts L1-286

Update repository integration fixtures to build the prepared samples before asserting the existing oxygen and temperature results.

  • packages/server/src/repositories/daily-metrics-repository.integration.test.ts L14, L159
  • packages/server/src/repositories/whoop-sensor-repository.integration.test.ts L8, L68, L125

Add the health-sensor model to shared ClickHouse router fixtures and update correlation-query expectations to look for the serving table.

  • analytics/models/read_models/read_model_microbatch.sql.test.ts L130
  • packages/server/src/routers/clickhouse-integration-test-helpers.ts L47, L688-692
  • packages/server/src/routers/clickhouse-integration-test-models.ts L2, L43, L73-77
  • packages/server/src/routers/clickhouse-integration-test-read-models-b.integration.test.ts L8-13, L87-111
  • packages/server/src/routers/clickhouse-integration-test-read-models-b.ts L4-21
  • packages/server/src/routers/correlation-range.test.ts L73, L113

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Storybook previews for efca8641 are ready:

This comment updates automatically on each PR push.

@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.40%. Comparing base (d21ce8f) to head (90f1997).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3247   +/-   ##
=======================================
  Coverage   97.40%   97.40%           
=======================================
  Files        1144     1146    +2     
  Lines       63146    63163   +17     
  Branches    19617    19618    +1     
=======================================
+ Hits        61505    61522   +17     
  Misses       1595     1595           
  Partials       46       46           
Flag Coverage Δ
integration 56.78% <100.00%> (-0.01%) ⬇️
unit 91.23% <44.44%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@Asherlc
Asherlc marked this pull request as ready for review November 9, 2026 22:50
Copilot AI balanced review requested due to automatic review settings November 9, 2026 22:50
@sourcery-ai

sourcery-ai Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Sorry @Asherlc, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 17 hours by commenting @sourcery-ai review. Upgrade to get a review now.

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@mergestorm-vortex

mergestorm-vortex Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Mergestorm reviews this repository's pull requests

Every PR gets AI code review that flags real bugs before they merge. This account's 30 free credits for this monthly window are used, so reviews are paused at 90f1997 and resume automatically on Nov 29 (UTC).

The Starter plan includes 120 credits a month.

Upgrade for more reviews →

@codereviewbot-ai

Copy link
Copy Markdown

🤖 Review skipped: Repository rate limit exceeded. Free accounts are limited to 2 reviews per 4 hours per repository. Upgrade to a paid plan for unlimited reviews.

@Asherlc
Asherlc merged commit 127fb11 into main Nov 10, 2026
116 checks passed
@Asherlc
Asherlc deleted the fix-never-loads branch November 10, 2026 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants