Skip to content

fix(boot): non-fatal schema discovery, /health 503 with diagnostic - #125

Merged
EricAndrechek merged 15 commits into
mainfrom
issue-95
May 18, 2026
Merged

fix(boot): non-fatal schema discovery, /health 503 with diagnostic#125
EricAndrechek merged 15 commits into
mainfrom
issue-95

Conversation

@EricAndrechek

@EricAndrechek EricAndrechek commented May 12, 2026

Copy link
Copy Markdown
Member

Summary

Closes #95.

  • cmd/wavehouse no longer crash-loops when ClickHouse is unreachable on boot. The first SchemaRegistry.Refresh failure now sets a new api.BootState diagnostic instead of os.Exit(1); a goroutine calls the new SchemaRegistry.RetryRefresh (exponential backoff 2s → 60s) until success or shutdown.
  • While degraded, /health returns 503 {"status":"degraded","error":"schema discovery: …"} and /ready returns 503 {"status":"not ready", …} so an operator can curl /health to learn why the gateway isn't accepting traffic instead of grepping a restart-loop log. Once a Refresh succeeds, both flip back to 200 and stay there — transient runtime CH blips after boot are reflected in /ready only.
  • 12 new unit tests cover the retry loop (happy path, retries-until-success, context cancel, bounded backoff, nil-callback safety, clamped invalid bounds) and the health handler (BootState get/set, Liveness degraded → ready transition, Readiness boot-degraded reporting). make ci passes locally: combined coverage 81.2% (≥ 80% gate), discovery 98.1%, all per-suite gates green, all 30 E2E SDK tests pass.

Test plan

  • make verify (tidy + fmt + vulncheck + lint)
  • make test-unit (unit gate: 74.5% ≥ 70%)
  • make test-sdk, make test-integration, make test-e2e
  • make ci end-to-end: combined gate 81.2% ≥ 80%
  • E2E SDK suite confirms the happy-path boot (ClickHouse up at start) still runs the synchronous initial Refresh and is fully healthy by the time tests run
  • Manual boot test against a stopped ClickHouse instance — curl /health returns 503 with diagnostic, then 200 once CH is brought up (not run locally; reviewers welcome to spot-check via make deps-down && bin/wavehouse in a separate terminal)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Startup no longer crashes when schema discovery fails: the server stays online. While background retries with exponential backoff run, /health returns 503 (degraded) and /ready returns 503 (not ready); both flip to 200 once discovery succeeds.
  • Documentation

    • Updated API, architecture, deployment, and changelog to describe boot-time degraded mode and recommended Kubernetes startup/liveness/readiness probe configuration.

Review Change Stack

Previously, any error from the initial SchemaRegistry.Refresh in
cmd/wavehouse — connection-refused, missing database, transient blip —
called os.Exit(1). The supervisor would restart the process every ~10s
in an unbounded loop, port 8080 never bound, and operators got
`connection refused` on probes even though ClickHouse was otherwise
healthy. The binary was unrecoverable without intervention.

Now the first Refresh failure is non-fatal: a new api.BootState records
the diagnostic, the server still binds :8080, and a background
goroutine calls SchemaRegistry.RetryRefresh (exp backoff 2s → 60s)
until success or shutdown. While BootState.Err() != nil, /health
returns 503 with {"status":"degraded","error":"..."} so operators can
curl the endpoint instead of grepping a restart-loop log; /ready does
the same. Once a Refresh succeeds, BootState.Set(nil) flips both
endpoints back to 200 and stays there — transient runtime CH blips
after that point are reflected in /ready only.

Closes #95.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the boot-time reliability of the application by making initial schema discovery non-fatal. Instead of exiting when ClickHouse is unreachable, the service now enters a degraded state and retries the connection in the background. This ensures the process remains running and provides clear diagnostic information via health endpoints, eliminating the need for operators to rely on restart-loop logs to debug startup connectivity issues.

Highlights

  • Boot-time Resilience: Modified schema discovery to be non-fatal, preventing the binary from crash-looping when ClickHouse is unreachable at startup.
  • Diagnostic Health Endpoints: Updated /health and /ready endpoints to return 503 with diagnostic messages during the boot retry phase, allowing operators to identify startup issues.
  • Exponential Backoff: Implemented a new RetryRefresh mechanism with exponential backoff (2s to 60s) to handle transient startup failures gracefully.
  • Comprehensive Testing: Added 12 new unit tests covering the retry loop, context cancellation, and health handler state transitions.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@github-actions github-actions Bot added documentation Improvements or additions to documentation go Pull requests that update go code area/api HTTP handlers, routing, middleware area/query Structured query AST, SQL builder area/docs Documentation, site/, README labels May 12, 2026
@github-actions
github-actions Bot requested a review from taitelee May 12, 2026 22:36
Comment thread internal/discovery/discovery_test.go Outdated
@claude

claude Bot commented May 12, 2026

Copy link
Copy Markdown

0 [MUST] · 1 [SHOULD] · 0 [MAY] — see inline thread for detail.

The implementation is clean: BootState is a well-scoped, mutex-guarded diagnostic; RetryRefresh is correct (context cancel, backoff clamping, nil-callback safety all handled); the goroutine in main.go ties cleanly to the process-lifetime context via defer cancel(); and the two concurrent refreshers (RetryRefresh + StartAutoRefresh) coexist safely through the existing sync.RWMutex on SchemaRegistry. Doc-sync is complete — docs/api.md, docs/architecture.md, docs/deployment.md, and CHANGELOG.md all updated. README.md doesn't document health probe semantics, so no update needed there.

The single [SHOULD] is a test-only style nit: TestRetryRefresh_ReturnsOnContextCancel uses time.Sleep(50ms) for goroutine sync, which violates the project convention (AGENTS.md: "use channel sync not goroutine scheduling assumptions"). The sleep is also redundant — the test is correct without it.

Ship it — address or push back on the inline thread before merge.

@claude

Per Claude review on #125: TestRetryRefresh_ReturnsOnContextCancel
used a 50ms sleep before cancel() that violated AGENTS.md ("use channel
sync not goroutine scheduling assumptions") and was unnecessary for
correctness — `done` is the only sync we need, and the RetryRefresh
loop observes ctx.Done() whether cancel fires before or after the
goroutine enters its select. Verified with `go test -race -count=10`.

Also corrects a stale doc comment on HealthHandler.Boot that said
"consulted by Liveness" — Readiness consults it too, by design (a
kubelet readiness probe should see "not ready" while boot is failing).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request makes boot-time schema discovery non-fatal, preventing the binary from crash-looping if ClickHouse is unreachable at startup. It introduces a BootState to track startup diagnostics, an exponential backoff retry loop in the SchemaRegistry, and updates the /health and /ready endpoints to surface a 503 'degraded' status during the boot process. Feedback focuses on documenting these architectural changes in AGENTS.md per the style guide, sanitizing error messages in health probes to prevent sensitive data exposure, and optimizing the retry loop by using time.NewTimer to avoid potential memory leaks.

Comment thread cmd/wavehouse/main.go
Comment thread internal/api/health.go
Comment thread internal/api/health.go
Comment thread internal/discovery/discovery.go
Per Gemini's review on #125. Boot-time degraded mode + retry loop is
a meaningful design decision worth documenting alongside the other
'Async ingestion', 'DLQ', 'Active Sweeper' style entries — future
agents working on cmd/wavehouse will want to know /health is allowed
to 503 by design during initial Refresh.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Comment thread docs/deployment.md Outdated
Comment thread internal/discovery/discovery_test.go Outdated
@claude

claude Bot commented May 12, 2026

Copy link
Copy Markdown

1 [MUST] · 1 [SHOULD] · 0 [MAY] — see inline threads for detail.

The core implementation is solid: BootState is correctly mutex-guarded, RetryRefresh handles context cancellation, nil callback, and backoff clamping properly, the goroutine in main.go ties to the process-lifetime context via defer cancel(), and both concurrent refreshers coexist safely through SchemaRegistry's existing sync.RWMutex. Doc-sync coverage (AGENTS.md KDD #15, CHANGELOG, api.md, architecture.md, deployment.md) is complete.

Inline threads:

  • docs/deployment.md line 235 — [MUST] K8s liveness probe interaction
  • internal/discovery/discovery_test.go line 529 — [SHOULD] 1s wall-clock sleep in unit suite

Iterate — the [MUST] must be addressed before merge.

The most important thing: the deployment doc claims "The supervisor (docker, systemd, k8s) won't restart the binary every ~10s" but this is false for Kubernetes when /health is used as a livenessProbe (which is exactly what the K8s example in that same file shows). With /health returning 503 for potentially minutes during boot retries, K8s will kill and restart the pod after failureThreshold consecutive liveness failures (default 30s) — recreating the crash loop problem at a slower cadence. The fix is a startupProbe in the K8s example, or restricting the 503 to /ready only. See the inline thread for a concrete suggestion.

@claude

Two Claude-review fixes from #125:

1) [MUST] docs/deployment.md previously claimed the binary "won't
   restart" — false for Kubernetes when /health is wired to livenessProbe
   (the example we ship), since kubelet still kills the pod after
   failureThreshold × periodSeconds. The retry-loop window can exceed
   that, recreating the same restart problem the PR is solving. Fixed by
   (a) softening the prose to be accurate about the binary-vs-orchestrator
   distinction and (b) adding a startupProbe to the K8s example so
   liveness and readiness are gated until the first schema discovery
   succeeds. Docker HEALTHCHECK is unaffected (doesn't restart by default).

2) [SHOULD] TestRetryRefresh_ClampsInvalidBackoffs slept ~1s in the unit
   suite to observe the clamp-to-default behaviour. Extracted the clamp
   into a private clampBackoff helper so the invariant is now testable in
   microseconds via TestClampBackoff (6 table-driven cases). Old slow
   test removed; discovery coverage rose 89.6% → 89.8%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented May 12, 2026

Copy link
Copy Markdown

0 [MUST] · 0 [SHOULD] · 0 [MAY] — no open findings on HEAD `2144b3a`.

Both items from the previous review have been resolved:

  • [MUST] docs/deployment.md — `startupProbe` added to K8s example with correct `failureThreshold: 30` and explanatory prose. The "supervisor won't restart" claim is now accurate.
  • [SHOULD] internal/discovery/discovery_test.go — `time.Sleep(50ms)` replaced with channel sync (`done <- sr.RetryRefresh(...)` + `select { case err := <-done: ... }`); the test is now correct without scheduling assumptions.

Ship it — address or push back on any open inline threads before merging (ruleset requires resolution), then this is good to go.

@claude

@coderabbitai

coderabbitai Bot commented May 13, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Schema discovery boot failures are now non-fatal. The process binds :8080, tracks degradation in BootState, returns 503 from /health and /ready with diagnostics while RetryRefresh runs with exponential backoff, and clears BootState on success so probes return 200 thereafter.

Changes

Non-fatal Boot with Diagnostic Health Probes

Layer / File(s) Summary
BootState type and health probe integration
internal/api/health.go, internal/api/health_test.go
New BootState concurrency-safe type and Boot field on HealthHandler; Liveness and Readiness return 503 with diagnostic JSON when BootState.Err() is non-nil. Tests cover BootState lifecycle and probe behavior during boot degradation.
Retry mechanism with exponential backoff
internal/discovery/discovery.go, internal/discovery/discovery_test.go
clampBackoff normalizes durations; SchemaRegistry.RetryRefresh retries Refresh with exponential backoff up to maxBackoff, calls optional onAttempt after failures, returns nil on success or ctx.Err() on cancellation. Tests validate retry sequencing, cancellation, bounding, and nil-callback safety.
Boot orchestration in main & runHealthCheck tests
cmd/wavehouse/main.go, cmd/wavehouse/health_test.go
Creates an early process-lifetime context, performs initial registry.Refresh(ctx), records boot degradation in bootState on failure, launches asynchronous RetryRefresh, clears bootState on success and then starts auto-refresh; wires a single preconfigured HealthHandler with BootState. Command tests assert runHealthCheck exit-code behavior for 200/503/connection-refused/invalid-port.
Boot chain and integration resilience tests
internal/api/boot_chain_test.go, tests/integration/boot_resilience_test.go
Adds end-to-end unit and integration tests that exercise failure-then-recover boot flow, assert /health reports degraded until schema discovery succeeds, assert sticky /health vs conditional /ready across ClickHouse stop/start cycles, and verify expected retry counts.
Documentation and design notes
AGENTS.md, CHANGELOG.md, docs/src/content/docs/api.md, docs/src/content/docs/architecture.md, docs/src/content/docs/deployment.md
Adds design decision and changelog entry for non-fatal boot. API/docs updated: /health returns 200 only after initial schema discovery and 503 with diagnostic JSON during boot retries; /ready requires boot completion and ClickHouse reachability. Deployment doc includes Kubernetes startupProbe guidance and YAML examples.
Minor MQ TODOs
internal/mq/embedded.go
Two TODO comments added for future JetStream improvements (reserved-space validation and sync_interval configurability).

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant HealthHandler
  participant BootState
  participant SchemaRegistry
  participant ClickHouse
  Client->>HealthHandler: GET /health or GET /ready
  HealthHandler->>BootState: Err()
  alt BootState.Err() != nil
    BootState-->>HealthHandler: diagnostic error
    HealthHandler-->>Client: 503 with diagnostic JSON
  else
    BootState-->>HealthHandler: nil
    HealthHandler->>SchemaRegistry: (for readiness) ping / check
    SchemaRegistry->>ClickHouse: Query schema / ping
    ClickHouse-->>SchemaRegistry: response
    SchemaRegistry-->>HealthHandler: readiness result
    HealthHandler-->>Client: 200 or 503 based on readiness
  end
  Note over SchemaRegistry,ClickHouse: On boot failure, main starts RetryRefresh(ctx) in background\nSchemaRegistry->>SchemaRegistry: RetryRefresh with exponential backoff
  SchemaRegistry->>ClickHouse: Retry Refresh attempts
  ClickHouse-->>SchemaRegistry: fail/succeed responses
  SchemaRegistry-->>BootState: on success clear BootState (Set(nil))
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested labels

area/infra

Suggested reviewers

  • taitelee

Poem

🐰 I hopped through boot where errors lay,
Kept sockets warm and served the day,
Five-oh-three sang the warning tune,
Retries counted under the moon,
Then two-oh-oh — the garden blooms.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 53.57% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'fix(boot): non-fatal schema discovery, /health 503 with diagnostic' clearly and concisely summarizes the main change: making schema discovery non-fatal and providing health diagnostics via 503 responses.
Linked Issues check ✅ Passed All primary objectives from issue #95 are met: schema discovery is non-fatal with exponential backoff retry [#95], port 8080 binds during degradation [#95], /health and /ready return 503 with diagnostics while degraded [#95], /health becomes sticky at 200 after recovery while /ready reflects runtime CH reachability [#95], and comprehensive test coverage validates these behaviors.
Out of Scope Changes check ✅ Passed All code changes are directly scoped to the boot-time schema discovery retry objective: boot state tracking, health handler updates, retry logic, comprehensive tests, and documentation updates align with issue #95 scope without unrelated modifications.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-95

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CHANGELOG.md`:
- Line 9: The Markdown heading "### Changed" currently lacks required
surrounding blank lines; edit CHANGELOG.md to insert one blank line immediately
before and one blank line immediately after the "### Changed" heading so it has
a blank line above and below, satisfying the MD022 lint rule.
- Line 10: Update the wording in CHANGELOG.md to remove the contradiction:
change the sentence that currently claims both endpoints "stay 200 for the rest
of the process lifetime" to clarify that once BootState.Set(nil) succeeds the
/health endpoint remains sticky-200 for the rest of the process lifetime while
/ready remains conditional on current ClickHouse reachability (i.e.,
BootState.Err() is nil makes /health 200 permanently but /ready can still return
503 on transient runtime blips); reference SchemaRegistry.RetryRefresh and
BootState.Set(nil)/BootState.Err() in the sentence to make the behavior
explicit.

In `@cmd/wavehouse/main.go`:
- Around line 123-147: The current startup can run registry.RetryRefresh and
registry.StartAutoRefresh concurrently causing overlapping Refresh calls and
stale bootState; ensure StartAutoRefresh is only started after boot recovery
(i.e., after a successful registry.Refresh or when the retry loop succeeds) or
make any successful Refresh (including the periodic auto-refresh) clear
bootState; specifically, move or guard the call to registry.StartAutoRefresh so
it only executes after bootState is cleared (or call bootState.Set(nil) inside
the success path of the periodic refresh handler), referencing registry.Refresh,
registry.RetryRefresh, registry.StartAutoRefresh and bootState.Set to locate and
update the logic.

In `@docs/api.md`:
- Line 86: Update the phrasing in the `/health` docs to say the gateway “isn't
ready to serve traffic yet” instead of “hasn't bound traffic yet”; edit the
sentence describing boot-degraded behavior (mentioning boot-degraded, `/health`,
and port `:8080`) so it clarifies that the service binds `:8080` and serves
diagnostics but is not yet ready to serve traffic, keeping the rest of the
explanation about schema discovery backoff and `/ready` behavior unchanged.

In `@internal/api/health_test.go`:
- Around line 100-163: Replace the manual JSON parsing and assertions in
TestHealth_Liveness_BootDegraded, TestHealth_Liveness_BootReadyFlipsTo200, and
TestHealth_Readiness_BootDegradedReports503 with the shared test helpers: call
testutil.AssertJSONContains(t, w, http.StatusServiceUnavailable, "connection
refused") (or the appropriate status/substring) for the degraded/error checks
and use testutil.AssertJSONResponse(t, w, http.StatusOK,
map[string]string{"status":"ok"}) (or matching expected map) for the OK
readiness case; update the calls around h.Liveness and h.Readiness invocations
and remove the json.Unmarshal/resp variable and individual header/status
assertions that these helpers cover.

In `@internal/discovery/discovery_test.go`:
- Around line 404-509: These four separate tests for RetryRefresh
(TestRetryRefresh_SucceedsOnFirstAttempt, TestRetryRefresh_RetriesUntilSuccess,
TestRetryRefresh_ReturnsOnContextCancel, TestRetryRefresh_BackoffIsBounded and
the NilOnAttempt case) should be collapsed into a single table-driven suite:
create tests := []struct{name string; errs []error; initial time.Duration; max
time.Duration; setupCtx func() (context.Context, func()); wantErr bool;
onAttempt func(error); assertions func(t *testing.T, connFake interface{},
captured []error, start time.Time)} and iterate with for _, tt := range tests {
t.Run(tt.name, func(t *testing.T){ t.Parallel(); sr, conn := newFakeRegistry(t,
tt.errs); ctx, cancel := tt.setupCtx(); defer cancel(); start := time.Now(); var
captured []error; err := sr.RetryRefresh(ctx, tt.initial, tt.max, func(e error){
if tt.onAttempt!=nil { tt.onAttempt(e) } captured = append(captured, e) }); if
tt.wantErr { require.Error(t, err) } else { require.NoError(t, err) };
tt.assertions(t, conn, captured, start) }) }, ensuring each original assertion
(call counts, captured errors, elapsed bounds, context cancellation behavior,
nil onAttempt safety) is implemented in the corresponding test case entry and
referencing RetryRefresh and newFakeRegistry to locate the logic.
- Around line 494-497: The wall-clock upper bound in the test is too tight and
causes CI flakiness; in the block using start and elapsed (computed via elapsed
:= time.Since(start)) update the assertion that checks the upper bound
(currently assert.Less(t, elapsed, 100*time.Millisecond)) to a more permissive
threshold (e.g., 200–250ms) so slow shared runners won't fail while keeping the
lower bound assert.GreaterOrEqual(t, elapsed, 10*time.Millisecond) intact;
adjust only the upper-bound value in the assertion that references elapsed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a78f2fb3-1182-4e2b-a2fa-a4fbba952fa4

📥 Commits

Reviewing files that changed from the base of the PR and between 18303b4 and 2144b3a.

📒 Files selected for processing (10)
  • AGENTS.md
  • CHANGELOG.md
  • cmd/wavehouse/main.go
  • docs/api.md
  • docs/architecture.md
  • docs/deployment.md
  • internal/api/health.go
  • internal/api/health_test.go
  • internal/discovery/discovery.go
  • internal/discovery/discovery_test.go

Comment thread CHANGELOG.md
Comment thread CHANGELOG.md Outdated
Comment thread cmd/wavehouse/main.go
Comment thread docs/api.md Outdated
Comment thread internal/api/health_test.go
Comment thread internal/discovery/discovery_test.go
Comment thread internal/discovery/discovery_test.go Outdated
@github-project-automation github-project-automation Bot moved this from Backlog to Ready in WaveHouse Task Board May 13, 2026
@EricAndrechek EricAndrechek moved this from Ready to In review in WaveHouse Task Board May 13, 2026
EricAndrechek and others added 2 commits May 13, 2026 16:29
Resolves AGENTS.md KDD collision: keep main's #15 Observability invariants
and #16 Bearer-token CORS posture; renumber non-fatal boot to #17. Docs
auto-merged from docs/*.md → docs/src/content/docs/*.md (Astro restructure
landed on main in #7).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four CodeRabbit findings on commit 2144b3a:

1) [Minor] CHANGELOG MD022 + sticky-/health vs conditional-/ready wording.
   The previous prose said both endpoints "stay 200 for the rest of the
   process lifetime" then contradicted itself by noting /ready reflects
   transient blips. Rewrote to make explicit: /health is sticky-200 after
   the first Refresh success (it answers "did boot complete once"); /ready
   remains conditional on current ClickHouse reachability (it answers "can
   I serve traffic right now"). Also added the missing blank line above
   ### Changed and updated the doc paths to the post-Astro-merge locations.

2) [Major] StartAutoRefresh + RetryRefresh race. Two goroutines were
   calling Refresh concurrently during boot (mutex-safe, but stale
   bootState if the auto-refresh tick succeeded first). Restructured so
   StartAutoRefresh only starts after the first successful Refresh (sync
   or retry). One Refresh caller at a time, no stale-bootState window.

3) [Minor] /health docs bind-state phrasing — "hasn't bound traffic yet"
   was inaccurate (the binary IS bound on :8080 in boot-degraded mode, it
   serves /health diagnostics — it just isn't accepting ingest/query
   traffic). Reworded to "isn't ready to serve traffic yet" with an
   explicit clarifying sentence.

4) [Major] TestRetryRefresh_BackoffIsBounded wall-clock upper bound
   tightened to 100ms was too aggressive for shared CI runners — the
   15ms-of-real-sleeps budget can stretch past 100ms under scheduler
   pressure. Relaxed to 250ms, which still catches a real unbounded-
   backoff regression (orders-of-magnitude blowup) without flaking.

Pushed back on two:
- testutil.AssertJSONResponse/AssertJSONContains aren't a clean fit for
  these tests (substring-on-error-message + deliberate header coverage
  on the 503 branches — see TestHealth_Readiness_PingFails comment).
  Existing health_test.go tests use the same manual-parsing pattern; one
  consistent style across the file > selective helper adoption.
- 5-into-1 table-driven consolidation of the RetryRefresh tests would
  produce a struct with mostly-nil optional fields per row (different
  setup, different assertions, ctx-cancel needs goroutine+chan
  orchestration). Current form is short, top-down readable. The neighbour
  TestClampBackoff IS table-driven (6 pure-fn input/output rows) — used
  where it actually fits.

All four Gemini concerns from the earlier review remain resolved
(Gemini explicitly acknowledged each pushback in the merged review
threads on 2c57041 — sanitization, time.After memory leak, AGENTS.md
placement). make verify + make test-unit pass; discovery coverage holds
at 89.8%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@coderabbitai coderabbitai Bot added the area/infra CI, build, deploy, Docker, release label May 13, 2026
@claude

claude Bot commented May 13, 2026

Copy link
Copy Markdown

0 [MUST] · 0 [SHOULD] · 0 [MAY] — no open findings on HEAD dcb452b.

All items from prior review cycles are resolved:

  • Prior Claude [MUST]: K8s livenessProbe restart-loop — fixed with startupProbe in deployment doc.
  • Prior Claude [SHOULD]: time.Sleep scheduling assumption in cancel test — replaced with channel sync.
  • Gemini: AGENTS.md KDD entry, error exposure posture, time.After memory concern — all addressed or pushed back with sound reasoning.
  • CodeRabbit [Major]: StartAutoRefresh + RetryRefresh race — fixed in dcb452b by gating StartAutoRefresh until after the first successful Refresh (sync or retry path).
  • CodeRabbit [Major]: CHANGELOG wording (/health sticky-200 vs /ready conditional) and MD022 — fixed.
  • CodeRabbit two stylistic pushbacks (testutil helpers, table-driven consolidation) — pushback reasoning is sound and consistent with existing test style in the file.

Implementation notes for the record: BootState is correctly mutex-guarded; RetryRefresh context cancellation and nil-callback paths are handled and tested; the two concurrent-refresher paths (RetryRefresh goroutine / StartAutoRefresh) are now mutually exclusive by construction; process-lifetime context is wired before the retry goroutine is spawned so shutdown is clean. Doc sync covers AGENTS.md §KDD #17, CHANGELOG, docs/api.md, docs/architecture.md, docs/deployment.md — complete per the AGENTS.md sync table.

Ship it — resolve any open CodeRabbit threads before merge (ruleset requires thread resolution), then this is good to go.

@claude

@github-actions github-actions Bot removed the area/infra CI, build, deploy, Docker, release label May 13, 2026
Comment thread cmd/wavehouse/main.go Outdated
taitelee
taitelee previously approved these changes May 15, 2026

@taitelee taitelee left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New bootstrap architecture that allows the server to start and answer health probes even if ClickHouse is initially unreachable. With a new background RetryRefresh loop with exponential backoff, WaveHouse avoids immediate startup crashes and remains in a "degraded" state until the database becomes available. The main.go file now properly handles this lifecycle by injecting a bootState into the health handler and making sure background tasks are tied to a process wide context for clean shutdowns. New unit tests added to confirm that backoff intervals are capped and that the system gracefully handles service interruptions without data loss or logging noise.

EricAndrechek and others added 2 commits May 18, 2026 09:17
EnsureDLQStream, policy.NewStore, and pipes.NewStore previously took
context.Background(). Since #125 creates the process-lifetime ctx earlier
(for the schema discovery retry goroutine), these three init calls can
now share it — letting SIGINT/SIGTERM cancel them cleanly rather than
relying on Docker SIGKILL as the only escape.

All three operate against embedded JetStream (in-process), so the
practical risk of a hang is near zero — but the inconsistency with the
surrounding ctx usage was noted in PR #125 review (taitelee) and the
change is mechanical with zero behavioral impact in the happy path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@EricAndrechek
EricAndrechek dismissed stale reviews from taitelee and coderabbitai[bot] via 04709d8 May 18, 2026 13:19
@EricAndrechek
EricAndrechek enabled auto-merge (squash) May 18, 2026 13:21
@EricAndrechek
EricAndrechek requested a review from taitelee May 18, 2026 13:21
@claude

claude Bot commented May 18, 2026

Copy link
Copy Markdown

Review — 0 [MUST], 0 [SHOULD], 0 [MAY]

Re-review against HEAD 04709d8 ("refactor(main): tie KV/DLQ inits to process-lifetime ctx"). No new inline threads to open.

What the final commit adds: moves ctx, cancel creation to before NATS/DLQ/policy/pipes initialization (was after hub creation). EnsureDLQStream, policy.NewStore, and pipes.NewStore now use the process-lifetime context instead of context.Background(). This is strictly correct — if a SIGTERM arrives during any of those init calls they will now cancel cleanly, and the retry goroutine's ctx.Done() channel is wired from the moment it spawns rather than from when the signal handler registers. No ordering hazard: StartAutoRefresh only touches the ClickHouse conn and the in-memory schema map, not NATS, so the retry goroutine running concurrently with NATS init is safe.

Everything I re-checked against AGENTS.md:

  • CorrectnessBootState protected by sync.RWMutex. StartAutoRefresh gated correctly: success path starts it inline; retry path starts it only after RetryRefresh returns nil — no overlap possible. defer cancel() covers both normal return and early error returns, so the retry goroutine always observes shutdown. clampBackoff prevents busy-loop on zero/negative bounds. Context cancellation short-circuits the onAttempt callback (shutdown-noise guard) and the select exits immediately via ctx.Done().
  • Security — no new surface. The two advisory items (raw CH error in unauthenticated /health, time.After timer leak) remain documented and accepted by the author; both were acknowledged in prior review rounds and neither is a blocking concern at this call-path's scope.
  • Performance — retry loop touches only the boot phase; no hot-path allocations.
  • Testing — 12 unit tests (BootState, RetryRefresh variants, health degraded/ready transitions), boot_chain_test.go wiring test with production types, Docker HEALTHCHECK probe tests, and the 4-row sticky-health-vs-conditional-ready matrix against a real testcontainer. The concurrentBuffer in discovery_test.go correctly synchronises slog handler writes under -race.
  • Doc syncdocs/api.md, docs/architecture.md, docs/deployment.md, CHANGELOG.md, and AGENTS.md all updated. K8s startupProbe example and boot-degraded mode section are accurate.
  • Coverage scriptmeetsThreshold integer-arithmetic gate is mathematically correct (covered*100 >= total*threshold is exact at the boundary). Per-suite excludes correctly apply only to per-suite renders, not to the merged total, so cmd/wavehouse/main.go's e2e coverage still counts toward the project-wide gate.

Ship it — architecture is sound, wiring is correct, coverage is strong, all required doc updates are present. Prior advisory threads (timer leak, error sanitization) are tracked and the author's documented decision to accept both stands.

@EricAndrechek EricAndrechek moved this from In progress to Ready in WaveHouse Task Board May 18, 2026
@github-project-automation github-project-automation Bot moved this from Ready to In progress in WaveHouse Task Board May 18, 2026
@EricAndrechek
EricAndrechek merged commit f8aa173 into main May 18, 2026
12 checks passed
@EricAndrechek
EricAndrechek deleted the issue-95 branch May 18, 2026 13:58
@github-project-automation github-project-automation Bot moved this from In progress to Done in WaveHouse Task Board May 18, 2026
EricAndrechek added a commit that referenced this pull request May 18, 2026
…ator

After merging main's boot-resilience and health work (#125, #122), e2e
coverage dipped to 49.9% (gate is 50%) because the Readiness handler and
the cmd/wavehouse/health.go probe binary were uncovered by the SDK
harness. Both are production code paths the operator-facing contract
(k8s readiness, Docker HEALTHCHECK) depends on — exercising them in the
e2e harness is principled, not a coverage hack.

Brings e2e from 49.9% → 50.9%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EricAndrechek added a commit that referenced this pull request May 20, 2026
## Summary

Umbrella PR setting up shared Claude Code + AI agent infrastructure for
the WaveHouse team. Two work streams:

1. **AI rules drift cleanup** — corrected stale references that AI tools
(Claude Code, Gemini Code Assist, Copilot, CodeRabbit) were following
blindly.
2. **Claude Code native tooling** — committed `.claude/` configuration
and `.githooks/` so every teammate gets identical dev affordances out of
the box, with agent-specific gating layered on top.

The team just got Max 20x subscriptions across the board; this lands the
team-wide config so everyone is on the same agentic dev experience by
default.

## Scope

### 1. AI rules drift cleanup

- **17 doc-path references corrected** across AGENTS.md +
CONTRIBUTING.md to `docs/src/content/docs/*.md` (the actual Astro
Starlight location, not the old flat layout).
- **`.github/copilot-instructions.md` shrunk to a pointer** — was
drifting on Go 1.25 (vs current 1.26.3) and 60% coverage (vs current 80%
total / 70% unit per `.testcoverage.yml`).
- **`.gemini/styleguide.md`** — stale `#67` / 60% claim fixed (issue
closed, 70% restored); duplicated doc-sync bullet collapsed to defer to
AGENTS.md (already authoritative).
- **`.github/labeler.yml`** — dropped non-existent
`cmd/wavehouse-{api,worker}/**` entries; fixed
`tests/{compose.yaml,sdk/**}` → `tests/e2e/...`; added
`cmd/wavehouse/**` to `area/infra`.
- **`.github/prompts/pr-review.md`** — doc-sync list collapsed;
vestigial "tenant" wording dropped (no tenant model in WaveHouse);
hard-wrap reflowed (180 → 81 lines).
- **AGENTS.md** — `cmd/*/main.go` (plural) → `cmd/wavehouse/main.go`
(one binary); `tests/fixtures/` → `tests/e2e/fixtures/`; removed stale
"update `triage.yml` area enumeration" step (workflow now discovers
`area/*` labels dynamically); fixed internal-package count.
- **CONTRIBUTING.md** — vestigial "tenant isolation" wording removed.
- **TODO.md deleted** — audit summary below.

### 2. Claude Code native tooling

- **`.claude/`** — shared configuration: `settings.json` (deny rules +
worktree config + three hooks wired), `agents/pre-push-reviewer.md`,
`hooks/agent-bash-gate.sh` (PreToolUse Bash gate),
`hooks/review-marker.sh` (PostToolUse Agent marker writer),
`hooks/gofumpt-on-save.sh` (auto-format), `skills/pr-review-locally/`,
`skills/pr-sync-with-main/`, `commands/cover.md`.
- **`.githooks/`** — universal team hooks installed by `make tools`:
`pre-commit` runs `make verify`; `pre-push` requires
`tmp/ci-passed-<HEAD-sha>` marker (written by `make ci`).
- **`.config/wt.toml`** — worktrunk project hooks so parallel-agent
worktrees install `.githooks/` correctly.
- **AGENTS.md §"Agent PR Discipline"** — new section codifying the
agent-only ruleset:
- Drafts-only PR creation; human-only
ready/approve/request-changes/reviewer-add transitions.
- Bot reviewer re-triggers go through PR comments (`@coderabbitai
review`, `@gemini-code-assist`, `@claude` / `/review`).
- **Pre-push self-review mandatory** on PR branches: agent invokes
`pre-push-reviewer` subagent in fresh context. `ship_it` requires zero
findings at any severity — any `[MUST]` / `[SHOULD]` / `[MAY]` forces
iterate; the orchestrator loops review → fix → review until clean.
- **Honest-agent marker policy**: `--no-verify` regex-blocked + the
obvious marker-write idioms denied at the permission layer (`Bash(touch
tmp/ci-passed:*)`, `Write`/`Edit` on the canonical paths); everything
else is a documented rule, not regex-enforced. Bash can write a file by
a dozen paths and regex enforcement is a porous game of whack-a-mole.
- **`docs/src/content/docs/claude-code.md`** — contributor-facing page
documenting the four-layer model (universal git hooks → agent gate →
ergonomic hooks → skills/agents/commands), quick setup, and discipline
rules.
- **CHANGELOG.md** — `[Unreleased]` entry covering all of the above.

## Out-of-tree GitHub changes that pair with the AI-rules cleanup

Done via `gh` CLI as part of the same audit:

- **Closed #46** (Graceful Shutdown) — verified shipped in
`cmd/wavehouse/main.go:378-393` (SIGINT/SIGTERM → bounded shutCtx →
ingestStream.Stop → srv.Shutdown → promSrv.Shutdown).
- **Scope notes added to #44, #50, #94** with current-status / boundary
info (ldflags shipped vs `/version` remaining; DLQ shipped vs retry
remaining + scope boundary with #91; per-component logger source field
as a #94 complement).
- **Opened 4 new issues from orphan TODOs**: #143 (pprof), #144 (K8s
`/healthz` + per-dep health), #145 (RequireRoles fail-closed), #146
(split `internal/api/` into focused subpackages).

## TODO.md audit (one-time, for record)

| Bucket | Count | Disposition |
|--------|-------|-------------|
| Already shipped per closed issues (#11, #14, #16, #28, #40, #41, #42,
#45) + current code | ~12 | Deleted from TODO |
| Tracked as open issues (#32, #33, #34, #37, #39, #44, #48, #49, #50,
#51, #94) | ~12 | Kept as issues, scope notes added where useful |
| In-flight via open PRs (#83, #92, #119, #122, #125, #136, #137) | 4 |
Untouched |
| #46 Graceful Shutdown | 1 | Verified shipped, closed with comment |
| Orphan items | 4 | Split into #143-146 |
| Aspirational ("more tests", "update README") | 2 | Deleted — covered
by AGENTS.md doc-sync rules |

Projects #7 board + triage automation is now the single canonical
backlog.

## Test plan

- [x] `make ci` passes locally for each push (gated by
`.githooks/pre-push`)
- [x] CI green on the latest HEAD (8fbd7db)
- [x] PR-title-lint accepts the title (`chore: claude code native
improvements`)
- [x] All `docs/src/content/docs/*.md` paths in AGENTS.md resolve to
real files
- [x] Labeler workflow auto-labels correctly per the updated paths
- [x] `pre-push-reviewer` subagent loop reached `VERDICT: ship_it` with
zero findings under the strict rubric before the final push (validated
end-to-end across five iterations on this branch — each surfacing a real
doc-sync / off-by-one / quote-strip issue and forcing a fix before the
marker auto-wrote)
- [x] `agent-bash-gate.sh` quote-strip generalization sanity-tested
live: `echo "git push to deploy"` passes through; `git push --no-verify`
and `git commit --no-verify` still block (`bash -n` clean, JSON wiring
valid)
- [ ] Human review

## Related issues

- Closed during this work: #46
- Scope notes added: #44, #50, #94
- New follow-up issues created: #143, #144, #145, #146

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Claude Code integration: local pre-push reviewer with strict
ship/iterate/block verdicts, push gating via CI/review markers,
automatic review-marker creation, and a coverage-reporting command.

* **Documentation**
* Comprehensive Claude Code & agent docs, new skill guides for PR
review/sync, updated README/CONTRIBUTING/CHANGELOG/styleguide, and site
sidebar/page additions.

* **Chores**
* Added git and agent hooks, CI marker creation, worktrunk config,
labeler tweaks, and simplified Copilot instructions.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/Wave-RF/WaveHouse/pull/147?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api HTTP handlers, routing, middleware area/docs Documentation, site/, README area/infra CI, build, deploy, Docker, release area/query Structured query AST, SQL builder documentation Improvements or additions to documentation go Pull requests that update go code

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

boot: schema discovery should not be fatal — retry with backoff, serve diagnostic /health 503

2 participants