Skip to content

feat: gracefully drain the webhook server on shutdown (opt-in) - #67

Open
BarthV wants to merge 2 commits into
AbsaOSS:mainfrom
qube-rt:feat/graceful-shutdown
Open

BarthV wants to merge 2 commits into
AbsaOSS:mainfrom
qube-rt:feat/graceful-shutdown

Conversation

@BarthV

@BarthV BarthV commented Jul 10, 2026

Copy link
Copy Markdown

The webhook API server (main -> server.Start -> external-dns api.StartHTTPApi) had no signal handling, so SIGTERM from a normal pod termination exited the process immediately and could abort an in-flight /records apply. external-dns applies a record and its ownership TXT as separate sequential WAPI calls, so a mid-apply exit can strand a half-written or half-deleted record in the grid.

Run the API server directly (reusing the external-dns api handlers) with a signal-triggered shutdown. On SIGINT/SIGTERM the server stops accepting new connections and, when SERVER_SHUTDOWN_TIMEOUT is positive, drains in-flight requests for up to that long before exiting; the first signal is caught and the handler is then deregistered so a second signal still terminates immediately.

The drain is OPT-IN: SERVER_SHUTDOWN_TIMEOUT defaults to 0, which closes the server immediately and preserves the previous immediate-exit behaviour; operators enable draining by setting a positive timeout and a matching terminationGracePeriodSeconds. SERVER_READ_TIMEOUT/SERVER_WRITE_TIMEOUT are deliberately left unwired (a write timeout shorter than a slow apply would cut it mid-flight) — see the comments in serve.

Start now returns a bind error instead of calling log.Fatal, so it is testable and main owns the exit.

@BarthV

BarthV commented Jul 10, 2026

Copy link
Copy Markdown
Author

Hi !

This PR is 1 of the 2 (the other is #66 ) required to mitigate/lower the risk of leaving orphaned A / TXT records on infoblox. The code is heavily claude reviewed.

I've benchtested / stressed both of them in lab and I'm now running them in production. It works as expected, only a benign risk of lone TXT still persist, but cannot be solved because of the atomic WAPI call design (non transactional).

Feel free to ask questions or raise any remark.
Thanks

@k0da

k0da commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Hi, I'll take a look asap

@kuritka

kuritka commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Hello @BarthV , could you please update the PR to address lint issues and make sure the pipeline passes?

Thank you

@BarthV
BarthV force-pushed the feat/graceful-shutdown branch from 37aac37 to 4d6990e Compare July 13, 2026 14:39
@BarthV

BarthV commented Jul 13, 2026 •

Copy link
Copy Markdown
Author

Hello @BarthV , could you please update the PR to address lint issues and make sure the pipeline passes?

Thank you

Lint fixes has been pushed (commit squashed into the same single commit)

@k0da k0da left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@k0da

k0da commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

@BarthV Hi, thanks for PRs. Could you please sign your commits before we merge?

The webhook API server (main -> server.Start -> external-dns api.StartHTTPApi)
had no signal handling, so SIGTERM from a normal pod termination exited the
process immediately and could abort an in-flight /records apply. external-dns
applies a record and its ownership TXT as separate sequential WAPI calls, so a
mid-apply exit can strand a half-written or half-deleted record in the grid.

Run the API server directly (reusing the external-dns api handlers) with a
signal-triggered shutdown. On SIGINT/SIGTERM the server stops accepting new
connections and, when SERVER_SHUTDOWN_TIMEOUT is positive, drains in-flight
requests for up to that long before exiting; the first signal is caught and the
handler is then deregistered so a second signal still terminates immediately.

The drain is OPT-IN: SERVER_SHUTDOWN_TIMEOUT defaults to 0, which closes the
server immediately and preserves the previous immediate-exit behaviour;
operators enable draining by setting a positive timeout and a matching
terminationGracePeriodSeconds. SERVER_READ_TIMEOUT/SERVER_WRITE_TIMEOUT are
deliberately left unwired (a write timeout shorter than a slow apply would cut
it mid-flight) — see the comments in serve.

Start now returns a bind error instead of calling log.Fatal, so it is testable
and main owns the exit.
@BarthV
BarthV force-pushed the feat/graceful-shutdown branch from 4d6990e to d115840 Compare September 22, 2026 21:48
@BarthV

BarthV commented Sep 22, 2026

Copy link
Copy Markdown
Author

@k0da @kuritka
Rebased & signed (sorry for the huge lag)

@k0da k0da closed this Oct 27, 2026
@k0da k0da reopened this Oct 27, 2026
@BarthV

BarthV commented Oct 27, 2026

Copy link
Copy Markdown
Author

rebase & fixed conflicts

@k0da
k0da requested a balanced review from Copilot and removed request for januarios October 27, 2026 21:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shutdown can report a server failure as a successful exit, and the drain test does not reliably verify that shutdown waits for active requests.

2 open issues
What changed in this PR

Adds opt-in graceful shutdown to the webhook API so in-flight record changes can finish during pod termination, while retaining immediate shutdown by default.

Changes:

  • Runs the existing webhook handlers through a directly managed HTTP server with signal handling.
  • Adds SERVER_SHUTDOWN_TIMEOUT and returns bind errors to main.
  • Adds tests for draining, immediate closure, signal-handler deregistration, and bind failures.
File Description
README.md Lists the shutdown timeout setting and default.
cmd/​webhook/​main.go Handles errors returned by server startup.
cmd/​webhook/​init/​server/​server.go Manages the HTTP server and signal-triggered shutdown.
cmd/​webhook/​init/​server/​server_test.go Adds shutdown and startup-error coverage.
cmd/​webhook/​init/​configuration/​configuration.go Defines the shutdown timeout option.
cmd/​webhook/​init/​configuration/​configuration_test.go Checks that draining defaults to disabled.

Review effort: Balanced

// error rather than the http.ErrServerClosed that Close/Shutdown produces.
// Without this read that error would be lost and the process would exit 0.
if err := <-errCh; err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Errorf("webhook server stopped with error during shutdown: %s", err)
Comment on lines +499 to +500
cancel() // trigger graceful shutdown mid-apply
close(sp.release) // let the in-flight apply finish while the drain waits
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants