Repository navigation
Conversation
|
Hi ! This PR is 1 of the 2 (the other is #66 ) required to mitigate/lower the risk of leaving orphaned A / TXT records on infoblox. The code is heavily claude reviewed. I've benchtested / stressed both of them in lab and I'm now running them in production. It works as expected, only a benign risk of lone TXT still persist, but cannot be solved because of the atomic WAPI call design (non transactional). Feel free to ask questions or raise any remark. |
|
Hi, I'll take a look asap |
|
Hello @BarthV , could you please update the PR to address lint issues and make sure the pipeline passes? Thank you |
37aac37 to
4d6990e
Compare
Lint fixes has been pushed (commit squashed into the same single commit) |
|
@BarthV Hi, thanks for PRs. Could you please sign your commits before we merge? |
The webhook API server (main -> server.Start -> external-dns api.StartHTTPApi) had no signal handling, so SIGTERM from a normal pod termination exited the process immediately and could abort an in-flight /records apply. external-dns applies a record and its ownership TXT as separate sequential WAPI calls, so a mid-apply exit can strand a half-written or half-deleted record in the grid. Run the API server directly (reusing the external-dns api handlers) with a signal-triggered shutdown. On SIGINT/SIGTERM the server stops accepting new connections and, when SERVER_SHUTDOWN_TIMEOUT is positive, drains in-flight requests for up to that long before exiting; the first signal is caught and the handler is then deregistered so a second signal still terminates immediately. The drain is OPT-IN: SERVER_SHUTDOWN_TIMEOUT defaults to 0, which closes the server immediately and preserves the previous immediate-exit behaviour; operators enable draining by setting a positive timeout and a matching terminationGracePeriodSeconds. SERVER_READ_TIMEOUT/SERVER_WRITE_TIMEOUT are deliberately left unwired (a write timeout shorter than a slow apply would cut it mid-flight) — see the comments in serve. Start now returns a bind error instead of calling log.Fatal, so it is testable and main owns the exit.
4d6990e to
d115840
Compare
|
rebase & fixed conflicts |
There was a problem hiding this comment.
🟡 Changes recommended
Shutdown can report a server failure as a successful exit, and the drain test does not reliably verify that shutdown waits for active requests.
2 open issues
What changed in this PR
Adds opt-in graceful shutdown to the webhook API so in-flight record changes can finish during pod termination, while retaining immediate shutdown by default.
Changes:
- Runs the existing webhook handlers through a directly managed HTTP server with signal handling.
- Adds
SERVER_SHUTDOWN_TIMEOUTand returns bind errors tomain. - Adds tests for draining, immediate closure, signal-handler deregistration, and bind failures.
| File | Description |
|---|---|
| README.md | Lists the shutdown timeout setting and default. |
| cmd/webhook/main.go | Handles errors returned by server startup. |
| cmd/webhook/init/server/server.go | Manages the HTTP server and signal-triggered shutdown. |
| cmd/webhook/init/server/server_test.go | Adds shutdown and startup-error coverage. |
| cmd/webhook/init/configuration/configuration.go | Defines the shutdown timeout option. |
| cmd/webhook/init/configuration/configuration_test.go | Checks that draining defaults to disabled. |
Review effort: Balanced
| // error rather than the http.ErrServerClosed that Close/Shutdown produces. | ||
| // Without this read that error would be lost and the process would exit 0. | ||
| if err := <-errCh; err != nil && !errors.Is(err, http.ErrServerClosed) { | ||
| log.Errorf("webhook server stopped with error during shutdown: %s", err) |
| cancel() // trigger graceful shutdown mid-apply | ||
| close(sp.release) // let the in-flight apply finish while the drain waits |

The webhook API server (main -> server.Start -> external-dns api.StartHTTPApi) had no signal handling, so SIGTERM from a normal pod termination exited the process immediately and could abort an in-flight /records apply. external-dns applies a record and its ownership TXT as separate sequential WAPI calls, so a mid-apply exit can strand a half-written or half-deleted record in the grid.
Run the API server directly (reusing the external-dns api handlers) with a signal-triggered shutdown. On SIGINT/SIGTERM the server stops accepting new connections and, when SERVER_SHUTDOWN_TIMEOUT is positive, drains in-flight requests for up to that long before exiting; the first signal is caught and the handler is then deregistered so a second signal still terminates immediately.
The drain is OPT-IN: SERVER_SHUTDOWN_TIMEOUT defaults to 0, which closes the server immediately and preserves the previous immediate-exit behaviour; operators enable draining by setting a positive timeout and a matching terminationGracePeriodSeconds. SERVER_READ_TIMEOUT/SERVER_WRITE_TIMEOUT are deliberately left unwired (a write timeout shorter than a slow apply would cut it mid-flight) — see the comments in serve.
Start now returns a bind error instead of calling log.Fatal, so it is testable and main owns the exit.