Skip to content

Close workflow audit log writers - #6110

Merged
JAORMX merged 3 commits into
stacklok:mainfrom
kocaemre:fix/workflow-auditor-close-file
Sep 25, 2026
Merged

JAORMX merged 3 commits into
stacklok:mainfrom
kocaemre:fix/workflow-auditor-close-file

Conversation

@kocaemre

Copy link
Copy Markdown
Contributor

Summary

  • Workflow audit logging can currently leak a file descriptor when Config.LogFile is set because NewWorkflowAuditor opens a log writer but the returned auditor does not retain or expose a way to close it.
  • Retain the workflow auditor log writer, add WorkflowAuditor.Close(), and close the optional workflow auditor from coreVMCP.Close() and core construction error paths.
  • Share close handling with the HTTP auditor and avoid closing os.Stdout/os.Stderr when audit logging uses default output.

Fixes #6094

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Unit tests (task test)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)
  • Manual testing (describe below)

Manual testing:

  • PATH=/usr/local/go/bin:/root/go/bin:$PATH task lint
  • PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -race ./pkg/audit ./pkg/vmcp/core
  • PATH=/usr/local/go/bin:/root/go/bin:$PATH go test ./pkg/vmcp/composer

API Compatibility

  • This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Changes

File Change
pkg/audit/auditor.go Reuse a shared close helper and avoid closing stdout/stderr.
pkg/audit/workflow_auditor.go Retain the log writer and expose Close().
pkg/audit/workflow_auditor_test.go Cover file-backed close, stdout handling, and close error propagation.
pkg/vmcp/core/core_vmcp.go Close workflow auditor resources during shutdown and constructor cleanup paths.

Does this introduce a user-facing change?

No. This releases an internal audit log file descriptor when the workflow auditor owner is closed.

@jerm-dro jerm-dro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on — the diagnosis and the design are right. Retaining the writer, factoring out the shared close helper, and catching the stdout trap rather than inheriting it are all exactly what #6094 asked for, and putting the auditor's lifecycle on coreVMCP (next to stopStore and healthMonitor) is the correct owner.

Two blockers, both mechanical, both about the tests rather than the fix:

  1. The two stdout tests pass against the unfixed code — os.Stdout.Close() returns nil, so require.NoError can't distinguish fixed from broken. As written they'd stay green if the guard were removed.
  2. The coreVMCP change is the code that actually plugs the leak, and it has no coverage — including the three New error paths.

Details and suggested code in the line comments.

Comment thread pkg/audit/workflow_auditor_test.go Outdated
Comment thread pkg/vmcp/core/core_vmcp.go
@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 46bd4b0 to 812b3ed Compare August 15, 2026 17:38
@kocaemre

Copy link
Copy Markdown
Contributor Author

Addressed the two mechanical blockers in 812b3ed7.

Changes:

  • moved the Auditor stdout regression into auditor_test.go and made it observe fd usability with os.Stdout.Write(nil) after Close();
  • updated the WorkflowAuditor stdout subtest to assert the stdout descriptor remains usable after Close() too;
  • added core-level coverage for file-backed workflow audit lifecycle:
    • coreVMCP.Close() reaches and closes the workflow auditor's retained file descriptor;
    • New() closes the workflow audit file on the workflow-validation error path;
    • New() closes the workflow audit file on the health-monitor creation error path.

Local verification:

  • go test -ldflags=-extldflags=-Wl,-w -run 'Test(Auditor_CloseDoesNotCloseStdout|WorkflowAuditor_Close)$' ./pkg/audit → passed
  • go test -ldflags=-extldflags=-Wl,-w -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core → passed
  • task lint → passed (golangci-lint + go vet, 0 issues)

I also started task test; it is still running locally because it invokes the full race-enabled unit suite across the repo.

@codecov

codecov Bot commented Aug 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.75758% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.91%. Comparing base (e532cf0) to head (11f3a31).

Files with missing lines Patch % Lines
pkg/vmcp/core/core_vmcp.go 70.37% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6110      +/-   ##
==========================================
- Coverage   78.98%   78.91%   -0.07%     
==========================================
  Files         782      782              
  Lines       78065    78093      +28     
==========================================
- Hits        61658    61629      -29     
- Misses      16402    16459      +57     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@kocaemre

Copy link
Copy Markdown
Contributor Author

Follow-up on the local task test run I mentioned above: the full race-enabled suite eventually failed in two integration tests outside this PR's touched packages/code paths:

  • pkg/transport/proxy/streamable: TestPostSSE_ProgressIsolationBetweenSessions
  • pkg/vmcp/server: TestIntegration_SSEGetConnectionSurvivesWriteTimeout

I re-ran both failures in isolation and at package scope with the same race/ldflags shape, and they passed:

  • go test -ldflags=-extldflags=-Wl,-w -race -count=3 -run '^TestPostSSE_ProgressIsolationBetweenSessions$' ./pkg/transport/proxy/streamable → passed
  • go test -ldflags=-extldflags=-Wl,-w -race ./pkg/transport/proxy/streamable → passed
  • go test -ldflags=-extldflags=-Wl,-w -race -count=3 -run '^TestIntegration_SSEGetConnectionSurvivesWriteTimeout$' ./pkg/vmcp/server → passed
  • go test -ldflags=-extldflags=-Wl,-w -race ./pkg/vmcp/server → passed

So I don't see evidence that those two full-suite failures are caused by the workflow-auditor close changes.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 812b3ed to 17d4e5b Compare August 18, 2026 20:40
@kocaemre

Copy link
Copy Markdown
Contributor Author

Added one more focused core close-path regression in 17d4e5ba.

The new subtest pre-closes the retained workflow auditor and then calls coreVMCP.Close(), so the close-error logging branch in core_vmcp.go is now exercised without making Close() fail. Local coverage for coreVMCP.Close moved from 77.8% to 88.9%, and the previously uncovered close-error block is covered (core_vmcp.go:563.52,565.5 count 1).

Verification:

  • go test -ldflags=-extldflags=-Wl,-w -run 'Test(Auditor_CloseDoesNotCloseStdout|WorkflowAuditor_Close)$' ./pkg/audit → passed\n- go test -ldflags=-extldflags=-Wl,-w -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core → passed\n- go test -ldflags=-extldflags=-Wl,-w -coverprofile=/tmp/core6110-after.cover ./pkg/vmcp/core → passed, 87.6% package coverage\n- task lint → passed (golangci-lint + go vet, 0 issues)\n- git diff --check origin/main..HEAD → passed

@jerm-dro jerm-dro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both original blockers are resolved — verified locally against 17d4e5ba: the stdout tests now observe the descriptor (they fail against unfixed code, which was the whole point), and TestNew_CloseReleasesWorkflowAuditLogFile passes. Thanks for going further than asked on the error paths and the close-error branch.

One remaining issue, on the new /proc/self/fd helper — details in the line comment. It's a test portability problem, not a problem with your fix; the fix itself behaves correctly on macOS. Everything else looks good.

Comment thread pkg/vmcp/core/core_vmcp_test.go
@kocaemre

Copy link
Copy Markdown
Contributor Author

Addressed in 8f1f4267: added the runtime.GOOS != "linux" guard to requireNoOpenFDForPath, so /proc/self/fd verification still runs on Linux CI and skips cleanly on macOS.

Local verification on Linux:

PATH=/tmp/go1.26/bin:$PATH go test -ldflags=-extldflags=-Wl,-w -v ./pkg/vmcp/core -run 'TestNew_ErrorPathsCloseWorkflowAuditLogFile|TestNew_CloseReleasesWorkflowAuditLogFile' -count=1

Result: PASS (github.com/stacklok/toolhive/pkg/vmcp/core 0.057s).

Note: full task test could not run in my local container initially because the system Go was 1.18 while this repo requires Go 1.26; targeted test above was run after installing Go 1.26 locally.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 8f1f426 to f8b41d9 Compare September 7, 2026 11:12
@kocaemre

kocaemre commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main (500f4784) and force-pushed the rebased head f8b41d9e.

No code changes beyond the existing two PR commits; this was a rebase-only maintenance update to keep the branch current after upstream moved.

Local verification in this cron environment:

git diff --check origin/main..HEAD
# passed

git log --format='%h %s%n%b' origin/main..HEAD
# f8b41d9e Guard Linux-only fd test helper
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
# ab2f7aa7 Close workflow audit log writers
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>

I attempted to re-run the previous focused Go checks, but this cron shell currently resolves go to go1.18.1, and the current repo requires go 1.26.0, so the test command is blocked before execution with:

go: errors parsing go.mod:
.../go.mod:3: invalid go version '1.26.0': must match format 1.23

GitHub CI has been triggered on the refreshed branch; I will rely on those current CI results for the full ToolHive task/test/lint matrix.

@kocaemre

kocaemre commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Current rebase CI has finished successfully on f8b41d9e.

Follow-up status:

  • the remaining /proc/self/fd portability blocker is addressed by 8f1f4267 / rebased head f8b41d9e via the runtime.GOOS != "linux" skip guard;
  • the branch is rebased onto current main (500f4784);
  • GitHub checks on this head are all green, including Go tests/lint, docs/codegen, E2E core/operator matrices, and Codecov project.

Local evidence from the rebase environment is still the same as above:

  • git diff --check origin/main..HEAD → passed
  • focused Go tests were blocked locally by the cron shell's Go 1.18 vs repo go 1.26.0 requirement, so I relied on the current GitHub CI run for the full ToolHive validation matrix.

I think this is ready for re-review when you have a chance.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from f8b41d9 to 79c345e Compare September 7, 2026 15:19
@kocaemre

kocaemre commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main (a83059cd) after upstream moved again and force-pushed rebased head 79c345e8.

No code changes beyond replaying the existing two PR commits on top of the current branch.

Local verification in this cron environment:

git diff --check origin/main..HEAD
# passed

git log --format='%h %s%n%b' origin/main..HEAD
# 79c345e8 Guard Linux-only fd test helper
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
# 7ed0068b Close workflow audit log writers
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>

I also retried the previously focused Go checks, but this cron shell still resolves go to go1.18.1 while the repo now declares go 1.26.0, so both test commands are blocked before package execution with:

go: errors parsing go.mod:
/root/oss-sprint/toolhive/go.mod:3: invalid go version '1.26.0': must match format 1.23

GitHub CI has been triggered for the refreshed head; I'll rely on that current CI matrix for Go/test/lint validation.

@kocaemre

kocaemre commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

Current refreshed head 79c345e8 has finished green on GitHub CI after the latest rebase.

Status snapshot:

  • mergeable=MERGEABLE, mergeStateStatus=BLOCKED only because the stale CHANGES_REQUESTED review is still recorded
  • status check rollup: 43/43 completed successfully, 0 pending, 0 failed
  • includes the ToolHive Go test/lint/docs/codegen/E2E/operator matrices and Codecov project check

Local evidence from the rebase remains:

  • git diff --check origin/main..HEAD → passed
  • DCO sign-offs preserved on both PR commits

I believe this is ready for re-review now that the post-rebase CI is fully green.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 79c345e to ed493c8 Compare September 10, 2026 05:05
@kocaemre

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main (cdb04c94f) and force-pushed rebased head ed493c812.

No code changes beyond replaying the existing two PR commits on top of current upstream. This keeps the branch current while it is waiting on re-review for the previously addressed CHANGES_REQUESTED thread.

Local verification after the refresh:

git diff --check origin/main..HEAD
# passed

PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -ldflags=-extldflags=-Wl,-w -run 'Test(Auditor_CloseDoesNotCloseStdout|WorkflowAuditor_Close)$' ./pkg/audit
# ok  github.com/stacklok/toolhive/pkg/audit  0.023s

PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core
# ok  github.com/stacklok/toolhive/pkg/vmcp/core  0.092s

git log --format='%h %s%n%b' origin/main..HEAD
# ed493c812 Guard Linux-only fd test helper
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
# 195535102 Close workflow audit log writers
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>

I also ran the repo-level lint task; it still fails before reaching this PR's touched files on the pre-existing gci findings in pkg/authserver/server/provider.go and pkg/authserver/server_impl.go, so I left that unrelated formatting churn out of this focused PR:

PATH=/usr/local/go/bin:/root/go/bin:$PATH task lint
# pkg/authserver/server/provider.go:391:1: File is not properly formatted (gci)
# pkg/authserver/server_impl.go:194:1: File is not properly formatted (gci)
# task: Failed to run task "lint": exit status 1

GitHub CI has been retriggered on the refreshed head.

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Panel review of ed493c81286ef57c81184ebd8e3c989e3a165733 against main cdb04c94f2462553fc27572bc8db9bb88a0c3c78 and #6094.

Blocking — VMCP close discards the newly exposed audit-close failure

pkg/vmcp/core/core_vmcp.go:559-573 logs a workflowAuditor.Close() error but always returns nil. That makes the newly owned file's close/flush failure unobservable to callers of the public VMCP.Close() error contract. Preserve the first close's error under sync.Once and return it; join it with any other teardown failure as appropriate, while retaining the documented nil result on later calls.

Blocking — constructor cleanup also loses audit-close failures

pkg/vmcp/core/core_vmcp.go:145-152 logs and discards a workflow-auditor close error on subsequent New failures (for example newWorkflowInstruments at :180-184). Return or join that cleanup error with the triggering construction error so neither cause is hidden.

The fix otherwise meets #6094's core goal: it retains the writer, avoids closing stdout/stderr, and covers the file-backed close path. Reuse and duplication reviewers found no additional issues.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from ed493c8 to c545425 Compare September 10, 2026 09:38
@kocaemre

Copy link
Copy Markdown
Contributor Author

Addressed the new close-error propagation blockers in c5454253a.

What changed:

  • coreVMCP.Close() now preserves and returns cleanup errors from the first sync.Once-guarded close call while keeping later Close() calls idempotent (nil).
  • workflow auditor close failures are joined into constructor error paths, so cleanup failures are no longer hidden behind workflow telemetry/validation/health-monitor setup errors.
  • updated the existing close-path regression to pre-close the retained workflow auditor, assert Close() returns an os.ErrClosed-wrapping error, and assert a later Close() still returns nil.

Local verification:

export PATH=/usr/local/go/bin:/root/go/bin:$PATH

go version
# go version go1.26.5 linux/amd64

git diff --check origin/main..HEAD
# passed

go test -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	0.063s

go test -race -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	1.251s

go test -race ./pkg/vmcp/core -count=1
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	1.318s

golangci-lint run --allow-parallel-runners ./pkg/vmcp/core
# 0 issues.

Repo-level lint was also attempted and is still blocked before this PR's touched package by the pre-existing/out-of-scope gci findings in pkg/authserver/server/provider.go and pkg/authserver/server_impl.go:

task lint
# pkg/authserver/server/provider.go:391:1: File is not properly formatted (gci)
# pkg/authserver/server_impl.go:194:1: File is not properly formatted (gci)
# task: Failed to run task "lint": exit status 1

GitHub CI has been retriggered on the pushed head and was queued at the time of this comment.

@kocaemre

Copy link
Copy Markdown
Contributor Author

Follow-up status on the pushed review-fix head c5454253a:

  • GitHub check rollup is now fully green: 41/41 checks completed successfully, 0 pending, 0 failed.
  • mergeable is still reported as UNKNOWN at this instant while GitHub recomputes, and reviewDecision remains CHANGES_REQUESTED from the stale review state.

This is the same head from my previous comment that addressed the close-error propagation blockers; no additional code changes were made in this status-only follow-up.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from c545425 to 2a9c805 Compare September 11, 2026 07:13
@kocaemre

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main (b934b7c7) and force-pushed rebased head 2a9c805e.

No code changes beyond replaying the existing three PR commits on top of current upstream.

Local verification after the refresh:

git diff --check upstream/main..HEAD
# passed

export PATH=/usr/local/go/bin:/root/go/bin:$PATH

go version
# go version go1.26.5 linux/amd64

go test -ldflags=-extldflags=-Wl,-w -run 'Test(Auditor_CloseDoesNotCloseStdout|WorkflowAuditor_Close)$' ./pkg/audit
# ok  	github.com/stacklok/toolhive/pkg/audit	0.025s

go test -run 'TestNew_(CloseReleasesWorkflowAuditLogFile|ErrorPathsCloseWorkflowAuditLogFile|ValidatesWorkflows)$' ./pkg/vmcp/core
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	0.072s

go test -race ./pkg/vmcp/core -count=1
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	1.305s

golangci-lint run --allow-parallel-runners ./pkg/audit ./pkg/vmcp/core
# 0 issues.

I also attempted the repo-level Taskfile lint:

task lint
# task: [lint] golangci-lint run --allow-parallel-runners ./...
# task: Failed to run task "lint": exit status 137

So full repo lint was blocked by the runner killing the repo-wide golangci-lint process in this cron environment; the touched packages passed targeted tests/race/lint. GitHub CI has been retriggered on the refreshed head.

@kocaemre

Copy link
Copy Markdown
Contributor Author

Follow-up on the current red CI after the refresh to 2a9c805e:

  • The only failing GitHub check is Tests / Test Go Code (ubuntu-8cores-32gb) from run 34573411721 / job 103180344855.
  • The failed job log points at pkg/vmcp/server's TestForwarding_Logging_RealBackend, not this PR's touched packages:
TestForwarding_Logging_RealBackend (1m1.57s)
forwarding_realbackend_integration_test.go:533: timed out waiting for notifications/message notification: context deadline exceeded

I re-ran the relevant local evidence on the current PR head:

PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -v -race ./pkg/audit ./pkg/vmcp/core
# pkg/audit passed; pkg/vmcp/core also passed when run without the CI ldflags below.

PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -v -race ./pkg/vmcp/core
# ok  github.com/stacklok/toolhive/pkg/vmcp/core  1.299s

PATH=/usr/local/go/bin:/root/go/bin:$PATH go test -v -race ./pkg/vmcp/server -run '^TestForwarding_Logging_RealBackend$' -count=1
# ok  github.com/stacklok/toolhive/pkg/vmcp/server  31.305s

git diff --check HEAD~3..HEAD
# passed

One local environment note: the repo task test / task test-coverage wrappers are currently blocked in this runner by gotestfmt panicking with BUG: Empty package name encountered; the first combined package command also hit this runner's /usr/bin/ld: unrecognized option '-w' when using the Taskfile ldflags. The scoped no-ldflags package reruns above passed, including the same test that failed in CI.

@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 2a9c805 to 7e11c57 Compare September 11, 2026 15:53
@kocaemre

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main (41dec70ef) and force-pushed rebased head 7e11c572c.

No code changes beyond replaying the existing three PR commits on top of upstream. This should also rerun the previously red Tests / Test Go Code (ubuntu-8cores-32gb) check; the prior red run was from 2a9c805e and failed in TestForwarding_Logging_RealBackend with:

forwarding_realbackend_integration_test.go:533: timed out waiting for notifications/message notification: context deadline exceeded

Local validation on this runner:

git rebase origin/main
git diff --check origin/main..HEAD

git diff --check passed. I also attempted the repo-prescribed targeted task test PKG=./pkg/vmcp, but this cron runner only has Go 1.18.1 while current go.mod requires Go 1.26.0 / upstream has already bumped toward Go 1.27-era dependencies, so generation fails before tests with missing stdlib packages such as cmp, iter, maps, slices, and crypto/fips140.

Current GitHub status after the push: checks have started on 7e11c572c; several are already green and the main test/lint/doc/codegen jobs are still in progress.

@kocaemre

Copy link
Copy Markdown
Contributor Author

CI/status follow-up for the rebased head 7e11c572c:

  • GitHub check rollup is now green: all reported checks completed successfully, including Tests / Test Go Code (ubuntu-8cores-32gb), Linting / Lint Go Code, E2E core/conformance jobs, docs, codegen, operator tests, license headers, grype, zizmor, and Codecov.
  • gh pr view now reports mergeable=MERGEABLE, mergeStateStatus=BLOCKED, and reviewDecision=CHANGES_REQUESTED, so the remaining blocker appears to be reviewer re-review/dismissal rather than CI.

Command evidence:

gh pr checks 6110 --repo stacklok/toolhive
# all listed checks returned pass, including Tests / Test Go Code and Linting / Lint Go Code

gh pr view 6110 --repo stacklok/toolhive --json mergeable,mergeStateStatus,reviewDecision
# mergeable=MERGEABLE, mergeStateStatus=BLOCKED, reviewDecision=CHANGES_REQUESTED

No new code changes in this follow-up; this is just the completed CI snapshot after the branch refresh.

@kocaemre

Copy link
Copy Markdown
Contributor Author

Follow-up validation for the current head 7e11c572 after addressing the latest review:

  • VMCP.Close() now preserves/returns the first workflow-auditor close error during the first sync.Once execution and keeps later Close() calls idempotent (nil).
  • New error-path cleanup now joins workflow-auditor close failures with the triggering constructor error, so neither cause is hidden.
  • Added/kept regression coverage for the retained close path and constructor cleanup paths.

Local validation from this branch:

$ export PATH=/usr/local/go/bin:$PATH
$ go version
go version go1.27.0 linux/amd64
$ go test ./pkg/vmcp/core ./pkg/audit
ok  	github.com/stacklok/toolhive/pkg/vmcp/core	0.078s
ok  	github.com/stacklok/toolhive/pkg/audit	0.141s

I also attempted the repo-preferred targeted command:

$ task test -- pkg/vmcp/core
...
task: Failed to run task "test": task: Failed to run task "gen": exit status 1

That failure was local toolchain setup only: the existing mockgen binary was built with Go 1.26 and fails under this repo's Go 1.27 module (package requires newer Go version go1.27 (application built with go1.26)). I did not regenerate or modify generated files.

GitHub check-runs on 7e11c572 are green in the one-shot check I ran (no non-success/non-skipped check-runs returned); the combined commit status endpoint still reports pending.

Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
@kocaemre
kocaemre force-pushed the fix/workflow-auditor-close-file branch from 7e11c57 to 11f3a31 Compare September 13, 2026 03:36
@kocaemre

Copy link
Copy Markdown
Contributor Author

Refreshed this PR branch onto current main and force-pushed rebased head 11f3a315.

No code changes beyond replaying the existing three PR commits on top of upstream; DCO sign-offs are preserved.

Local verification after the refresh:

git diff --check upstream/main..HEAD
# passed

git log --format='%h %s%n%b' upstream/main..HEAD
# 11f3a315 Return workflow auditor close errors
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
# f5fcfa11 Guard Linux-only fd test helper
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
# 005de1ec Close workflow audit log writers
# Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>

export PATH=/usr/local/go/bin:/root/go/bin:$PATH

go version
# go version go1.27.0 linux/amd64

go test ./pkg/vmcp/core ./pkg/audit
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	0.092s
# ok  	github.com/stacklok/toolhive/pkg/audit	0.133s

go test -race ./pkg/vmcp/core -count=1
# ok  	github.com/stacklok/toolhive/pkg/vmcp/core	1.281s

Repo-preferred wrapper checks attempted:

task test -- pkg/vmcp/core
# blocked during `task gen`: existing mockgen binary reports it was built with Go 1.26 while repo deps require Go 1.27

task lint
# blocked before package analysis: golangci-lint binary was built with Go 1.26 while the repo targets Go 1.27.0

GitHub CI has been retriggered on the refreshed head; at the immediate snapshot the check rollup was queued (16 checks), with no failures reported yet.

@kocaemre

Copy link
Copy Markdown
Contributor Author

Current refreshed head 11f3a315 has finished green on GitHub CI.

Status snapshot:

  • mergeable=MERGEABLE, mergeStateStatus=BLOCKED, reviewDecision=CHANGES_REQUESTED (stale review state awaiting re-review/dismissal)
  • check rollup: all reported checks completed successfully, including Go tests/lint, docs/codegen, E2E core/operator matrices, grype, zizmor, and Codecov

Local evidence from the refresh remains:

  • git diff --check upstream/main..HEAD → passed
  • go test ./pkg/vmcp/core ./pkg/audit → passed
  • go test -race ./pkg/vmcp/core -count=1 → passed

No new code changes in this follow-up; this is the completed CI snapshot after the branch refresh.

@JAORMX

JAORMX commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Sorry for taking so long to get back to this! Thanks for sticking with it and keeping the branch up to date. I'm going through the latest changes now, including the close-error handling from my last review.

@JAORMX JAORMX left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sticking with this! The concerns from my previous review are addressed: the first core Close returns cleanup errors, later calls remain idempotent, and core construction failures preserve both the original error and any audit-close error. The descriptor tests and Linux-only guard are in place too.

I did a fresh pass over 11f3a31. There are two surrounding server-lifecycle cases we should fix in a follow-up PR: preserving core.Close errors in server-construction rollback, and ensuring workflows finish their final audit logging before the writer closes when graceful HTTP shutdown times out. These are worth addressing, but I don't think we should hold this focused leak fix for that broader lifecycle work.

All 42 GitHub checks are green. My local task test run exceeded its execution timeout, so I'm not counting that as a local pass.

Looks good to merge. Thanks!

@JAORMX
JAORMX merged commit 6180ced into stacklok:main Sep 25, 2026
43 checks passed
@JAORMX JAORMX mentioned this pull request Sep 25, 2026
4 of 10 tasks
@github-actions github-actions Bot mentioned this pull request Sep 26, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WorkflowAuditor leaks the audit log file descriptor

3 participants