Skip to content

fix(dataplane): make the self-hosted environment worker path usable #2818

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vipzj wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(dataplane): make the self-hosted environment worker path usable #2818

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3802133
fix(dataplane): verify environment API key off the netty event loop
vipzj Aug 23, 2026
73ce3f9
fix(dataplane): schedule self-hosted worker endpoints on boundedElastic
vipzj Aug 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 38 additions & 29 deletions ...ice-dataplane/src/main/java/io/agentscope/builder/web/api/SelfHostedWorkerController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;

/**
* Outbound-worker data plane for {@code self_hosted}: pending tool_use listing, tool_result
Expand Down Expand Up @@ -74,10 +75,11 @@ public Mono<List<Map<String, Object>>> pendingTools(
@PathVariable("sessionId") String sessionId,
Authentication auth) {
return Mono.fromCallable(
() -> {
requireEnvironmentWorker(auth, environmentId, sessionId);
return pendingHandsToolService.listPending(sessionId);
});
() -> {
requireEnvironmentWorker(auth, environmentId, sessionId);
return pendingHandsToolService.listPending(sessionId);
})
.subscribeOn(Schedulers.boundedElastic());

@oss-maintainer oss-maintainer Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This moves the handler bodies off the event loop, but failure #1 in the description (401 on every worker request) lives upstream of them and is not touched here: EnvironmentKeyAuthFilter.filter calls matchesApiKey -> apiKeyVerifier.matches eagerly, and the bound method is ControlPlaneClient::verifyEnvironmentKey, which still ends in bodyToMono(Map.class).block() (ControlPlaneClient.java:496-511). A WebFlux WebFilter runs on the Netty event loop, so that block() still throws block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-epoll-N, the catch swallows it as return false, and the filter falls through without setting env:<id> — the controller then 401s before any of these subscribeOn operators are reached. The PR's file list is only these two controllers, so no worker route can actually authenticate yet. Is a second commit (making the filter reactive, e.g. Mono.fromCallable(...).subscribeOn(boundedElastic()) around the verification, or a reactive verifyEnvironmentKey) missing from the push?

}

/** Posts one or more tool results and resumes the suspended turn. */
Expand All @@ -88,27 +90,33 @@ public Mono<List<SessionEventDto>> toolResults(
@RequestBody ToolResultsRequest body,
Authentication auth) {
return Mono.fromCallable(
() -> {
ManagedSessionDto session =
requireEnvironmentWorker(auth, environmentId, sessionId);
if (body == null || body.results() == null || body.results().isEmpty()) {
throw ApiException.invalidRequest(
"missing_results", "results is required", "results");
}
List<ToolResultBlock> blocks = new ArrayList<>();
List<SessionEventDto> recorded = new ArrayList<>();
for (Map<String, Object> payload : body.results()) {
ToolResultBlock block = SessionTurnRunner.toolResultFromPayload(payload);
blocks.add(block);
Map<String, Object> stored = new LinkedHashMap<>(payload);
stored.putIfAbsent("tool_use_id", block.getId());
recorded.add(
eventLog.append(
sessionId, SessionEventTypes.USER_TOOL_RESULT, stored));
}
turnRunner.resumeWithToolResults(session, blocks);
return recorded;
});
() -> {
ManagedSessionDto session =
requireEnvironmentWorker(auth, environmentId, sessionId);
if (body == null
|| body.results() == null
|| body.results().isEmpty()) {
throw ApiException.invalidRequest(
"missing_results", "results is required", "results");
}
List<ToolResultBlock> blocks = new ArrayList<>();
List<SessionEventDto> recorded = new ArrayList<>();
for (Map<String, Object> payload : body.results()) {
ToolResultBlock block =
SessionTurnRunner.toolResultFromPayload(payload);
blocks.add(block);
Map<String, Object> stored = new LinkedHashMap<>(payload);
stored.putIfAbsent("tool_use_id", block.getId());
recorded.add(
eventLog.append(
sessionId,
SessionEventTypes.USER_TOOL_RESULT,
stored));
}
turnRunner.resumeWithToolResults(session, blocks);
return recorded;
})
.subscribeOn(Schedulers.boundedElastic());
}

/** Downloads the session agent's skills bundle for local staging on the worker. */
Expand All @@ -118,10 +126,11 @@ public Mono<Map<String, Object>> skills(
@PathVariable("sessionId") String sessionId,
Authentication auth) {
return Mono.fromCallable(
() -> {
requireEnvironmentWorker(auth, environmentId, sessionId);
return skillsBundleService.bundleForSession(sessionId);
});
() -> {
requireEnvironmentWorker(auth, environmentId, sessionId);
return skillsBundleService.bundleForSession(sessionId);
})
.subscribeOn(Schedulers.boundedElastic());
}

private ManagedSessionDto requireEnvironmentWorker(
Expand Down
51 changes: 28 additions & 23 deletions ...ce-dataplane/src/main/java/io/agentscope/builder/web/api/WorkerEnvironmentController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;

/**
* REST surface for out-of-process Environment Workers on {@code self_hosted} environments.
Expand Down Expand Up @@ -76,19 +77,20 @@ public Mono<ResponseEntity<EnvironmentWorkQueue.WorkItem>> poll(
@RequestParam("workerId") String workerId,
@RequestParam(name = "timeoutMs", defaultValue = "25000") long timeoutMs) {
return Mono.fromCallable(
() -> {
try {
Optional<EnvironmentWorkQueue.WorkItem> item =
workQueue.poll(environmentId, workerId, timeoutMs);
return item.map(this::withSessionMetadata)
.map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.noContent().build());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new ResponseStatusException(
HttpStatus.SERVICE_UNAVAILABLE, "Poll interrupted");
}
});
() -> {
try {
Optional<EnvironmentWorkQueue.WorkItem> item =
workQueue.poll(environmentId, workerId, timeoutMs);
return item.map(this::withSessionMetadata)
.map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.noContent().build());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new ResponseStatusException(
HttpStatus.SERVICE_UNAVAILABLE, "Poll interrupted");
}
})
.subscribeOn(Schedulers.boundedElastic());

@oss-maintainer oss-maintainer Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

poll is a 25 s long-poll by default, and Schedulers.boundedElastic() is the shared pool — the same one DataSessionApiController (BLOCKING), SessionTurnRunner, SessionEventLog and InternalToolConfirmationController subscribe on, capped at roughly 10x core-count workers. A handful of self-hosted workers polling concurrently can therefore park every boundedElastic worker and turn the 401 into app-wide latency. Two options worth considering: a dedicated newBoundedElastic("worker-poll", ...) scheduler for this route, or pushing the queue poll down to a non-blocking Mono/Sinks wait so a waiting worker costs no thread at all. With subscribeOn as written the blocking behaviour is fixed but the capacity ceiling is just relocated.

}

private EnvironmentWorkQueue.WorkItem withSessionMetadata(EnvironmentWorkQueue.WorkItem item) {
Expand All @@ -109,15 +111,17 @@ public Mono<List<EnvironmentWorkQueue.WorkItem>> listWork(
@RequestParam(value = "state", required = false) String state,
Authentication auth) {
requireUserAuth(auth);
return Mono.fromCallable(() -> workQueue.list(environmentId, state));
return Mono.fromCallable(() -> workQueue.list(environmentId, state))
.subscribeOn(Schedulers.boundedElastic());
}

/** Returns per-status counts and oldest queued age for the environment. */
@GetMapping("/work/stats")
public Mono<CoordinationStore.WorkStats> workStats(
@PathVariable("id") String environmentId, Authentication auth) {
requireUserAuth(auth);
return Mono.fromCallable(() -> workQueue.stats(environmentId));
return Mono.fromCallable(() -> workQueue.stats(environmentId))
.subscribeOn(Schedulers.boundedElastic());
}

/** Returns a single work item by id. */
Expand All @@ -128,14 +132,15 @@ public Mono<EnvironmentWorkQueue.WorkItem> getWork(
Authentication auth) {
requireUserAuth(auth);
return Mono.fromCallable(
() ->
workQueue
.get(workId)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"Unknown work item: " + workId)));
() ->
workQueue
.get(workId)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"Unknown work item: " + workId)))
.subscribeOn(Schedulers.boundedElastic());

@oss-maintainer oss-maintainer Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The write-side endpoints in this controller are still on the event loop: ack (~L156), heartbeat (~L182), stop (~L189) and updateWork (~L198) return Mono.fromRunnable without subscribeOn, and they call the same blocking queue (workQueue.get/ack/heartbeat/stop). ack is on the hot path of the exact loop this PR is trying to unblock — poll -> ack -> execute -> tool-results — so the 502 class the description reports should remain reachable there. Same treatment (or the shared helper noted below) would make the fix uniform.

}

/**
Expand Down
Loading