fix(#5066): map the 'fast' async queue to the MPMC DisruptorBlockingQueue - #5081
Conversation
…ueue
The issue offered a second drain pass or documenting the limitation.
Investigation found the real defect one level deeper, making both
options wrong:
- PushPullBlockingQueue (what 'fast' mapped to) is an explicit
single-producer/single-consumer design ("Transfers from a single
thread writer to a single thread reader") with no CAS on its tail
sequence. Every async worker queue has MANY producers (any
application thread, cross-scheduling workers, waitCompletion markers,
the closing thread), so the contract was violated in normal
operation, not just at shutdown. The new multi-producer regression
test demonstrated it empirically: 8 producers x 250 no-op tasks LOST
tasks on the old impl (latch never reached zero). The 'fast' setting
is also selected by the high-performance profile, so this was not an
obscure corner.
- A second drain pass cannot be made total anyway: a producer that
snapshotted executorThreads before shutdown can offer arbitrarily
late, and no finite number of passes covers that; the producer-side
remove-undo is the only shape that works regardless of timing, and it
needs remove(Object).
Fix: 'fast' now maps to DisruptorBlockingQueue, the flagship MPMC
variant from the same Conversant jar (no new dependency): lock-free
CAS-claimed sequences honor the multi-producer contract, remove(Object)
is implemented (cursor-blocked scan), so scheduleTask's post-shutdown
undo now works identically on both impls, and capacity 1 is supported
(explicitly allowed by Capacity.getCapacity), matching the test
configs. Capacities round up to the next power of two, as they already
did with the old ring. removeQuietly's UnsupportedOperationException
catch is kept as defensive-only for hypothetical future impls, with the
comment updated.
Red-first evidence (AsyncFastQueueShutdownUndoTest):
- fastQueueMustSupportRemoveForThePostShutdownUndo: threw
UnsupportedOperationException on the old impl; green after the swap.
- fastQueueMustNotLoseTasksUnderConcurrentProducers: timed out with
lost tasks on the old impl; deterministically green on the MPMC one.
Verified: mvn -pl engine compile; com.arcadedb.database.async battery
plus AsyncTest twice - default config (36 tests) and the fast-queue
variant via -Darcadedb.asyncOperationsQueueImpl=fast (36 tests) - all
green, so every existing shutdown/drain/stall regression now also
passes on the Disruptor queue. Release notes updated: the formerly
documented fast-queue residual gap is fixed.
|
Tick the box to add this pull request to the merge queue (same as
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Review — PR #5081: map the 'fast' async queue to
|
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5081 +/- ##
============================================
+ Coverage 65.63% 65.76% +0.13%
- Complexity 841 892 +51
============================================
Files 1687 1691 +4
Lines 135373 135819 +446
Branches 28969 29042 +73
============================================
+ Hits 88846 89323 +477
+ Misses 34411 34320 -91
- Partials 12116 12176 +60 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
… cold path; CPU note The review's one genuine pre-merge want, taken: capacity 1 is production-reachable (ASYNC_OPERATIONS_QUEUE_SIZE / parallelLevel floors to 1 on large pools) and the Conversant ring's minimum-size rounding is library-version-specific, so the stated guarantee is now pinned by fastQueueSupportsEffectiveCapacityOne - a parked-worker probe of the real 1-slot queue: offer, reject-when-full (not block/throw), remove(Object), slot reuse. A future disruptor bump cannot silently regress the post-shutdown undo primitive at the production floor. Point 2 taken as the requested note at the removeQuietly call site: remove(Object) on the Disruptor queue is an O(n) whole-queue-locking scan, acceptable only on the rare dead-worker post-shutdown race - it must never migrate onto the steady-state scheduling path. Nit taken: the release note now states the fast-profile CPU tradeoff (busy-spin before park, same as the previous implementation) for operators. The thread-name coupling, ignored join return and no-slow-tag observations are the reviewer's own verifications, kept as-is. AsyncFastQueueShutdownUndoTest 3/3 plus the async battery under -Darcadedb.asyncOperationsQueueImpl=fast, all green.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request resolves issue #5066 by replacing the single-producer/single-consumer PushPullBlockingQueue with the multi-producer DisruptorBlockingQueue for the 'fast' queue implementation. This change prevents silent task loss under concurrent producers and restores support for the remove(Object) operation needed during post-shutdown undo. A comprehensive regression test suite has been added to verify these behaviors. The review feedback suggests optimizing the test helper method findWorkerThread by traversing the ThreadGroup instead of calling Thread.getAllStackTraces(), which incurs high JVM overhead by triggering a global safepoint.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| private static DatabaseAsyncExecutorImpl.AsyncThread findWorkerThread(final DatabaseInternal db, final int slot) { | ||
| final String name = "AsyncExecutor-" + db.getName() + "-" + slot; | ||
| return (DatabaseAsyncExecutorImpl.AsyncThread) Thread.getAllStackTraces().keySet().stream() | ||
| .filter(t -> t.getName().equals(name)).findFirst() | ||
| .orElseThrow(() -> new IllegalStateException("Worker thread " + name + " not found")); | ||
| } |
There was a problem hiding this comment.
Using Thread.getAllStackTraces() is a heavy operation because it forces the JVM to dump stack traces for all active threads, which triggers a global safepoint. In a large test suite, this can lead to significant performance overhead and transient timing/flakiness issues under heavy CI load.\n\nInstead, you can traverse the active threads in the thread group using Thread.currentThread().getThreadGroup(), which is much more efficient and avoids capturing stack traces.
private static DatabaseAsyncExecutorImpl.AsyncThread findWorkerThread(final DatabaseInternal db, final int slot) {
final String name = "AsyncExecutor-" + db.getName() + "-" + slot;
ThreadGroup group = Thread.currentThread().getThreadGroup();
while (group.getParent() != null) {
group = group.getParent();
}
Thread[] threads = new Thread[group.activeCount() * 2];
final int count = group.enumerate(threads);
for (int i = 0; i < count; i++) {
if (threads[i] != null && name.equals(threads[i].getName())) {
return (DatabaseAsyncExecutorImpl.AsyncThread) threads[i];
}
}
throw new IllegalStateException("Worker thread " + name + " not found");
}|
@claude can you review the latest commits? |
|
Claude finished @lvca's task in 2m 42s —— View job Reviewing latest commits on PR #5081
(Note: Commit
|
…methods Round 2 fully closed out the round-1 asks with no correctness issues; the lone cosmetic nit taken: the capacity-1 test I added had landed after the private findWorkerThread helper without a separating blank line. Moved it up with the other @test methods and left the helper last, matching the class layout convention. Test-only reordering, no behavior change. AsyncFastQueueShutdownUndoTest 3/3.
Review: fix(#5066): map the 'fast' async queue to the MPMC DisruptorBlockingQueueThorough, well-argued fix. The root-cause analysis is correct and valuable: Verified locally: no dangling Note: I could not run Maven in this review environment (blocked), so statements about test-green status rely on the PR's stated verification rather than a re-run here. Points worth a look1. Confirm the capacity-1 guard is actually green in CI (medium). 2. Pre-existing divide-by-zero in the back-pressure calc (low, not introduced here). 3. Test message overstates what is verified (nit). 4. Performance / operationalThe Overall this is a solid, correct fix that closes the documented gap rather than papering over it. My only "please confirm" is point 1 (CI-green on capacity-1 against this library version); the rest are follow-ups/nits. |
…akly-consistent 0/0 snapshot
Round 3 endorsed the fix; taking the pre-existing divide-by-zero it
flagged (point 2), because THIS PR widens its window: the back-pressure
gauge reads remainingCapacity() and size() as separate calls, and the new
'fast' DisruptorBlockingQueue reports them as weakly-consistent ESTIMATES
(unlike ArrayBlockingQueue's exact counts), so a full-then-drained race
between the two reads can present remaining=0, size=0 -> ArithmeticException
on a live back-pressure path.
Extracted both duplicated inline forms into queueFullPercentage(queue):
each of remainingCapacity()/size() is read ONCE into a local (the old form
called each twice, compounding the skew) and the denominator is guarded
with Math.max(1, ...). Package-visible so the racy snapshot is testable.
Red-first: backPressureGaugeSurvivesTheRacyZeroDenominatorSnapshot drives
the gauge with a fake queue reporting 0/0 (the real TOCTOU cannot be
forced) - ArithmeticException without the guard, a sane 0-100 with it;
plus half-full=50 and full=100 sanity points.
Also took the nit (point 3): the concurrent-producers assertion message
now says 'must all execute' - a CountDownLatch detects loss, not
duplicates. Point 1 (capacity-1 green on disruptor 1.2.21) confirmed by
the local run; point 4 (findWorkerThread stack-dump) left as-is per the
review ('fine as-is for a regression test').
AsyncFastQueueShutdownUndoTest 4/4, AsyncTest and the helping/bail
backstops green.
Code Review — PR #5081: map the
|
…l-level note, thread-name coupling
Round 4 was another endorsement; the three doc-only minors taken (no code
behavior change):
1. queueFullPercentage javadoc now states the guarded 0/0 snapshot returns
100 ('full'), an ambiguous estimate biasing toward MORE back-pressure
for that one self-correcting iteration - so a future reader does not
assume it biases toward 'empty'.
2. Release note: for a non-power-of-two ASYNC_OPERATIONS_QUEUE_SIZE /
parallelLevel, 'fast' (Disruptor rounds up to a power of two) applies
back-pressure at a slightly different real fill level than 'standard'
(exact size) - operator-visible, not a regression.
3. findWorkerThread carries a comment pointing at the AsyncThread
constructor's thread-name format it couples to.
Point 3 from the review (idle CPU on high-performance profile) needed no
action - already documented. AsyncFastQueueShutdownUndoTest 4/4.
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
🟢 Coverage 83.33% diff coverage · -7.09% coverage variation
Metric Results Coverage variation ✅ -7.09% coverage variation Diff coverage ✅ 83.33% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (f3fab92) 135373 100999 74.61% Head commit (c4cd00b) 167657 (+32284) 113202 (+12203) 67.52% (-7.09%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#5081) 6 5 83.33% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Code Review: fix(#5066) map the 'fast' async queue to DisruptorBlockingQueueOverall this is a high-quality, well-reasoned fix. It correctly diagnoses the root cause one level deeper than the issue suggested: Strengths
Considerations (minor, mostly for the record)
Nits
No blocking concerns. The concurrency contract fix is correct, the change is minimal and well-scoped, and the tests are convincing. LGTM. |
… cold path; CPU note The review's one genuine pre-merge want, taken: capacity 1 is production-reachable (ASYNC_OPERATIONS_QUEUE_SIZE / parallelLevel floors to 1 on large pools) and the Conversant ring's minimum-size rounding is library-version-specific, so the stated guarantee is now pinned by fastQueueSupportsEffectiveCapacityOne - a parked-worker probe of the real 1-slot queue: offer, reject-when-full (not block/throw), remove(Object), slot reuse. A future disruptor bump cannot silently regress the post-shutdown undo primitive at the production floor. Point 2 taken as the requested note at the removeQuietly call site: remove(Object) on the Disruptor queue is an O(n) whole-queue-locking scan, acceptable only on the rare dead-worker post-shutdown race - it must never migrate onto the steady-state scheduling path. Nit taken: the release note now states the fast-profile CPU tradeoff (busy-spin before park, same as the previous implementation) for operators. The thread-name coupling, ignored join return and no-slow-tag observations are the reviewer's own verifications, kept as-is. AsyncFastQueueShutdownUndoTest 3/3 plus the async battery under -Darcadedb.asyncOperationsQueueImpl=fast, all green. (cherry picked from commit 6580d19)
…methods Round 2 fully closed out the round-1 asks with no correctness issues; the lone cosmetic nit taken: the capacity-1 test I added had landed after the private findWorkerThread helper without a separating blank line. Moved it up with the other @test methods and left the helper last, matching the class layout convention. Test-only reordering, no behavior change. AsyncFastQueueShutdownUndoTest 3/3. (cherry picked from commit e82693e)
…akly-consistent 0/0 snapshot
Round 3 endorsed the fix; taking the pre-existing divide-by-zero it
flagged (point 2), because THIS PR widens its window: the back-pressure
gauge reads remainingCapacity() and size() as separate calls, and the new
'fast' DisruptorBlockingQueue reports them as weakly-consistent ESTIMATES
(unlike ArrayBlockingQueue's exact counts), so a full-then-drained race
between the two reads can present remaining=0, size=0 -> ArithmeticException
on a live back-pressure path.
Extracted both duplicated inline forms into queueFullPercentage(queue):
each of remainingCapacity()/size() is read ONCE into a local (the old form
called each twice, compounding the skew) and the denominator is guarded
with Math.max(1, ...). Package-visible so the racy snapshot is testable.
Red-first: backPressureGaugeSurvivesTheRacyZeroDenominatorSnapshot drives
the gauge with a fake queue reporting 0/0 (the real TOCTOU cannot be
forced) - ArithmeticException without the guard, a sane 0-100 with it;
plus half-full=50 and full=100 sanity points.
Also took the nit (point 3): the concurrent-producers assertion message
now says 'must all execute' - a CountDownLatch detects loss, not
duplicates. Point 1 (capacity-1 green on disruptor 1.2.21) confirmed by
the local run; point 4 (findWorkerThread stack-dump) left as-is per the
review ('fine as-is for a regression test').
AsyncFastQueueShutdownUndoTest 4/4, AsyncTest and the helping/bail
backstops green.
(cherry picked from commit 248acdd)
…l-level note, thread-name coupling
Round 4 was another endorsement; the three doc-only minors taken (no code
behavior change):
1. queueFullPercentage javadoc now states the guarded 0/0 snapshot returns
100 ('full'), an ambiguous estimate biasing toward MORE back-pressure
for that one self-correcting iteration - so a future reader does not
assume it biases toward 'empty'.
2. Release note: for a non-power-of-two ASYNC_OPERATIONS_QUEUE_SIZE /
parallelLevel, 'fast' (Disruptor rounds up to a power of two) applies
back-pressure at a slightly different real fill level than 'standard'
(exact size) - operator-visible, not a regression.
3. findWorkerThread carries a comment pointing at the AsyncThread
constructor's thread-name format it couples to.
Point 3 from the review (idle CPU on high-performance profile) needed no
action - already documented. AsyncFastQueueShutdownUndoTest 4/4.
(cherry picked from commit c4cd00b)
Fixes #5066.
Verdict per option in the issue
The issue offered two shapes: a second drain pass on shutdown, or documenting the limitation. Investigation found the real defect one level deeper, which makes both options the wrong fix:
fastimpl violated its own concurrency contract in normal operation.PushPullBlockingQueueis an explicit single-producer/single-consumer design ("Transfers from a single thread writer to a single thread reader", no CAS on the tail sequence). Every async worker queue here has MANY producers: any application thread callingscheduleTask, cross-scheduling workers,waitCompletionmarkers, the closing thread. This is not theoretical - the new multi-producer regression test demonstrated lost tasks empirically on the old impl (8 producers x 250 no-op tasks, completion latch never reached zero). Note thehigh-performanceprofile selectsfast, so this was not an obscure opt-in corner.executorThreadsbefore shutdown can offer arbitrarily late, so no finite number of drain passes closes the window; the producer-side remove-undo is the only shape that works regardless of timing - and it needsremove(Object).The fix
arcadedb.asyncOperationsQueueImpl=fastnow maps toDisruptorBlockingQueue, the flagship MPMC variant from the same Conversant jar (no new dependency, same Apache 2.0 artifact):remove(Object)is implemented (cursor-blocked scan-and-shift), soscheduleTask's post-shutdown undo now works identically on both queue impls - the fix(#4953,#4954,#4961): async executor stall detector, shutdown draining, concurrency cleanups #5062-documented residual gap is gone;Capacity.getCapacity), matching small test configs; capacities round up to the next power of two, as the old ring already did.removeQuietly'sUnsupportedOperationExceptioncatch is kept as defensive-only for hypothetical future impls, with its comment and operator WARNING updated.Red-first evidence
AsyncFastQueueShutdownUndoTest(both red on the old impl, green after the swap):fastQueueMustSupportRemoveForThePostShutdownUndo: the exact primitive the undo depends on - threwUnsupportedOperationExceptionbefore.fastQueueMustNotLoseTasksUnderConcurrentProducers: timed out with lost tasks before; deterministically green on a compliant MPMC impl.Verification
mvn -pl engine compile; the fullcom.arcadedb.database.asyncbattery plusAsyncTestrun twice: default config (36 tests) and the fast-queue variant via-Darcadedb.asyncOperationsQueueImpl=fast(36 tests) - all green, so every existing shutdown/drain/stall/helping regression also passes on the Disruptor queue. Release notes updated to record that the formerly documented fast-queue gap is fixed.