Repository navigation
Query execution interleaving #8356
Description
The idea of query execution interleaving first popped up in Druid discussions in the context of parallel result combining in Brokers in the discussion between me and @jihoonson, see #6629 (comment).
In general, this is not a new idea to reduce head-of-line blocking effects. See The Tail at Scale, Dean and Barroso, 2013.
In Druid, interleaving may be implemented inside PrioritizedExecutorService. On Historicals, the element of processing is naturally a segment, when a Broker sends a request to a Historical to process many segments for a large query. On Brokers, interleaving may be implemented as discussed in #6629 (comment).
Interleaving will play especially well together with online aggregation (#7087) but would be useful even if implemented on its own independently of online aggregation.
Activity
clintropolis commentedon Sep 24, 2019
@leventov, I've opened a proposal issue for this based on that thread (still work in progress), #8577, as well as a basic implementation to prove it viable, though with a few missing pieces. I would totally appreciate your thoughts on where to take it next.