Repository navigation
Realtime scans from MSQ cannot reliably read complex types #18340
Description
When ScanQueryFrameProcessor delegates work to a realtime server, it sends a scan query and writes the response into a frame. Their serialization when selected in a scan query is done by Jackson, and it is common for such objects to have "one-way" serialization (they can be serialized using Jackson, but cannot be deserialized).
For aggregating queries, typically these objects are deserialized by the associated aggregator's AggregatorFactory#deserialize method rather than by Jackson. This is problematic for scan queries, because they don't use AggregatorFactory. This leads to errors when attempting to write the resulting objects to a frame, because they aren't the expected type.
To fix it, we need to serialize and deserialize these complex objects in a coherent way. There are a couple options that come to mind,
- We could add a new parameter to
scanthat causes it to serialize complex objects usingTypeStrategyrather than allowing Jackson to handle it. MSQ realtime delegation would use this parameter. Then, when reading, we use the sameTypeStrategyto deserialize the object. - We could rewrite
scanqueries in MSQ's realtime delegation code to wrap all complex types incomplex_encode_base64, which would accomplish more or less the same thing- it would use theTypeStrategyto serialize the object.
This bug affects all MSQ queries that use realtime data and delegate scan involving complex types. An example query would be like:
SELECT "hllSketch"
FROM tbl
WHERE "foo" = 'bar'Queries that involve aggregations on complex types wouldn't be affected.
Activity
gianm commentedon Aug 6, 2025
Note that this bug affects scan pushdown only. Aggregation pushdown, which is more commonly what happens with complex types, is not affected.
kgyrtkirk commentedon Jan 12, 2026
I will move this out from 36.0.0 (into 37.0.0) for now - it doesn't seem like a new issue which should block the release.
cecemei commentedon Apr 7, 2026
@gianm whats the status for this issue? should we move this to 38?
gianm commentedon Apr 8, 2026
Yes let's move it to 38.
nozjkoitop commentedon Jun 11, 2026
Hey @gianm
This looks a lot like the issue I was seeing before. You might want to take a look at #16607 (later refreshed as #18859), it fixes a very similar MSQ realtime path problem with complex types/HLL sketches
Not sure it covers every case here, but it seems closely related, it might be worth reopening #18859 and merging the latest changes rather than starting from scratch
kfaraz commentedon Aug 5, 2026
Moving this milestone from 38.0.0 to 39.0.0 as there has not been a PR yet.
zhang-arvin commentedon Aug 20, 2026
I'd like to work on this issue. Could you please assign it to me?
fix: handle byte[] complex types from realtime segments in MSQ scans