Fixed ClassCastException during Multi-Stage Queries on real-time segments by nozjkoitop · Pull Request #16607 · apache/druid

nozjkoitop · 2024-06-14T13:29:39Z

Fixed issue when msq queries throw ClassCastException with "includeSegmentSource": "REALTIME" and supervisor running.

Caused by: java.lang.ClassCastException: class [B cannot be cast to class org.apache.druid.query.aggregation.datasketches.hll.HllSketchHolder ([B is in module java.base of loader 'bootstrap'; org.apache.druid.query.aggregation.datasketches.hll.HllSketchHolder is in unnamed module of loader java.net.URLClassLoader @38792286)
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchHolderObjectStrategy.toBytes(HllSketchHolderObjectStrategy.java:31)
	at org.apache.druid.segment.serde.ComplexMetricSerde.toBytes(ComplexMetricSerde.java:119)
	at org.apache.druid.frame.field.ComplexFieldWriter.writeTo(ComplexFieldWriter.java:65)
	at org.apache.druid.frame.write.RowBasedFrameWriter.writeDataUsingFieldWriters(RowBasedFrameWriter.java:291)
	at org.apache.druid.frame.write.RowBasedFrameWriter.writeData(RowBasedFrameWriter.java:246)
	at org.apache.druid.frame.write.RowBasedFrameWriter.addSelection(RowBasedFrameWriter.java:122)
	at org.apache.druid.msq.querykit.scan.ScanQueryFrameProcessor.populateFrameWriterAndFlushIfNeeded(ScanQueryFrameProcessor.java:348)
	at org.apache.druid.msq.querykit.scan.ScanQueryFrameProcessor.populateFrameWriterAndFlushIfNeededWithExceptionHandling(ScanQueryFrameProcessor.java:329)
	at org.apache.druid.msq.querykit.scan.ScanQueryFrameProcessor.runWithLoadedSegment(ScanQueryFrameProcessor.java:231)
	at org.apache.druid.msq.querykit.BaseLeafFrameProcessor.runIncrementally(BaseLeafFrameProcessor.java:87)
	at org.apache.druid.msq.querykit.scan.ScanQueryFrameProcessor.runIncrementally(ScanQueryFrameProcessor.java:158)
	at org.apache.druid.frame.processor.FrameProcessors$1FrameProcessorWithBaggage.runIncrementally(FrameProcessors.java:75)
	at org.apache.druid.frame.processor.FrameProcessorExecutor$1ExecutorRunnable.runProcessorNow(FrameProcessorExecutor.java:230)
	... 8 more

After some investigation was found that selector's getObject() for real-time segments was returning a byte array representation of HllSketch but still was trying to convert it to a byte[]. While HllSketchHolderObjectStrategy expects to have HllSketchHolder instance in toBytes() leading to the ClassCastException.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

…hat include columns of HLLSketch type resulted in a ClassCastException.

LakshSingla · 2024-06-15T08:13:29Z

+    if (val == null) {
+      return ByteArrays.EMPTY_ARRAY;
+    } else if (val instanceof byte[]) {
+      return (byte[]) val;
+    } else {


I think this is incorrect because there might exist a serde where the deserialized value is byte[], however, the serialized value can be something else.

For example, different compression techniques can transform a deserialized byte[] array to a different serialized byte[] array. This would be unique to the complex type, so it shouldn't be a universal change. Instead, the following changes can be made:

If this only occurs in the HllSketchBuild serde, you can make this change specific to the HllSketchBuild serde.

Else, if this occurs with all the complex types while running MSQ tasks on complex metrics, you can make the changes in MSQ-specific classes (after confirming where the byte[] array is coming from).

Thanks for pointing that out! It looks like this issue might not be limited to just the HLLSketch. None of the ObjectStrategies seem to fit this scenario, as they all expect specific objects for the toBytes() method. Using it with a byte array directly would lead to the CCE. I'll look into why selector.getObject is returning a byte array for a real-time segment maybe it could be fixed there as well. It is not appears in MSQ classes directly tho, it shows up at ComplexFieldWriter.writeTo()

np! Since the issue is bubbling up from the column value selectors, you can check who creates those, and why they are returning a byte array instead of the desired object type.

I made some research and it looks like it's normal for selector.getObject to return a byte[] for real-time segments. This happens because it uses RowBasedColumnValueSelector instead of ObjectColumnSelector. It doesn't go through GenericIndexed.copyBufferAndGet() and strategy.fromByteBuffer(), and just grabs the byte[] directly from the ByteBuffer using rowSupplier.get() (at least in case of ComplexType from RealTime segment MSQ).

Right now, we don't have an ObjectStrategy that needs a byte[] as input, so initial setup would likely cause CCE failures. But you're right, we might need to handle this in the future. So, I've moved this check to the ObjectStrategy as a defaul void, making it easy to override if needed.

  {
    if (val != null) {
-      byte[] bytes = getObjectStrategy().toBytes(val);
+      byte[] bytes = getObjectStrategy().objectToBytes(val);


…ith-hllsketch-in-realtime

LakshSingla · 2024-06-25T04:29:19Z

Thanks for the changes @nozjkoitop. I'll review them shortly.

nozjkoitop · 2024-07-05T11:24:34Z

Thanks for the changes @nozjkoitop. I'll review them shortly.

@LakshSingla Could I remind you to have a look on that please?

LakshSingla · 2024-07-11T04:48:04Z

Sorry for the delay @nozjkoitop and thanks for the update 🚀
Can you please add a test in the MSQSelectTest or MSQInsertTest to validate that the PR fails before with the exception you are seeing and passes after this patch? This should be easy to reproduce. LMK if you need help with that.

The new approach is a lot cleaner than the previous one, but I still wanna see if there's a way to restrict the change to MSQ, since I suspect that there's a faulty selector at play. The MSQ test case would help in debugging this as well.

…ith-hllsketch-in-realtime

nozjkoitop · 2024-07-12T16:59:28Z

Can you please add a test in the MSQSelectTest or MSQInsertTest to validate that the PR fails before with the exception you are seeing and passes after this patch? This should be easy to reproduce.

I've added new test-case with hyperUnique column in select from realtime segment, please have a look

nozjkoitop · 2024-07-24T06:57:59Z

@LakshSingla WDYT?

LakshSingla · 2024-07-24T07:02:11Z

Hey @nozjkoitop apologies for the delay. Let me take a look at the patch and get back to you in a couple of days. It seems correct for the most part. I just need to think if there's a less invasive way to do this given that the regression is in MSQ. I wasn't able to get to it due to a few other patches I was working on, but I'll prioritize this change. Thanks a lot for being patient with me!

nozjkoitop · 2024-08-27T08:17:44Z

Hey @LakshSingla, just a kindly reminder about this PR :)

adarshsanjeev · 2024-09-27T09:08:18Z

@nozjkoitop I think @LakshSingla's comment was to only make changes that affect MSQ, since the bug is seen only there. I think your changes might affect other queries as well.

nozjkoitop · 2024-09-27T10:15:08Z

@nozjkoitop I think @LakshSingla's comment was to only make changes that affect MSQ, since the bug is seen only there. I think your changes might affect other queries as well.

Hi @adarshsanjeev ! The main issue is in ComplexFieldWriter, and it's caused by the ValueSelector potentially returning a byte[], which the writer doesn't expect. This needs to be handled, and could be addressed it in either the writeTo() method or serde.toBytes(). Both of these classes are part of Druid's processing, and honestly, I don’t see any other places that would be more isolated for handling this.

github-actions · 2024-11-27T00:22:26Z

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

…ith-hllsketch-in-realtime # Conflicts: # extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java

nozjkoitop · 2024-12-18T16:47:29Z

Hi @LakshSingla I've checked the latest druid and described issue is still there, although now there is a NPE on the ui, but provided patch still resolves an issue, so I merged the changes from master. Please take a look.

github-actions · 2025-02-17T00:22:43Z

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

nozjkoitop · 2025-03-05T14:14:04Z

Issue is still there in v32, merged master to solve conflicts

github-actions · 2025-05-05T00:25:29Z

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

github-actions · 2025-06-02T00:26:03Z

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

gianm · 2025-06-03T19:11:49Z

Something seems off with the rebase on master, there's a bunch of extra files. That being said, it seems to me like this error points at a situation where some aspect of the design is sketchy. I will reopen it. @adarshsanjeev has been looking recently into realtime queries in MSQ- would you mind taking a look at this bug?

adarshsanjeev · 2025-06-04T06:17:32Z

Sure, I can take a look at this bug

nozjkoitop · 2025-06-04T07:57:00Z

@adarshsanjeev i see that i messed up the merging before, now ive reset the branch to here and merged it properly. Shall i push that?

…ith-hllsketch-in-realtime

github-actions · 2025-08-11T00:28:07Z

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

github-actions · 2025-09-09T00:23:24Z

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

Fixed issue where running Multi-Stage Queries on real-time segments t…

d62532b

…hat include columns of HLLSketch type resulted in a ClassCastException.

github-actions Bot added the Area - Segment Format and Ser/De label Jun 14, 2024

Checkstyle fix

c3ac9dd

LakshSingla reviewed Jun 15, 2024

View reviewed changes

Moved parameter validation to ObjectStrategy

8271050

github-advanced-security AI found potential problems Jun 19, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/master' into feature-fix-msq-w…

0f17dd3

…ith-hllsketch-in-realtime

nozjkoitop requested a review from LakshSingla June 24, 2024 06:52

nozjkoitop added 2 commits July 12, 2024 10:36

Merge remote-tracking branch 'upstream/master' into feature-fix-msq-w…

7e902a3

…ith-hllsketch-in-realtime

Added test-case to MSQSelectTest

82fc405

github-actions Bot added Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jul 12, 2024

github-actions Bot added the stale label Nov 27, 2024

nozjkoitop added 2 commits December 18, 2024 13:14

Merge remote-tracking branch 'upstream/master' into feature-fix-msq-w…

24fea78

…ith-hllsketch-in-realtime # Conflicts: # extensions-core/multi-stage-query/src/test/java/org/apache/druid/msq/test/MSQTestBase.java

Added columnTypes to expected query

5cbd4ac

github-actions Bot removed the stale label Dec 19, 2024

github-actions Bot added the stale label Feb 17, 2025

github-actions Bot added the Area - Documentation label Mar 5, 2025

github-actions Bot added the Area - Web Console label Mar 5, 2025

github-actions Bot removed the stale label Mar 6, 2025

github-actions Bot added the stale label May 5, 2025

vivek807 added a commit to deep-bi/druid that referenced this pull request May 6, 2025

Cherry picked apache#16607.

ce67e74

github-actions Bot closed this Jun 2, 2025

gianm reopened this Jun 3, 2025

github-actions Bot removed the stale label Jun 4, 2025

nozjkoitop added 2 commits June 4, 2025 13:25

upstream merged

d4d8eda

Merge remote-tracking branch 'upstream/master' into feature-fix-msq-w…

7ffe8b5

…ith-hllsketch-in-realtime

nozjkoitop force-pushed the feature-fix-msq-with-hllsketch-in-realtime branch from 6ca88c6 to 7ffe8b5 Compare June 10, 2025 11:09

nozjkoitop added 5 commits June 10, 2025 13:16

Minor post-merge fix

27f5157

checkstyle

cd2e5e4

trigger checks

75c0af1

fix tests

f2a1c38

checkstyle

f930236

github-actions Bot added the stale label Aug 11, 2025

github-actions Bot closed this Sep 9, 2025

nozjkoitop mentioned this pull request Dec 19, 2025

Fixed ClassCastException during Multi-Stage Queries on real-time segments #18859

Closed

4 of 9 tasks

nozjkoitop mentioned this pull request Jun 11, 2026

Realtime scans from MSQ cannot reliably read complex types #18340

Open

Conversation

nozjkoitop commented Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LakshSingla Jun 15, 2024

Choose a reason for hiding this comment

Uh oh!

nozjkoitop Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

LakshSingla Jun 17, 2024

Choose a reason for hiding this comment

Uh oh!

nozjkoitop Jun 19, 2024

Choose a reason for hiding this comment

Uh oh!

Check notice

LakshSingla commented Jun 25, 2024

Uh oh!

nozjkoitop commented Jul 5, 2024

Uh oh!

LakshSingla commented Jul 11, 2024

Uh oh!

nozjkoitop commented Jul 12, 2024

Uh oh!

nozjkoitop commented Jul 24, 2024

Uh oh!

LakshSingla commented Jul 24, 2024

Uh oh!

nozjkoitop commented Aug 27, 2024

Uh oh!

adarshsanjeev commented Sep 27, 2024

Uh oh!

nozjkoitop commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 27, 2024

Uh oh!

nozjkoitop commented Dec 18, 2024

Uh oh!

github-actions Bot commented Feb 17, 2025

Uh oh!

nozjkoitop commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 5, 2025

Uh oh!

github-actions Bot commented Jun 2, 2025

Uh oh!

gianm commented Jun 3, 2025

Uh oh!

adarshsanjeev commented Jun 4, 2025

Uh oh!

nozjkoitop commented Jun 4, 2025

Uh oh!

github-actions Bot commented Aug 11, 2025

Uh oh!

github-actions Bot commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nozjkoitop commented Jun 14, 2024 •

edited

Loading

nozjkoitop commented Sep 27, 2024 •

edited

Loading

nozjkoitop commented Mar 5, 2025 •

edited

Loading