Repository navigation
Error in coordinator/overlord when loading scan-query #4835
Description
I'm getting errors like this
2017-09-21T21:17:34.025034+00:00 monitowl-dev wh_druid_overlord[27859]: Exception in thread "main" com.google.inject.CreationException: Unable to create injector, see the following errors:
2017-09-21T21:17:34.025617+00:00 monitowl-dev wh_druid_overlord[27859]: 1) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.
2017-09-21T21:17:34.026017+00:00 monitowl-dev wh_druid_overlord[27859]: while locating io.druid.query.GenericQueryMetricsFactory
2017-09-21T21:17:34.026441+00:00 monitowl-dev wh_druid_overlord[27859]: for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)
2017-09-21T21:17:34.026811+00:00 monitowl-dev wh_druid_overlord[27859]: at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:36) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.027141+00:00 monitowl-dev wh_druid_overlord[27859]: 2) No implementation for io.druid.query.GenericQueryMetricsFactory was bound.
2017-09-21T21:17:34.027372+00:00 monitowl-dev wh_druid_overlord[27859]: while locating io.druid.query.GenericQueryMetricsFactory
2017-09-21T21:17:34.027621+00:00 monitowl-dev wh_druid_overlord[27859]: for the 1st parameter of io.druid.query.scan.ScanQueryQueryToolChest.<init>(ScanQueryQueryToolChest.java:48)
2017-09-21T21:17:34.027988+00:00 monitowl-dev wh_druid_overlord[27859]: while locating io.druid.query.scan.ScanQueryQueryToolChest
2017-09-21T21:17:34.028260+00:00 monitowl-dev wh_druid_overlord[27859]: for the 1st parameter of io.druid.query.scan.ScanQueryRunnerFactory.<init>(ScanQueryRunnerFactory.java:52)
2017-09-21T21:17:34.028492+00:00 monitowl-dev wh_druid_overlord[27859]: at io.druid.query.scan.ScanQueryDruidModule.configure(ScanQueryDruidModule.java:41) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.query.scan.ScanQueryDruidModule)
2017-09-21T21:17:34.028749+00:00 monitowl-dev wh_druid_overlord[27859]: 2 errors
2017-09-21T21:17:34.028975+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
2017-09-21T21:17:34.029211+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
2017-09-21T21:17:34.029438+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
2017-09-21T21:17:34.029665+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:99)
2017-09-21T21:17:34.029944+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:73)
2017-09-21T21:17:34.030185+00:00 monitowl-dev wh_druid_overlord[27859]: #011at com.google.inject.Guice.createInjector(Guice.java:62)
2017-09-21T21:17:34.030399+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:390)
2017-09-21T21:17:34.030678+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:63)
2017-09-21T21:17:34.030892+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40)
2017-09-21T21:17:34.031104+00:00 monitowl-dev wh_druid_overlord[27859]: #011at io.druid.cli.Main.main(Main.java:108)while having scan-query extension in druid.extensions.loadList, which is in the _common/common.runtime.properties file.
I've confirmed that this does not affect historicals, nor brokers and that removing scan-query from extensions fixes the problem.
Is this expected? Should I be splitting the extensions definitions for different nodes now? What I'm worried about that our local tasks [or middle managers, or whatever actually serves incoming data :-)] (spawned by overlord) will not see it, then, which will be no good.
Activity
KenjiTakahashi commentedon Sep 22, 2017
Oh, I forgot! This is on 0.10.1.
stevenchen3 commentedon Sep 25, 2017
Got the same here.
almike commentedon Sep 25, 2017
yeah this is a major issue for us as well
vogievetsky commentedon Sep 26, 2017
It appears that in Druid 0.10.1 (vs 0.10.0) non-query nodes (coordinator, overlord) can not load query extensions.
Previously in 0.10.0 they were simply ignored.
This is technically a regression I pray to the Druid gods that it will be fixed in 0.11.0
There is a very simple work-around for now: only load the scan query on the query nodes, broker, historical, etc.
almike commentedon Sep 26, 2017
thanks @vogievetsky, we will try that out!
KenjiTakahashi commentedon Sep 26, 2017
@vogievetsky Thanks, that's what I figured out as well. Still not sure about my question at the bottom of the first post, though. It is not clear to me what will happen in such scenario.
leventov commentedon Sep 26, 2017
This is caused by #4429 and IMO it's not a regression, it's a first step towards not loading everything everywhere. So not loading scan-query on coordinator and overlord is not a "workaround", it's "the right thing to do".
gianm commentedon Sep 26, 2017
@leventov I don't agree, it's a real pain for users if they need to specify a different list of extensions for each node type. Especially since new users often don't fully understand what node types are doing what jobs. In the past, extensions could be loaded everywhere and they would just end up doing nothing if they don't apply to a particular node type.
I guess it does make sense to not actually load everything everywhere for efficiency sake, but for users' sanity, how about instead having extensions be able to declare what kinds of node types they should be loaded on.
KenjiTakahashi commentedon Sep 26, 2017
I am fine with specifying different extensions for different nodes (although @gianm argument makes sense, too), I just need to know what will happen when a task spawned locally by overlord needs extension that overlord itself cannot load? My understanding is that currently, overlord's config is being passed down to local tasks.
BTW: The
scan-querycase should "fix itself" with the next release, because it got merged to core.gianm commentedon Sep 26, 2017
@KenjiTakahashi, if the extension must be serialized as part of the task (for example a custom firehose) then the overlord must be able to load it. The scan query isn't needed to be serialized as part of the task, so, it doesn't need to be loaded on the overlord.
KenjiTakahashi commentedon Sep 26, 2017
Yes, but how will a task know that it has to be loaded? Maybe I'm not being clear here, sorry. I mean we're using the
index_realtimetasks spawned locally on overlord. There is no separatepropertiesfile for tasks/middlemanagers/<whatever is below overlord>. But still I should be able to query the incoming data, right? How would I use scan-query with that, if it's not specified in overlord config?leventov commentedon Sep 26, 2017
@KenjiTakahashi you can configure custom properties for forked task JVMs, but not overlord using
druid.indexer.fork.property, see http://druid.io/docs/latest/configuration/indexing-service.htmlKenjiTakahashi commentedon Sep 28, 2017
So I can pass any property through this directly to task, bypassing overlord itself? That makes sense, gonna try it out, thanks.
gianm commentedon Jan 9, 2018
Should be irrelevant now since scan-query became merged into the core code in #4751.
jkukul commentedon Mar 26, 2018
Could
0.10.1release notes be updated with a statement like:Query extensions must be removed from the list of extensions on non-query nodes (coordinator, overlord).
This would save some trouble for users upgrading from
0.10.0(and earlier).lamberken commentedon Mar 9, 2020
hi @gianm, got the same in
0.17.0#9483
anandgm007 commentedon Aug 6, 2020
Same Issue in 0.18.1.
Any Resolution/Work-around for this issue?
gianm commentedon Aug 6, 2020
@anandgm007 If you are having the issue from the original description, you can fix it by removing
scan-queryfrom your extensions list. It is part of core now.anandgm007 commentedon Aug 7, 2020
@gianm Thanks for the reply. I don;t have scan-query in the extensions List. Found out my issue is exactly mentioned here in 9483. Wanted to add the moving average extension and i changed the load_extension in broker alone.
Here is my load_extention in _common, common.runtime.properties
druid.extensions.loadList=["druid-s3-extensions", "mysql-metadata-storage","druid-hdfs-storage", "druid-kafka-indexing-service", "druid-datasketches","druid-moving-average-query"]
Getting the following error in router.log when i restart the broker service
2020-08-06T13:52:10,538 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
while locating com.google.inject.Provider<org.apache.druid.query.QuerySegmentWalker>
for the 1st parameter of org.apache.druid.query.movingaverage.MovingAverageQueryToolChest.(MovingAverageQueryToolChest.java:61)
at org.apache.druid.query.movingaverage.MovingAverageQueryModule.configure(MovingAverageQueryModule.java:48) (via modules: com.google.inject.u
til.Modules$OverrideModule -> org.apache.druid.query.movingaverage.MovingAverageQueryModule)
Should i include "druid-moving-average-query" in the overload also? right now its just the broker.
gianm commentedon Aug 12, 2020
I think for the moving-average-query in particular, it should only be in the broker/runtime.properties, not the other services.