Missing hadoop client dependency #8338

Affected Version

Noticed in version 0.14.2, looks like it affects version 0.14.0 through to 0.15.1

Description

Upgrading out test clusters to 0.14.2 resulted in Hadoop Ingestion Tasks failing with the below stack trace. It looks like since 0.14.0 the dependency hadoop-mapreduce-client-core-{$version}.jar is no longer present in the extension druid-hdfs-storage.

This dependency is present in the hadoopDependencies folder, however if my understanding of the classloader code is right, we purposefully do not include these jars in the mapreduce job's classpath.

Looks related to #6828 and #6967
java.lang.NoSuchMethodError: org.apache.hadoop.mapreduce.jobhistory.EventWriter.<init>(Lorg/apache/hadoop/fs/FSDataOutputStream;)V at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.createEventWriter(JobHistoryEventHandler.java:420) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.setupEventWriter(JobHistoryEventHandler.java:459) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:549) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$1.run(JobHistoryEventHandler.java:318) ~[hadoop-mapreduce-client-app-2.7.3.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]