[FLINK-39190] Relocate Jackson in filesystem plugin POMs to prevent classpath conflicts

[driessamyn-sparta]

What is the purpose of the change

This PR relocates Jackson in filesystem plugin shade configurations

All Hadoop-based filesystem plugins (s3-hadoop, s3-presto, azure, gs, oss) pull in jackson-databind 2.17.x transitively via AWS SDK v1 or Hadoop, but do not relocate it. If the plugin JAR ends up on the job classpath, its older unshaded Jackson shadows any newer version used by the job, causing for example:
NoSuchFieldError: CLEAR_CURRENT_TOKEN_ON_CLOSE

StreamReadFeature.CLEAR_CURRENT_TOKEN_ON_CLOSE was added in Jackson 2.18.0, so any job using jackson-dataformat-csv 2.18+ fails at runtime with this error.

Brief change log

Added a com.fasterxml.jackson relocation to the maven-shade-plugin configuration of each affected module, using a module-specific shaded package (e.g. org.apache.flink.fs.s3hadoop.shaded.jackson), consistent with how flink-shaded-jackson handles the core runtime.

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes (no new dependencies, but changes how the are handled)
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: don't know

Documentation

• Does this pull request introduce a new feature? no

[flinkbot]

CI report:

• e229847 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[MartijnVisser]

@driessamyn-sparta Thnx for the patch. I do have some concerns, because what happens if the internal implementation of AWS SDK v1 relies on Jackson somewhere? Relocating Jackson would then break then the implementation.

[driessamyn-sparta]

Thnx for the patch. I do have some concerns, because what happens if the internal implementation of AWS SDK v1 relies on Jackson somewhere? Relocating Jackson would then break then the implementation.

Thanks, @MartijnVisser

My understanding is that given we shade the AWS SDK, relocation should work.
The shade config uses :, which means the AWS SDK v1 classes are bundled into the fat jar too. When the shade plugin applies the com.fasterxml.jackson -> org.apache.flink.fs.s3hadoop.shaded.jackson relocation, it rewrites all bytecode references in every included class, including the AWS SDK classes.
So the SDK's compiled Jackson calls are transparently redirected to the relocated package, which exists in the jar.

This is hard to verify using a unit test as the test would need to operate on the packaged jar.
However, we can inspect the fat jar to verify this manually after creating the fat jar:

JAR=$(find flink-filesystems/flink-s3-fs-hadoop/target \
      -name "flink-s3-fs-hadoop-*.jar" \
      -not -name "original-*" \
      -not -name "*sources*" | head -1)

echo "Inspecting: $JAR"
echo ""

echo "Check 1: Relocated Jackson classes are present ==="
jar tf "$JAR" | grep "org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper.class"
echo ""

echo "Check 2: Original com/fasterxml/jackson classes are NOT present ==="
ORIGINAL=$(jar tf "$JAR" | grep "^com/fasterxml/jackson" || true)
if [ -z "$ORIGINAL" ]; then
  echo "PASS: no com/fasterxml/jackson classes found in shaded jar"
else
  echo "FAIL: unrelocated Jackson classes found:"
  echo "$ORIGINAL" | head -5
fi
echo ""

echo "Check 3: AWS SDK bytecode references relocated Jackson ==="
MARKER="org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper"
if unzip -p "$JAR" "com/amazonaws/util/json/Jackson.class" 2>/dev/null | grep -qa "$MARKER"; then
  echo "PASS: com.amazonaws.util.json.Jackson references relocated Jackson ($MARKER)"
else
  echo "FAIL: relocated Jackson reference not found in AWS SDK class"
fi

Inspecting: flink-filesystems/flink-s3-fs-hadoop/target/flink-s3-fs-hadoop-2.3-SNAPSHOT.jar

Check 1: Relocated Jackson classes are present ===
org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper.class

Check 2: Original com/fasterxml/jackson classes are NOT present ===
PASS: no com/fasterxml/jackson classes found in shaded jar

Check 3: AWS SDK bytecode references relocated Jackson ===
PASS: com.amazonaws.util.json.Jackson references relocated Jackson (org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper)