Skip to content

fix: convert anonymous classes to named inner classes for Jackson serialization - #20070

Open
zhang-arvin wants to merge 2 commits into
apache:masterfrom
zhang-arvin:fix/7599-aggregator-factory-jackson
Open

zhang-arvin wants to merge 2 commits into
apache:masterfrom
zhang-arvin:fix/7599-aggregator-factory-jackson

Conversation

@zhang-arvin

Copy link
Copy Markdown

Purpose

Fix Jackson serialization failure for Long/Double/Float First/Last aggregator factories.

Anonymous inner classes used as NIL_AGGREGATOR and NIL_BUFFER_AGGREGATOR were being serialized
as LongLastAggregatorFactory$1 instead of the registered @JsonTypeName. This caused the error:

Could not resolve type id 'LongLastAggregatorFactory$1' into a subtype of AggregatorFactory

Changes

Converted all anonymous subclasses of Aggregator and BufferAggregator in the following 6 factory
classes to private static named inner classes:

  • LongLastAggregatorFactory
  • DoubleLastAggregatorFactory
  • FloatLastAggregatorFactory
  • LongFirstAggregatorFactory
  • DoubleFirstAggregatorFactory
  • FloatFirstAggregatorFactory

Each factory previously had two anonymous classes (NIL_AGGREGATOR and NIL_BUFFER_AGGREGATOR)
that overrode aggregate() to be no-ops. These are now named inner classes (e.g.,
NilLongLastAggregator, NilLongLastBufferAggregator) that correctly extend their parent classes
with explicit constructors.

Closes #7599

…ialization

Anonymous subclasses in Long/Double/Float First/Last aggregator factories
were being serialized as LongLastAggregatorFactory$1 instead of the
registered @JsonTypeName. This caused 'Could not resolve type id' errors
during Jackson deserialization.

Converted all NIL_AGGREGATOR and NIL_BUFFER_AGGREGATOR anonymous classes
to private static named inner classes in all 6 factory classes.

Closes apache#7599

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 1
P2 2
P3 0
Total 3

Reviewed 6 of 6 changed files.

The review found one unresolved Jackson-type risk, two remaining anonymous NIL paths, and missing serialization regression coverage.


This is an automated review by Codex GPT-5.6-Luna(max)

)
private static final Aggregator NIL_AGGREGATOR = new NilLongLastAggregator();

private static class NilLongLastAggregator extends LongLastAggregator

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Named NIL class does not fix the Jackson type ID

NilLongLastAggregator extends LongLastAggregator, not AggregatorFactory. The Jackson subtype registration applies to the factory, while the static NIL field is not serialized. If the NIL object is serialized directly, it still has no registered longLast type, so the change does not demonstrably fix the reported serialization failure.

NilColumnValueSelector.instance(),
false
)
private static final Aggregator NIL_AGGREGATOR = new NilDoubleFirstAggregator();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Anonymous NIL implementations remain elsewhere

The conversion is incomplete: anonymous NIL implementations remain in StringFirstAggregatorFactory, StringLastAggregatorFactory, and the numeric Any factories. The same class-name serialization risk therefore remains for those absent-column paths if runtime aggregators are serialized.

NilColumnValueSelector.instance(),
false
)
private static final Aggregator NIL_AGGREGATOR = new NilLongLastAggregator();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] No serialization regression test covers the fix

Existing serde tests only deserialize hard-coded factory JSON; they do not serialize a factory or exercise NIL factorization. Add round-trip coverage for the affected types and the missing-column branch.

The existing testSerde never serializes, so it cannot observe the type id
Jackson actually writes. A regression that put an anonymous class on the
jackson-facing surface would emit an id like LongLastAggregatorFactory$1
and the old test would still pass.

Adds:
- testSerdeRoundTrip: serialize -> assert no generated $N class name in the
  JSON -> deserialize -> assert equality, name, cache key, and that a second
  round trip is stable.
- testNilAggregatorIsUsedForMissingColumn: exercises the NilColumnValueSelector
  branch of factorize(), asserts the shared instance is reused, that
  aggregate() is a no-op, and that serializing the factory never surfaces the
  NIL aggregator's class.

Addresses review feedback on apache#20070.
@zhang-arvin

Copy link
Copy Markdown
Author

Thanks @FrankChen021 — the review is right on all three counts, and the P1 in particular goes deeper than the diff does. Here is where I landed after tracing it.

P1 — the change does not fix the reported failure

Confirmed. NilLongLastAggregator extends LongLastAggregator, not AggregatorFactory, and NIL_AGGREGATOR is only ever returned from factorize()/factorizeBuffered() as a runtime object. Jackson never sees it:

  • AggregatorFactory is what carries @JsonSubTypes registration (AggregatorsModule line 139+); neither Aggregator nor BufferAggregator is registered.
  • NIL_AGGREGATOR appears exactly once outside its declaration: return NIL_AGGREGATOR; at line 118.
  • Therefore renaming the anonymous class cannot change any type id Jackson writes.

This also means the type id in the report — LongLastAggregatorFactory$1 — must have come from a factory subclass, not an aggregator subclass. That code path no longer exists in the tree: #7599 was actually fixed by #10332, which touched StringFirstAggregateCombiner/StringLastAggregateCombiner and EarliestLatestAnySqlAggregator. #7599 is already closed and merged via that PR, so the Closes #7599 here was referencing an already-resolved issue.

So this PR's premise does not hold. I did not want to silently drop the work, though, because the reviewability problem you identified is real, so I kept the mechanical rename (it is harmless and arguably clearer) and added the thing that was actually missing:

P2 #3 — added the regression coverage

The existing testSerde only deserializes a hard-coded JSON string; it never serializes, so it structurally cannot see the type id Jackson emits. Added in this PR:

  • testSerdeRoundTrip — serialize → assert the JSON contains no generated $N class name → deserialize → assert equality/name/cache key → assert a second round trip is byte-stable.
  • testNilAggregatorIsUsedForMissingColumn — drives the NilColumnValueSelector branch, asserts the shared NIL instance is reused (assertSame), that aggregate() is a no-op, and that serializing the factory never surfaces NilLongLastAggregator.

mvn -o -pl processing test -Dtest=LongLastAggregationTest → 12 tests, 0 failures. I verified the new serialization assertion actually fails if an anonymous class is put on the jackson-facing surface, so it is a real guard rather than a tautology.

P2 #2 — confirmed, and I left it out deliberately

You are right that StringFirstAggregatorFactory, StringLastAggregatorFactory, DoubleAnyAggregatorFactory, LongAnyAggregatorFactory and FloatAnyAggregatorFactory still use anonymous NIL classes. I checked all 11 NIL-bearing factories: 6 are named (the ones this PR touches), 5 are still anonymous.

I deliberately did not extend the rename to them, because on the analysis above those anonymous classes are equally unreachable from Jackson — extending the rename would multiply the diff by ~3x while fixing nothing verifiable. If you'd rather have the sweep for uniformity, say so and I'll do it in one commit; I just don't want to justify it with a bug that does not exist.

Happy to close this PR instead if you judge that the right call — the test coverage is the only part with real value, and I can carry that into a fresh PR titled for what it actually does.

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The current head still does not address the failure described in the PR. The six renamed NIL classes are runtime Aggregator/BufferAggregator implementations returned by factorization; the Jackson subtype registration is for AggregatorFactory, and the factory serialization path exposes only its configured properties. Renaming these classes therefore cannot change the reported AggregatorFactory type id. The new tests serialize the factory itself, so they pass without exercising the production change.

Reviewed 7 of 7 changed files:

  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/first/DoubleFirstAggregatorFactory.java
  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/first/FloatFirstAggregatorFactory.java
  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/first/LongFirstAggregatorFactory.java
  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/last/DoubleLastAggregatorFactory.java
  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/last/FloatLastAggregatorFactory.java
  • processing/src/main/java/org/apache/druid/query/aggregation/firstlast/last/LongLastAggregatorFactory.java
  • processing/src/test/java/org/apache/druid/query/aggregation/firstlast/last/LongLastAggregationTest.java

Review scope included the incremental test commit, the full current merge-base diff, the prior review findings and author response, the AggregatorFactory Jackson mixin, the factory property accessors, the NIL factorization paths, and the numeric aggregator base classes.

Validation: static review only; git diff --check passed for the current diff.

Severity Findings
P0 0
P1 1
P2 1
P3 0
Total 2

This is an automated review by Codex GPT-5.6-Luna(max)

After addressing the findings or replying to the comments, you can request another review from me to trigger a new automated review.

NilColumnValueSelector.instance(),
false
)
private static final Aggregator NIL_AGGREGATOR = new NilLongLastAggregator();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Named NIL class does not fix the Jackson type ID

Finding: The new NilLongLastAggregator is still an Aggregator, while Jackson's subtype registration is on AggregatorFactory; the only references to this static instance are the factorize/factorizeBuffered return paths, and the factory serializer exposes only its name, fieldName, and timeColumn properties. Consequently, this rename cannot change the AggregatorFactory type id or resolve the reported LongLastAggregatorFactory$1 failure; if the runtime aggregator is serialized through another path, the new private class is still not registered.

Suggestion: Trace the actual object being serialized and fix or register that Jackson-facing type; otherwise retitle or remove this no-op production change and keep only a regression test for the real failure.

public void testSerdeRoundTrip() throws Exception
{
final DefaultObjectMapper mapper = new DefaultObjectMapper();
final String serialized = mapper.writeValueAsString(longLastAggFactory);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Round-trip tests do not exercise the changed NIL classes

Finding: Both added assertions serialize longLastAggFactory, not the Aggregator or BufferAggregator returned for a missing column, so the JSON only contains factory properties and would be identical before this refactor. The only missing-column test calls the on-heap factorize path for LongLast; the five other factories and all six NIL_BUFFER_AGGREGATOR paths remain uncovered. This lets regressions in the changed classes pass unnoticed and does not validate the claimed fix.

Suggestion: Add a focused test for the actual Jackson-facing value and cover the buffer and remaining factory variants, or narrow the production change to code that the test demonstrably protects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Could not resolve type id 'LongLastAggregatorFactory$1' into a subtype of AggregatorFactory

2 participants