Repository navigation
Kafka Tasks unable restore persisted segments properly #7252
Description
Affected Version
All versions after #6431
Description
Kakfa indexing tasks (likely kinesis as well) are unable to restore properly if segments have been incrementally persisted because they cannot deserialize SequenceMetadata, resulting in an error in the form of:
2019-03-12T02:18:50,238 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
com.fasterxml.jackson.databind.JsonMappingException: Unrecognized Type: [null]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:269) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:428) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:164) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:25) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.DeserializationContext.handleSecondaryContextualization(DeserializationContext.java:669) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:466) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3838) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3732) ~[jackson-databind-2.6.7.jar:2.6.7]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2639) ~[jackson-databind-2.6.7.jar:2.6.7]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.restoreSequences(SeekableStreamIndexTaskRunner.java:942) ~[druid-indexing-service-0.14.0-iap-pre3.jar:0.14.0-iap-pre3]
...
The issue is that SequenceMetadata is a private class using generic types defined in the outer class, SeekableStreamIndexTaskRunner<PartitionIdType, SequenceOffsetType>, which determine the map types which store offsets in the persisted sequence metadata json file. The file is written correctly, but the TypeReference given to the object mapper is not able to properly read it during deserialization, resulting in the above error.
Fix underway, but untangling this is sort of messy unfortunately.
Activity