Repository navigation
[Proposal] Add batching support in kafka-indexing-service #4373
Description
It would benefit some users if kafka-indexing service supported the capability to let users put multiple druid InputRow inside single kafka record.
This allows users to do batching while still using kafka sync producer which allows only one kafka record at a time.
I would imagine adding following method to InputRowParser interface.
default List<InputRow> parseBatch(T input)
{
return ImmutableList.of(parse(input));
}
current InputRow parse(T input) method would be deprecated and all of druid code would be adjusted to use parseBatch(input) instead.
kafka-indexing service will need to persist <offset,row-number-in-record> pair instead of just offset to support exactly once. I believe, this will end up being the biggest change really.
This approach would add batching support in general and not just for kafka-indexing .
Activity
[-][Proposal] Add batching support in kafka-indexing[/-][+][Proposal] Add batching support in kafka-indexing-service[/+]gianm commentedon Aug 16, 2017
See also #3439.
himanshug commentedon Oct 5, 2017
We had few conversations internally and it appears tracking record-offset within kafka record introduces additional metadata to be tracked and increases code complexity for the Kafka Indexing Service related code. So, for now the proposal is to not track record-offset within kafka record and that means if
InputRowParser.parseBatch(kafka-record-bytes)returns more than oneInputRowthenexactly-oncebehavior is not guaranteed but best effort. This behavior could be changed in future if necessary.So, for now, a user who requires
exactly-onceguarantee would use batching support provided by kafka so that each kafka record stays single Druid row.@gianm does that sound ok?
pjain1 commentedon Oct 5, 2017
The parse batch thing in KafkaIndexTask can be put behind a config, which when set explicitly will enable parsing of batch. This config can be documented with the warning that, when enabled task does not support exactly once and can also be printed as warning message in the task logs.
gianm commentedon Oct 6, 2017
@himanshug In principle, it sounds ok to me to have some non-exactly-once-features as long as they're disabled by default and clearly marked. I think by default we should be exactly once and users should need to change configs somehow if they want to break that.
However there is a difficulty with dropping exactly-once behavior: the segment-allocate stuff (pending segments etc) assumes that replica tasks will do exactly the same thing given the same set of inputs. If they don't do exactly the same thing then they can get out of sync with regard to which messages go in which segments, and that can cause the segment sets not to line up, and that can cause allocation errors.
Instead what do you think about keeping exactly-once behavior by basically forcing an entire batch to be processed all-or-nothing? In practice it would mean adding an
addAllmethod to Appenderator, and not doing any persists or publishes in the middle of a batch. To handle the situation where a limit (maxRowsPerSegment or something) could be hit in the middle of a batch, we could either persisting early (before a batch) or late (by allowing to going a little past the limit).I think this might be the cleanest approach.
himanshug commentedon Oct 6, 2017
@gianm sure, having explicit config to enable batching support would be there ( that is what @pjain1 explained in #4373 (comment) ) . so if that config is not explicitly enabled and we got multiple elements in list returned by
InputRowParser.parseBatch(kafka-record-bytes)then task would just loudly fail.I think you mean adding
addAll(..)method toAppenderatorDriveras all rows in the batch might not belong to same segment. In any case, we would have something like that to ensure replica consistency or else would add more comments here.gianm commentedon Oct 6, 2017
Ah yeah I did mean
addAllon AppenderatorDriver.If you do that then I think we don't need to worry about breaking exactly-onceness. As long as AppenderatorDriver can promise to either index the whole batch, or none of it (and I think it can promise that) then exactly-onceness is preserved.
What do you think?
himanshug commentedon Oct 9, 2017
@gianm yes adding
AppenderatorDriver.addAll(..)is fine. However regarding promising indexing of whole batch or none, do you mean to ignore "maxRowsInMemory" so that you index whole batch ? or do you mean adding some batch add capability to IncrementalIndex as well where it either accepts or rejects whole batch?gianm commentedon Oct 9, 2017
I was thinking of the Appenderator having an
addAllmethod too, and it should take care of making sure that the following are true:(1) might require ignoring "maxRowsInMemory" by going slightly under or over it. If it's going over, it will need some changes to IncrementalIndex. If it's going under, it will probably require implementing a "persist without committing" method in Appenderator such that the earlier sink can be persisted with part of a batch, and a new sink can be created to hold the rest of the batch. The "without committing" is important since it would be a violation to persist+commit a partial batch.
(2) might require ignoring "maxRowsPerSegment" by going slightly under or over it.
I think this all can be done in a deterministic way, such that replicas still make the same decisions as each other and we can still enable exactly-once ingestion without needing to track offsets-within-batches.
himanshug commentedon Oct 9, 2017
@gianm I and @pjain1 talked about it a little and it appears that things stay simple by going a bit over for both
maxRowInMemoryandmaxRowsPerSegment. So, let us update IncrementalIndex so that it can optionally go overmax-sizewhen asked. this would maintain exactly-once. (sidenote: we are already going "over" formaxRowsInSegmentlimit in #4815 to maintain consistency of checkpoints across replicas )above is probably simpler than the going "under" and "persist without commit" approach where we might have to keep disk state inconsistent (persisted data having more events than what is stored in commit metadata) for prolonged periods or persisting small sinks depending on implementation.
gianm commentedon Oct 9, 2017
That sounds good to me. I agree that going a little bit over maxRowsInMemory and maxRowsPerSegment will lead to the simplest solution.
JithinThomas commentedon Dec 5, 2017
Hi all, any updates on this feature? Few months ago, I had posted a question regarding a similar feature. Is this feature under development? Is it being planned as part of an upcoming release?
Thanks.
gianm commentedon Dec 5, 2017
Hi @JithinThomas, this patch is in master now so it will be available in the next release. It'll probably be numbered 0.12.0.
JithinThomas commentedon Dec 5, 2017
Hi @gianm , that's great to know! Is there an ETA on when 0.12.0 would be released? Thanks again.
gianm commentedon Dec 5, 2017
@JithinThomas it will be release-branched in about 2 weeks and then released after a qa & stabilization process that typically takes a couple months.
JithinThomas commentedon Dec 6, 2017
Thanks @gianm
cparaskeva commentedon May 6, 2019
Hello! Is this feature finally realized? Any related documentation on this?
gianm commentedon May 6, 2019
Ah ha, it is. It was introduced in #5081. However, at present, you do need to implement a custom extension to get it (none of the builtin parsers will split out messages into multiple rows). I expect we will add that functionality at some point, but have not yet.
cparaskeva commentedon May 7, 2019
@gianm Thank you! You just made my day! I supose you are referring to the so-called druid module ? Is there any code-ready module you have in mind ?
gianm commentedon May 7, 2019
@cparaskeva I am referring to that kind of extension, yeah. Check out this repo for an example extension, which includes a parser: https://github.com/implydata/druid-example-extension