You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Handoff should ignore segments that are dropped by drop rules #5868
We have seen problem scenarios like the following:
A user has load rules like: load 30 days, drop forever.
The user loads data with Kafka indexing that is older than 30 days. Generally this is a small proportion of the overall data.
The coordinator disables the segments immediately upon noticing them (since it is beyond the 30 day window).
The Kafka tasks time out during handoff (because the segments are never loaded).
To fix this, handoff should be made to ignore segments that are dropped by drop rules. This should be doable by adding some info about this to the coordinator API that tasks use to monitor handoff.
Related, but different: #5869. IMO #5869 should be done before this one, to prevent users from accidentally getting slightly-future data dropped.
Activity
gianm commentedon Jul 1, 2018
Even though this isn't directly related to Kafka indexing, I gave it a Kafka tag since that's the kind of indexing most likely to be affected.