Repository navigation
[DISCUSS] Dropping the task audit log table from metastore #5859
Description
Druid currently has a table to store task audit logs in metastore. Every taskAction is stored in this table if they need to be audited (https://github.com/druid-io/druid/blob/master/indexing-service/src/main/java/io/druid/indexing/common/actions/TaskAction.java#L47). Basically, all taskActions modifying metastore like SegmentInsertAction currently need to be audited. If a taskAction needs to be audited, it is serialized into a byte array and stored as a blob in metastore.
In Druid, there is a single use case for the task audit log table: an overlord API to get a list of segments generated by a task (http://<OVERLORD_IP>:/druid/indexer/v1/task/{taskId}/segments). This API needs to find all SegmentInsertActions and SegmentTransactionalInsertActions performed by the given task. AFAIK, this API isn't used widely. Outside of Druid, AFAIK, there is no good use case for task audit logs table.
The problem is that this table can cause an issue when inserting a large taskAction. For example, a SegmentInsertAction can be large if there are many segments to be inserted in that action. This can cause some issues. For example, in MySQL, it can cause the insertion error if the data size is larger than max_allowed_packet.
I tried to mitigate this in #5751, but looks not a good idea. Another suggestion in that PR was to store audit logs in deep storage, but it requires quite a lot of changes and introduces new design issues. I don't think we need to make a large effort to fix such an issue that has no good use case.
So, I'm suggesting dropping the task audit log table. Here is my suggestion.
- Disabling task audit logging by default in 0.13. We need to add a configuration for optionally enabling task audit logging. The overlord API should also be deprecated.
- Completely removing the task audit log table and the overlord API in 0.14.
Any thoughts?
Activity
zhangxinyu1 commentedon Jun 28, 2018
The only usage of the audit table in our company is recording the information about who and when change the rules of datasource.
jihoonson commentedon Jun 28, 2018
@zhangxinyu1 thanks for the comment. The audit logs for changing load/drop rules are stored in another table, named
druid_auditby default. The table issued here isdruid_tasklogstable which stores audit logs for changing segments.himanshug commentedon Jul 13, 2018
I don't remember needing to use that table ever since I have used Druid. So, from my side I think it is ok to remove it.
nishantmonu51 commentedon Jul 13, 2018
+1 on dropping task action audit table. Never seen anyone using that.
jihoonson commentedon Sep 26, 2018
#6368 has been merged to master. Will raise a follow-up pr to drop task audit log table for 0.14 later.
KenjiTakahashi commentedon Jan 18, 2019
I see I'm kind of late to the party here ;-). We are using the
/segmentsendpoint internally for validation purposes. I.e. we have long running Tasks with EventReceiverFirehose and the service that pushes data, then checks if appropriate segments/shards got created by the Task.jihoonson commentedon Jan 22, 2019
@KenjiTakahashi Would you please give us more details about how you use ‘/segments’ for validation? I’m not sure what you mean by the validation. How do you know what segments are expected to be generated?
KenjiTakahashi commentedon Mar 31, 2019
Sorry about such late response! I'm not working on this anymore, and it kinda slipped my attention...
Anyway, AFAIR, it works basically by knowing how much data will be sent and how it will be split into shards. After sending the data, it surely uses
/segmentsto check if correct number of shards got created, and perhaps some other details, as well.It works with multiple Tasks in parallel, independently, so it all has to be done at Task level.
/cc @neutrinus?
jihoonson commentedon Apr 10, 2019
@KenjiTakahashi thanks but I still don't understand how it works.. And also I'm wondering how useful your validation is. Is it very important to create the expected number of segments in your use case?
stale commentedon Jun 20, 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
jihoonson commentedon Jun 20, 2019
This issue still relevant.
github-actions commentedon Jul 2, 2023
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
github-actions commentedon Jul 31, 2023
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
Remove task action audit logging and druid_taskLog metadata table (#1…
Remove task action audit logging and druid_taskLog metadata table (ap…
Remove task action audit logging and druid_taskLog metadata table (ap…
Remove task action audit logging and druid_taskLog metadata table (ap…
Remove task action audit logging and druid_taskLog metadata table (ap…