azure extensions: task log kill crashes with "Not Implemented". #9212

Activity

1. lrosenman

   added

   Uncategorized problem report

   on Jan 17, 2020
2. lrosenman commented on Jan 17, 2020

   

   lrosenman

   on Jan 17, 2020

   Last edited by lrosenman

   Author

   More actions

   I'd also love (in the interim), a supported way to shrink the druid_tasks table that currently has >900,000 rows and takes 2.2G of storage, and gets locked (on the PostgreSQL slave I use for backups) sometimes while my backups are running killing the backup.
3. lrosenman commented on Jan 17, 2020

   

   lrosenman

   on Jan 17, 2020

   Last edited by lrosenman

   Author

   More actions

   Looking at s3, google, and azure, NONE of them implement
   killAll()
   killOlderThan(long timestamp)

   the only implementation of those two functions is in hdfs.

   so, how is one supposed to be able to shrink the size of the druid_tasks metadata table, and reduce storage usage in the cloud based tasklogs storage?
4. jihoonson

   added

   Bug

   and removed

   Uncategorized problem report

   on Feb 24, 2020
5. jihoonson commented on Feb 24, 2020

   

   jihoonson

   on Feb 24, 2020

   Contributor

   More actions

   @lrosenman thank you for the report. As you mentioned, those methods are not implemented properly for all deep storage types. I think this is a bug, but AFAIK, there is no workaround except for having an external cron job that periodically cleans up the table.
6. lrosenman commented on Feb 24, 2020

   

   lrosenman

   on Feb 24, 2020

   Author

   More actions

   how safe is deleting rows from the table? And given that the datestamps are varchars and not real timestamps, I'm loathe to try it via SQL.
7. jihoonson commented on Feb 25, 2020

   

   jihoonson

   on Feb 25, 2020

   Contributor

   More actions

   Yeah its schema is not very pretty. I would say you will be safe as long as you delete only the rows of inactive tasks that have the created_date older than (current_timestamp() - druid.indexer.storage.recentlyFinishedThreshold). The actual SQL used in Druid is DELETE FROM %s WHERE created_date < :date_time AND active = false.
8. lrosenman commented on Feb 25, 2020

   

   lrosenman

   on Feb 25, 2020

   Last edited by lrosenman

   Author

   More actions

   druid=# select count(*) from druid_tasks where created_date::timestamptz < now() - '30 days'::interval and not active;
    count
   --------
    949649
   (1 row)
   

   If I change the select count(*) to delete, is this safe?
9. jihoonson commented on Feb 25, 2020

   

   jihoonson

   on Feb 25, 2020

   Contributor

   More actions

   Yes. It should be safe.
10. bharadwajrembar commented on Mar 31, 2020

    

    bharadwajrembar

    on Mar 31, 2020

    More actions

    We are hit with the same for S3 on version 0.17.0. @jihoonson We use the Router for our UI. When we go into the Ingestion tab, the API fires this SQL query:

    SELECT  "task_id", "group_id", "type", "datasource", "created_time", "location", "duration", "error_msg",  
    CASE WHEN "status" = 'RUNNING' THEN "runner_status" ELSE "status" END AS "status", 
     (    CASE WHEN "status" = 'RUNNING' THEN     (CASE "runner_status" WHEN 'RUNNING' THEN 4 WHEN 'PENDING' THEN 3 ELSE 2 END)    ELSE 1    END  ) 
     AS "rank"FROM sys.tasks ORDER BY "rank" DESC, "created_time" DESC
    

    and it takes sometime to load.

    Does this query go to the Overlord? If so, is having a large number of old tasks in the metadata likely to affect performance or even potentially cause OOMs in the Overlord?
11. jihoonson commented on Mar 31, 2020

    

    jihoonson

    on Mar 31, 2020

    Contributor

    More actions

    @bharadwajrembar, yes the SQL internally calls an Overlord API which needs to materialize old tasks in memory which can cause OOM if you have a huge number of tasks. The OOM doesn't usually happen because it doesn't return tasks older than druid.indexer.storage.recentlyFinishedThreshold.

    BTW, this issue was fixed in #9523 which will be included in 0.18.0. I'm closing this issue for now. Please reopen it if you think this issue still exists.
12. jihoonson

    closed this as completedon Mar 31, 2020