Skip to content

feat: support permanent deletion of monthly workflow archives - #44052

Merged
hjlarry merged 8 commits into
langgenius:mainfrom
hjlarry:feat/delete-workflow-log-archives
Oct 14, 2026
Merged

hjlarry merged 8 commits into
langgenius:mainfrom
hjlarry:feat/delete-workflow-log-archives

Conversation

@hjlarry

@hjlarry hjlarry commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #44051.

Add owner-only permanent deletion of a selected month in Workflow Log Archives, with an irreversible-action confirmation, progress, retry, and lifecycle audit logs. Deletion durably blocks downloads, uses the existing workflow_archive queue to remove archive objects and prepared ZIPs, and removes catalog records after successful cleanup. Live workflow-run data is outside this action.

Keep retention cleanup resumable when a saved catalog cursor has been deleted: the cursor remains an exclusive ID boundary and the query retains its month, tenant, and shard filters.

Checklist

  • Linked the associated feature issue.
  • Added or updated tests for the affected behavior.
  • Updated the generated Console API contracts and endpoint documentation.
  • Ran full make lint && make type-check and frontend vp staged (focused checks above).

@hjlarry
hjlarry requested a review from a team October 14, 2026 06:51
@ghfind-review ghfind-review Bot added the review: top ghfind author score; see https://ghfind.com label Oct 14, 2026
@github-actions github-actions Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. web This relates to changes on the web. labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Pyrefly Diff

base → PR
--- /tmp/pyrefly_base.txt	2026-10-14 07:37:02.645675863 +0000
+++ /tmp/pyrefly_pr.txt	2026-10-14 07:36:54.385560362 +0000
@@ -5741,9 +5741,9 @@
 ERROR No matching overload found for function `typing.MutableMapping.update` called with arguments: (dict[str, object]) [no-matching-overload]
   --> tests/unit_tests/services/rag_pipeline/test_rag_pipeline_transform_service.py:46:18
 ERROR Argument `dict[str, dict[str, int | str]]` is not assignable to parameter `tables` with type `dict[str, ArchiveBundleTableManifestEntry]` in function `services.retention.workflow_run.archive_bundle_index.ArchiveBundleManifest.__init__` [bad-argument-type]
-   --> tests/unit_tests/services/retention/workflow_run/test_archive_download_preparation.py:153:16
+   --> tests/unit_tests/services/retention/workflow_run/test_archive_download_preparation.py:163:16
 ERROR Cannot index into `dict[str, bytes]` [bad-index]
-   --> tests/unit_tests/services/retention/workflow_run/test_archive_download_preparation.py:243:52
+   --> tests/unit_tests/services/retention/workflow_run/test_archive_download_preparation.py:343:52
 ERROR Argument `FakeRedis` is not assignable to parameter `redis` with type `RedisClientWrapper` in function `services.retention.workflow_run.archive_download_task_cache.WorkflowRunArchiveDownloadTaskCache.__init__` [bad-argument-type]
    --> tests/unit_tests/services/retention/workflow_run/test_archive_download_task_cache.py:124:55
 ERROR Argument `FakeRedis` is not assignable to parameter `redis` with type `RedisClientWrapper` in function `services.retention.workflow_run.archive_download_task_cache.WorkflowRunArchiveDownloadTaskCache.__init__` [bad-argument-type]
@@ -5753,9 +5753,9 @@
 ERROR Argument `FakeRedis` is not assignable to parameter `redis` with type `RedisClientWrapper` in function `services.retention.workflow_run.archive_download_task_cache.WorkflowRunArchiveDownloadTaskCache.__init__` [bad-argument-type]
    --> tests/unit_tests/services/retention/workflow_run/test_archive_download_task_cache.py:188:55
 ERROR Argument `(_task: WorkflowRunArchiveDownloadTask) -> None` is not assignable to parameter `dispatcher` with type `WorkflowRunArchiveDownloadTaskDispatcher | None` in function `_service` [bad-argument-type]
-   --> tests/unit_tests/services/retention/workflow_run/test_archive_log_service.py:303:69
+   --> tests/unit_tests/services/retention/workflow_run/test_archive_log_service.py:340:69
 ERROR Argument `(_storage_key: str, *, expires_in: int, filename: str) -> str` is not assignable to parameter `sign_download_url` with type `WorkflowRunArchiveDownloadUrlSigner` in function `services.retention.workflow_run.archive_log_service.WorkflowRunArchiveService.__init__` [bad-argument-type]
-   --> tests/unit_tests/services/retention/workflow_run/test_archive_log_service.py:368:27
+   --> tests/unit_tests/services/retention/workflow_run/test_archive_log_service.py:405:27
 ERROR Argument `list[FromClause]` is not assignable to parameter `tables` with type `Sequence[Table] | None` in function `sqlalchemy.sql.schema.MetaData.create_all` [bad-argument-type]
   --> tests/unit_tests/services/retention/workflow_run/test_restore_archived_workflow_run.py:58:16
 ERROR Argument `str` is not assignable to parameter `run_status` with type `SQLCoreOperations[WorkflowExecutionStatus] | WorkflowExecutionStatus` in function `models.workflow.WorkflowArchiveLog.__init__` [bad-argument-type]

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 74.62% 74.67% +0.05%
Strict coverage 74.28% 74.33% +0.05%
Typed symbols 60,703 60,881 +178
Untyped symbols 20,744 20,749 +5
Modules 3756 3763 +7

@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.29126% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.94%. Comparing base (ba5532a) to head (bb96d9b).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...ervices/retention/workflow_run/archive_deletion.py 80.23% 9 Missing and 8 partials ⚠️
web/service/console/query-policies.ts 66.66% 7 Missing ⚠️
...pi/repositories/workflow_run_archive_repository.py 85.18% 2 Missing and 2 partials ⚠️
...count-setting/workflow-log-archives-page/index.tsx 95.58% 3 Missing ⚠️
...etention/workflow_run/archive_download_adapters.py 33.33% 2 Missing ⚠️
...ntion/workflow_run/archive_download_preparation.py 94.73% 1 Missing and 1 partial ⚠️
...vices/retention/workflow_run/archive_month_lock.py 95.83% 2 Missing ⚠️
api/tasks/workflow_run_archive_deletion_tasks.py 85.71% 2 Missing ⚠️
...tention/workflow_run/bundle_archive_maintenance.py 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #44052      +/-   ##
==========================================
+ Coverage   88.92%   88.94%   +0.01%     
==========================================
  Files        5332     5336       +4     
  Lines      329404   329761     +357     
  Branches    65910    65980      +70     
==========================================
+ Hits       292923   293303     +380     
+ Misses      31282    31258      -24     
- Partials     5199     5200       +1     
Flag Coverage Δ
api 88.50% <90.40%> (+0.03%) ⬆️
cli 90.56% <ø> (ø)
dify-ui 92.25% <ø> (ø)
web 89.34% <88.76%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

MRZHUH
MRZHUH previously approved these changes Oct 14, 2026
@hjlarry
hjlarry added this pull request to the merge queue Oct 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 14, 2026
@hjlarry
hjlarry added this pull request to the merge queue Oct 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 14, 2026

@CryoThrust CryoThrust left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the deletion/download failure boundaries on cf8935b98c and found two recovery gaps below.

Eight local method-boundary controls/observations completed: ordinary deletion, no-request protection, explicit retry after partial failure, real Celery success/retry handling, and delayed uploads after lease loss. The probe executes the unchanged deletion/preparation/storage methods and task decorator with Celery 5.6.3 and Pydantic 2.12.5; catalog, storage client, cache and lease ownership are controlled collaborators. ZIP conversion is stubbed. This is not the full API suite or live DB/Redis/S3 validation.

Comment thread api/tasks/workflow_run_archive_deletion_tasks.py
Comment thread api/services/retention/workflow_run/archive_download_preparation.py

@MRZHUH MRZHUH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hjlarry
hjlarry disabled auto-merge October 14, 2026 07:46
@hjlarry
hjlarry added this pull request to the merge queue Oct 14, 2026
Merged via the queue into langgenius:main with commit c11b36c Oct 14, 2026
53 checks passed
@hjlarry
hjlarry deleted the feat/delete-workflow-log-archives branch October 14, 2026 07:51
hjlarry added a commit that referenced this pull request Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review: top ghfind author score; see https://ghfind.com size:XXL This PR changes 1000+ lines, ignoring generated files. web This relates to changes on the web.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support permanent deletion of monthly workflow log archives

3 participants