feat(soft-delete): deletion-retention purge of soft-deleted entities

UPDATING.md

@@ -368,6 +368,16 @@ Entity version history (the `version_transaction` / `*_version` shadow tables th
 
 The task ships in the default `CeleryConfig.beat_schedule`; a deployment that overrides `CELERY_CONFIG` without inheriting the default will log a startup warning that the prune task is absent (so it never silently stops running). Retention only prunes whatever history exists — capture itself is gated separately by `ENABLE_VERSIONING_CAPTURE` (ships off).
 
+### Deletion retention (soft-deleted entities are eventually purged)
+
+Soft-deleted dashboards, charts, and datasets are now permanently removed after a retention window (default 30 days; `SOFT_DELETE_RETENTION_DAYS`, `0` disables; settable per workspace at runtime via the `deletion-retention set-window` CLI, which takes precedence). The `deletion_retention.purge_soft_deleted` Celery beat task runs daily and removes each aged-out entity together with its M:N join rows, owned children, datasource permission, and version-history shadow rows. After purge an entity is **unrecoverable** — its detail and `/restore` endpoints return 404 and its version history is gone.
+
+The introducing release **defaults to dry-run** (`SOFT_DELETE_PURGE_DRY_RUN=True`): the task logs `would_purge` counts but deletes nothing, so operators can validate against production before activating real purging by setting it to `False`. Note `would_purge` is an **upper bound** — it counts every entity past the retention window without evaluating deletion blockers, so a real run may purge fewer (entities referenced by report schedules or set as a user's welcome dashboard are blocked and reported separately). The task only acts while the temporary `SOFT_DELETE` rollout flag is on.
+
+Deployments that replace the default `CELERY_CONFIG` must add `superset.tasks.deletion_retention` to the Celery `imports` and schedule the `deletion_retention.purge_soft_deleted` task themselves. The shipped Docker development config includes both entries.
+
+Operators can immediately erase a specific entity for compliance (GDPR) via `superset deletion-retention force-purge --uuid <uuid>`; this applies legacy hard-delete semantics — a live chart referencing a force-purged dataset is left without a datasource until re-pointed (the chart is not modified), and it purges the named entity even when it was never soft-deleted. Every purge writes an immutable, content-free audit record to the new `purge_audit_log` table that survives the entity it names: the **scheduled** purge fails closed (an entity whose audit row cannot be written is skipped and retried next run), while **force-purge** proceeds even if the audit write fails — the operator is present and deletion outranks audit for a compliance erasure.
+
 ### Webhook alerts/reports block private/internal hosts by default
 
 Webhook alert/report dispatch (`WebhookNotification.send`) now validates the target URL's host against the same private/internal-IP block applied to dataset import URLs. If the resolved host is in a loopback, link-local, private (RFC-1918), shared-CGNAT, or multicast range, the webhook is rejected with `NotificationParamException`.

docker/pythonpath_dev/superset_config.py

@@ -84,6 +84,7 @@ class CeleryConfig:
     broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
     imports = (
         "superset.sql_lab",
+        "superset.tasks.deletion_retention",
         "superset.tasks.scheduler",
         "superset.tasks.thumbnails",
         "superset.tasks.cache",
@@ -101,6 +102,13 @@ class CeleryConfig:
             "task": "reports.prune_log",
             "schedule": crontab(minute=10, hour=0),
         },
+        # Gated on the SOFT_DELETE feature flag, which is off by default: the
+        # task is scheduled either way, but purges nothing while the flag is
+        # unset. Enable it in FEATURE_FLAGS below to exercise retention locally.
+        "deletion_retention.purge_soft_deleted": {
+            "task": "deletion_retention.purge_soft_deleted",
+            "schedule": crontab(minute=0, hour=0),
+        },
     }
 
 

superset/cli/deletion_retention.py

@@ -0,0 +1,164 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Operator CLI for deletion retention.
+
+``force-purge`` and ``set-window`` are **operator-gated** — they are
+protected by deployment/shell access (the ``SECURITY.md`` operator trust
+boundary), not Flask-AppBuilder RBAC: a CLI invocation has no ``g.user``, so
+there is no ``403`` to enforce. A future REST route would carry real
+Admin RBAC.
+"""
+
+import logging
+from uuid import UUID
+
+import click
+from flask.cli import with_appcontext
+
+logger = logging.getLogger(__name__)
+
+#: Operator-facing entity names mapped to their table. Kept as table names
+#: rather than model classes so building the ``--type`` choices costs no model
+#: imports at CLI start-up; the class is resolved when the option is used.
+_PURGE_TYPES: dict[str, str] = {
+    "chart": "slices",
+    "dashboard": "dashboards",
+    "dataset": "tables",
+}
+
+
+def _resolve_model(entity_type: str | None) -> type | None:
+    """Map a ``--type`` value to its soft-delete model, or ``None`` for all.
+
+    ``None`` preserves the default search across every registered model, which
+    is what an operator holding only a UUID has to start from.
+    """
+    if entity_type is None:
+        return None
+    from superset.models.helpers import SoftDeleteMixin
+
+    table = _PURGE_TYPES[entity_type.lower()]
+    for model in SoftDeleteMixin._registered_subclasses:  # noqa: SLF001
+        if getattr(model, "__tablename__", None) == table:
+            return model
+    # Unreachable while _PURGE_TYPES tracks the registered models; a mismatch
+    # means a model was renamed or dropped without updating the map.
+    raise click.ClickException(
+        f"No soft-delete model is registered for type {entity_type!r}."
+    )
+
+
+@click.group()
+def deletion_retention() -> None:
+    """Manage purge of soft-deleted entities (operator-gated)."""
+
+
+@deletion_retention.command()
+@with_appcontext
+@click.option(
+    "--days",
+    "-d",
+    required=True,
+    type=int,
+    help="Retention window in days; 0 disables.",
+)
+def set_window(days: int) -> None:
+    """Set the per-deployment retention window (SharedKey, upsert)."""
+    from superset.key_value.shared_entries import upsert_shared_value
+    from superset.key_value.types import SharedKey
+
+    if days < 0:
+        raise click.BadParameter("--days must be >= 0")
+    upsert_shared_value(SharedKey.SOFT_DELETE_RETENTION_DAYS, days)
+    click.echo(
+        f"Soft-delete retention window set to {days} day(s) for this deployment."
+    )
+
+
+@deletion_retention.command()
+@with_appcontext
+def show_window() -> None:
+    """Print the effective retention window (shared value or env fallback)."""
+    from superset.commands.deletion_retention.window import resolve_retention_window
+
+    days = resolve_retention_window()
+    state = "disabled" if days == 0 else f"{days} day(s)"
+    click.echo(f"Effective soft-delete retention window: {state}.")
+
+
+@deletion_retention.command()
+@with_appcontext
+@click.option(
+    "--uuid",
+    "-u",
+    "uuid",
+    required=True,
+    # Validate up front: a malformed value must fail with a clean
+    # BadParameter message, not a StatementError traceback after the
+    # operator has already confirmed an irreversible prompt.
+    type=click.UUID,
+    help="UUID of the entity to purge.",
+)
+@click.option(
+    "--type",
+    "-t",
+    "entity_type",
+    type=click.Choice(sorted(_PURGE_TYPES), case_sensitive=False),
+    default=None,
+    help=(
+        "Restrict the purge to one entity type. UUIDs are unique per table "
+        "but not across them, so a bare UUID can match more than one entity; "
+        "the purge refuses to guess and asks for this option."
+    ),
+)
+@click.confirmation_option(
+    prompt="Force-purge is irreversible — the entity and its version history "
+    "will be permanently removed. Continue?"
+)
+def force_purge(uuid: UUID, entity_type: str | None) -> None:
+    """Immediately and irreversibly purge an entity by UUID (compliance)."""
+    from superset.commands.deletion_retention.force_purge import (
+        AmbiguousPurgeTargetError,
+        ForcePurgeCommand,
+    )
+
+    try:
+        result = ForcePurgeCommand(
+            str(uuid), model_cls=_resolve_model(entity_type)
+        ).run()
+    except AmbiguousPurgeTargetError as ex:
+        # The command refuses to guess between tables. Report that as a clean
+        # operator error naming the way out, not as a traceback -- this lands
+        # after the irreversible confirmation prompt has already been answered.
+        raise click.ClickException(
+            f"{ex} Re-run with --type, e.g. --type {sorted(_PURGE_TYPES)[0]}."
+        ) from ex
+    if not result.get("purged"):
+        if result.get("reason") == "blocked":
+            click.echo(
+                f"Entity uuid={uuid} was not purged because existing deletion "
+                f"rules block it: {result.get('blocked_reason')}."
+            )
+        else:
+            click.echo(f"No entity found for uuid={uuid} (nothing to purge).")
+        return
+    click.echo(
+        f"Purged {result['entity_type']} uuid={uuid}. "
+        f"Dangling charts: {len(result.get('dangling_chart_uuids') or [])}; "
+        f"dashboard_slices removed: {result.get('removed_dashboard_slices', 0)}; "
+        f"version rows removed: {result.get('version_rows_removed', 0)}."
+    )

superset/commands/deletion_retention/__init__.py

@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Deletion retention: purge of soft-deleted entities.
+
+Shared building blocks for the scheduled purge task
+(``superset.tasks.deletion_retention``) and the operator force-purge
+command, so the cascade cannot drift between the two surfaces.
+"""