API: Return 503 when SQLite locks during backfill creation

[lohitkolluri]

Closes: #66726

On SQLite, creating a backfill through the API can fail with a "database is locked" error when the scheduler, dag-processor, and API server all share the same database.

Before this PR, this caused two problems:

• The API returned HTTP 500 with a confusing stack trace
• The backfill row was left in the database with no actual runs, so retrying returned 409 "already running"

After this PR:

• The API returns HTTP 503 with a clear "database is locked, retry or use Postgres/MySQL" message
• The half-created backfill is cleaned up so you can retry safely

Changes

Create route — When the database is locked, return 503 with a clear message instead of letting the error bubble up as a 500.

Dry-run route — Same 503 handling, so previewing a backfill doesn't return a 500 either.

Cleanup of partial state — If the lock happens partway through creating a backfill, the half-created backfill is removed so the user can retry:

• removes the orphan backfill row
• removes any partial dag-run rows (and their TaskInstances via cascade)
• safe to call — if the cleanup itself fails, the original error still surfaces and the route still returns 503

SQLite detection — Teach the lock checker to recognize SQLite's "database is locked" message, since SQLite uses a human-readable string instead of a numeric error code like Postgres or MySQL.

OpenAPI / TypeScript — The 503 response is documented in the API spec and the generated UI client.

Tests

• test_create_backfill_database_locked — checks the 503 mapping works for the create route
• test_create_backfill_dry_run_database_locked — same check for the dry-run route
• test_create_backfill_cleans_up_orphan_on_lock_error — checks the cleanup removes the half-created backfill from a real partial state
• test_create_backfill_cleans_up_after_failed_transaction — checks the cleanup works when SQLAlchemy has already marked the session as needing a rollback
• test_create_backfill_cleanup_removes_partial_dag_runs — checks partial DagRuns, BackfillDagRun rows, and TaskInstances are all removed
• All existing backfill endpoint tests still pass

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 4.8) following the guidelines

[23tae]

Thanks for working on this.

#66726 also reports that the backfill may still be created even though the request fails.

This PR seems to improve the error response from 500 to 503, but it is not clear whether it changes the partial-success behavior. Could we clarify this in the PR description?

[23tae]

Thanks for the update. I left a few comments.

[henry3260]

Thanks for the fix!

[23tae]

I left a couple of comments.

Since implementing a retry loop in the API route introduces edge cases and makes the code complex, how about we just remove it entirely? Simply catching the OperationalError and raising an HTTP 503 on a database lock seems like a much cleaner approach.

[lohitkolluri]

@pierrejeambrun

Reproduced the 500 on macOS — 20 concurrent POST /backfills, 2/2 runs hit the lock + orphan. Repro steps and the full traceback posted on #66726.

Also caught an edge case in the cleanup: SQLAlchemy deactivates the session on flush failure, so the first session.execute() in the previous cleanup raised InvalidRequestError silently. Fixed with a session.rollback() at the top of the helper + a test that reproduces the real failed-session state.

Ready for re-review.

[pierrejeambrun]

Thanks for iterating on this. The direction is reasonable for the SQLite dev case, and I confirmed locally that the production path does return a clean 503 (Database is locked ... use PostgreSQL or MySQL) instead of the 500 — nice.

Two things block it though, both inline: the two ...database_locked tests actually fail as written, and the cleanup path that removes partial runs isn't really tested. A couple of smaller notes too.

[lohitkolluri]

Hey! A few CI checks were failing, so I pushed a patch to address them.

[pierrejeambrun]

LGTM

[boring-cyborg]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[github-actions]

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-3-test