Fix rare case when crawler can get deadlocked

[Pijukatel]

Crawler from this test https://github.com/apify/apify-sdk-python/blob/master/tests/integration/actor/test_crawlers_with_storages.py#L80 can become deadlocked never finishing

[8](https://github.com/apify/apify-sdk-python/actions/runs/21358210249/job/61470616028?pr=747#step:6:369)
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> Status: RUNNING, Message: 
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:23.291Z ACTOR: Pulling container image of build zi7siweXp20fGFNlp from registry.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:25.353Z ACTOR: Creating container.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:25.539Z ACTOR: Starting container.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:25.541Z ACTOR: Running under "LIMITED_PERMISSIONS" permission level.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:27.756Z [apify._configuration] WARN  Actor is running on the Apify platform, `disable_browser_sandbox` was changed to True.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:27.763Z [apify] INFO  Initializing Actor ({"apify_sdk_version": "3.1.1", "apify_client_version": "2.4.0", "crawlee_version": "1.3.0", "python_version": "3.10.19", "os": "linux"})
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.364Z [ParselCrawler] INFO  Current request statistics:
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.366Z ┌───────────────────────────────┬──────┐
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.368Z │ requests_finished             │ 0    │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.370Z │ requests_failed               │ 0    │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.372Z │ retry_histogram               │ [0]  │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.374Z │ request_avg_failed_duration   │ None │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.376Z │ request_avg_finished_duration │ None │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.378Z │ requests_finished_per_minute  │ 0    │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.380Z │ requests_failed_per_minute    │ 0    │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.381Z │ request_total_duration        │ 0s   │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.383Z │ requests_total                │ 0    │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.385Z │ crawler_runtime               │ 0s   │
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.387Z └───────────────────────────────┴──────┘
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.431Z [crawlee._autoscaling.autoscaled_pool] INFO  current_concurrency = 0; desired_concurrency = 10; cpu = 0; mem = 0; event_loop = 0.0; client_info = 0.0
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:28.436Z [ParselCrawler] INFO  Crawled 0/1 pages, 0 failed requests, desired concurrency 10.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:29.737Z [ParselCrawler] WARN  Retrying request to http://localhost:8080/ due to: Some error. File "/usr/src/app/src/main.py", line 26, in default_handler,     raise RuntimeError('Some error')
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:29.829Z [ParselCrawler] WARN  Retrying request to http://localhost:8080/ due to: Some error. File "/usr/src/app/src/main.py", line 26, in default_handler,     raise RuntimeError('Some error')
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:29.879Z [ParselCrawler] WARN  Retrying request to http://localhost:8080/ due to: Some error. File "/usr/src/app/src/main.py", line 26, in default_handler,     raise RuntimeError('Some error')
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.056Z [ParselCrawler] ERROR Request to http://localhost:8080/ failed and reached maximum retries
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.058Z  Traceback (most recent call last):
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.060Z   File "/usr/local/lib/python3.10/site-packages/crawlee/crawlers/_basic/_context_pipeline.py", line 114, in __call__
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.063Z     await final_context_consumer(cast('TCrawlingContext', crawling_context))
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.064Z   File "/usr/local/lib/python3.10/site-packages/crawlee/_utils/wait.py", line 37, in wait_for
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.066Z     return await asyncio.wait_for(operation(), timeout.total_seconds())
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.068Z   File "/usr/local/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.070Z     return fut.result()
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.072Z   File "/usr/local/lib/python3.10/site-packages/crawlee/router.py", line 108, in __call__
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.074Z     return await user_defined_handler(context)
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.076Z   File "/usr/src/app/src/main.py", line 26, in default_handler
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.077Z     raise RuntimeError('Some error')
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:30.079Z RuntimeError: Some error
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:38.599Z [ParselCrawler] INFO  Crawled 0/1 pages, 1 failed requests, desired concurrency 10.
[apify.python-sdk-tests-crawler-max-retries-generated-1U0Ejtq0 runId:VwSGval1HMpakuQEE] -> 2026-01-26T13:00:48.655Z [ParselCrawler] INFO  Crawled 0/1 pages, 1 failed requests, desired concurrency 10.

... Crawler not finishing and not doing any requests forever. 

[janbuchar]

Do you have any idea what the circumstances are that lead to a deadlock?

[Pijukatel]

Do you have any idea what the circumstances are that lead to a deadlock?

My wild guess is something in AutoscaledPool or maybe request_manager.is_finished? The autoscaled pool keeps running, it does not fetch any new request, so it was probably correctly marked as handled

[Pijukatel]

class RequestQueue:
...
    async def is_finished(self) -> bool:
        if self._add_requests_tasks:
            logger.debug('Background add requests tasks are still in progress.')
            return False

        if await self.is_empty():
            logger.debug('The request queue is empty.')
            return True

        return False

This behavior could happen if is_empty==True (so no new requests are scheduled by the AutoscaledPool, but due to something lingering in _add_requests_tasks the is_finished returns False), so the pool keeps running, not scheduling new tasks, but refusing to finish. But I have no clue how anything could still be left in _add_requests_tasks...

[MrFlynn]

I've also seen this happen a bunch on the Playwright crawler. I have plenty of example logs, though there's nothing particularly useful to be found in them but I can provide them if they'd be useful. I've also looked at request_queue directory and there are usually one or two URLs that have "handled_at": null. Those are usually (though not always) associated with earlier failures on the same URL that result in them being enqueued again.

Since I know how long my workload will take I added a bit of code to call the abort method on the AutoscalePool after a set timeout. Calling the stop method on the crawler doesn't work because it still thinks some page is being processed.

[Pijukatel]

I think that the most probable scenario is, that API call to mark request as handled fails (some random glitch). This puts the queue in a weird state, where the request is not properly marked as handled, and it prevents the RQ from finishing.

We have no handling around this scenario, and even the log that would indicate this scenario is on debug level, so by default not visible.

[Pijukatel]

@MrFlynn, I guess that you have been using the FileSystemStorageClient? I think I have a fix for the same scenario when using ApifyStorageClient. I will have to look into the other clients as well to see how to fix this.

[Pijukatel]

I am having difficulties reproducing this with FileSystemStorageClient. Would you have a reproducible example @MrFlynn?

I tried to use this. It ran for a while (requests_finished=1464 and requests_failed=1491) and I saw no issues. Crawler finished as expected:

from apify import Actor
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.http_clients import ImpitHttpClient

async def main() -> None:
    async with Actor:
        crawler = PlaywrightCrawler()

        @crawler.router.default_handler
        async def default_handler(context: PlaywrightCrawlingContext) -> None:
            if (hash(context.request.url) % 10) < 5:
                # Fail around half of the pages
                raise RuntimeError('Simulated error specific for this url')

            await context.enqueue_links()

        await crawler.run(
            [
                'https://crawlee.dev',
            ]
        )

[MrFlynn]

The project I've been running where I've noticed it can get deadlocked is this one. Occasionally the model-metrics.py script it will hang forever. This script is run once per hour by Github Actions and the request queue gets deadlocked maybe 1-2 per day. I've also seen this happen locally, although I haven't run it very many times on my machine since it's mostly a background job.

Here's an example log file from yesterday where it got stuck (see the long stream of "Crawled 312/313 pages, 0 failed requests, desired concurrency 6." toward the end before the custom cancellation logic kicked in).

[Pijukatel]

Thanks for the information. I will look into it and try to figure it out.

[B4nan]

I think that the most probable scenario is, that API call to mark request as handled fails (some random glitch). This puts the queue in a weird state, where the request is not properly marked as handled, and it prevents the RQ from finishing.

FYI in the JS version we retry those calls with a timeout. We do this with many API calls that could lead to undefined state of the crawler. We saw weird failures in the API calls and they even resulted in stucking the crawler (at least I thought so), since the client tried expo backoff with 9 retries. I know you did some timeout improvements on the client level in the past, but we should likely still handle this on crawlee level too.

https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts#L1625-L1631