Skip to content

SNS: Server unable to handle too much load — AWS SDK connection pool acquire timeout #10108

Open

Description

@mariuszbluestonepim

Under a heavy volume of SNS Publish calls, the AWS SDK fails with:

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.

When the publish rate is high, the moto server can't process messages fast enough to keep up, so the client's connection pool stays saturated and acquisition times out. It appears to be a single-process throughput limit.

Note: the SNS topic has an SQS subscription (SNS → SQS fan-out), so each publish also performs delivery to the subscribed queue inline within the publish request.

The same client configuration works without errors against LocalStack, which points to a server-side throughput difference rather than a client issue.

Environment

moto 5.2.2, server mode (motoserver/moto Docker image)
AWS SDK for Java v2 (Netty async client, default pool settings)

Expected: moto server sustains a heavy SNS publish load (including SNS → SQS delivery) without exhausting the client connection pool, comparably to LocalStack.

Activity

  1. bblommers commented on Jul 15, 2026

    @bblommers
    Collaborator

    Hi @mariuszbluestonepim! Can you share a small reproducible sample?

    And second - what do you consider a heavy volume? How many RPS/messages does one need to send before this becomes a problem?

  2. mariuszbluestonepim commented on Jul 20, 2026

    @mariuszbluestonepim
    Author

    Hi. Reproducible sample attached, plus the RPS numbers you asked for.

    Setup:
    build.gradle

    plugins {
        id 'java'
    }
    
    group = 'org.example'
    version = '1.0-SNAPSHOT'
    
    java {
        toolchain {
            languageVersion = JavaLanguageVersion.of(25)
        }
    }
    
    repositories {
        mavenCentral()
    }
    
    dependencies {
        implementation(platform("software.amazon.awssdk:bom:2.46.21"))
        implementation("software.amazon.awssdk:s3")
        implementation("software.amazon.awssdk:s3-transfer-manager")
        implementation("software.amazon.awssdk.crt:aws-crt:0.47.2")
        implementation("software.amazon.awssdk:sns")
        implementation("software.amazon.awssdk:sqs")
    }
    
    test {
        useJUnitPlatform()
    }
    
    

    docker-compose.yml

    services:
      motoserver:
        image: motoserver/moto:5.2.2
        environment:
          - MOTO_PORT=4466
          - MOTO_ACCOUNT_ID=000000000000
          - AWS_DEFAULT_REGION=eu-central-1
          - AWS_ACCESS_KEY_ID=dev
          - AWS_SECRET_ACCESS_KEY=dev
        ports:
          - "127.0.0.1:4466:4466"
        deploy:
          resources:
            limits:
              memory: 1g
      localstack:
        image: localstack/localstack:4.14.0
        environment:
          - LOCALSTACK_HOST=localstack
          - SQS_ENDPOINT_STRATEGY=path
          - AWS_DEFAULT_REGION=eu-central-1
          - AWS_ACCESS_KEY_ID=dev
          - AWS_SECRET_ACCESS_KEY=dev
          - AWS_DEFAULT_REGION=eu-central-1
        ports:
          - "127.0.0.1:4566:4566"
        deploy:
          resources:
            limits:
              memory: 1g
    

    Sample application:

    Test3.java

    Test results:
    motoserver:

    Endpoint http://localhost:4466 | ramp 100..1200 RPS (+100 every 5s)
    
    rps       attempted       ok   failed  inflight     ok/s
    100             500      500        0         0      100
    200            1001     1000        0         1      200
    300            1501     1342        0       160      268
    400            2001      671        0      1489      134
    500            2502      444        0      3546       88
    600            3002      355      642      5551       71
    700            3503      294     2208      6551       58
    800            4004      295     2711      7549       59
    900            4502      254     3250      8547       50
    1000           5000      221     3776      9550       44
    1100           5501      228     4780     10042       45
    1200           6004      210     5791     10045       42
    
    ===== SUMMARY =====
    endpoint            : http://localhost:4466
    first failing RPS   : 600
    total attempted     : 39021
    total ok            : 5871
    total failed        : 29003
      of which ACQUIRE-TIMEOUT: 21697
    received by SQS     : 3013
    failure breakdown:
       15504  TimeoutException: Acquire operation took longer than configured maximum time
        6193  TimeoutException: Acquire operation took longer than 10000 milliseconds.
        5786  IOException: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider ...
        1506  IllegalStateException: Too many outstanding acquire operations
          14  SocketException: Connection reset
           1  SdkInterruptedException: null
    

    localstack:

    Endpoint http://localhost:4566 | ramp 100..1200 RPS (+100 every 5s)
    
    rps       attempted       ok   failed  inflight     ok/s
    100             500      500        0         0      100
    200            1001      998        1         2      199
    300            1500     1493        0         9      298
    400            2002     1993        0        18      398
    500            2501     2495        0        24      499
    600            3002     3005        0        20      601
    700            3501     3377        0       144      675
    800            4001     4030        0       115      806
    900            4501     4001        0       615      800
    1000           5003     3988        0      1629      797
    1100           5501     4162        0      2968      832
    1200           6005     4025        0      4948      805
    
    ===== SUMMARY =====
    endpoint            : http://localhost:4566
    first failing RPS   : 200
    total attempted     : 39018
    total ok            : 36425
    total failed        : 1
      of which ACQUIRE-TIMEOUT: 0
    received by SQS     : 10241
    failure breakdown:
           1  SdkInterruptedException: null
    
    
  3. Mide69 commented on Jul 26, 2026

    @Mide69
    Contributor

    Can this be assigned to me if it is available?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions