Skip to content

S3: avoid full-source reads for UploadPartCopy ranges - #10565

Open
tonypottera24 wants to merge 2 commits into
getmoto:masterfrom
tonypottera24:fix/s3-ranged-copy-read
Open

tonypottera24 wants to merge 2 commits into
getmoto:masterfrom
tonypottera24:fix/s3-ranged-copy-read

Conversation

@tonypottera24

Copy link
Copy Markdown

UploadPartCopy reads the complete source object before slicing x-amz-copy-source-range. Copying many small parts from a large source therefore repeatedly materializes the full source.

Reuse FakeKey.get_value(start, end) from #10564 to read only the requested source bytes under the existing per-key lock. Source-version resolution and multipart part creation stay with their existing owners. A copy without a source range reads the complete source.

Depends on #10564. This branch contains its GET commit followed by the Copy commit. The Copy-specific change is 24acb5d6e; compare only this follow-up. It needs rebasing after the parent lands.

The HTTP regression test covers memory-backed and disk-backed sources, both unversioned and with a selected non-current version. It checks that copying 16 bytes reads only 16 bytes, completes the multipart upload and verifies the resulting object. All four variants fail on the parent with read(8192) instead of read(16).

Validation on Python 3.14.4, boto3/botocore 1.44.3:

  • S3 decorator suite with Docker Lambda cases deselected: 571 passed, 5 skipped, 2 xfailed.
  • S3 ServerMode suite with the same selection: 486 passed, 91 skipped, 4 deselected, 2 xfailed.
  • Repository-pinned Ruff 0.14.10 check/format and mypy moto/s3 passed.

A local decorator-mode comparison uses one disk-backed 128 MiB source and 32 source-range copies of 1 MiB each. Source upload/initiation and the subsequent byte verification/abort are outside the timed loop. Each sample verifies every copied part and aborts its multipart upload.

Source Source bytes read per sample Median copy-loop time
Parent GET change fd4fa84f30476da7916ca0bdee357b99dd781401 4 GiB 2.0949 s
This Copy change 24acb5d6e 32 MiB 0.08652 s

Parent wall-time samples: 2.035425, 2.107370, 2.094906 s. Changed samples: 0.086524, 0.091552, 0.084613 s. Both perform 32 source reads; source read amplification drops from 128x to 1x. These are local SDK/mock measurements, not AWS or network throughput.

Reproduce the measurement

Run this script in the same environment on the parent and Copy commits:

import json
import platform
import time
from unittest.mock import patch

import boto3
from moto import mock_aws
from moto.s3.models import s3_backends

with mock_aws():
    client = boto3.client("s3", region_name="us-east-1")
    client.create_bucket(Bucket="range-benchmark")
    data = bytes(range(256)) * (512 * 1024)
    client.put_object(Bucket="range-benchmark", Key="source", Body=data)
    backend = s3_backends["123456789012"]["aws"]
    key = backend.get_object("range-benchmark", "source")
    native_read = key._value_buffer.read
    results = []
    for repeat in range(3):
        upload_id = client.create_multipart_upload(
            Bucket="range-benchmark", Key="target"
        )["UploadId"]
        counts = {"calls": 0, "bytes": 0}

        def read(*args, **kwargs):
            result = native_read(*args, **kwargs)
            counts["calls"] += 1
            counts["bytes"] += len(result)
            return result

        with patch.object(key._value_buffer, "read", side_effect=read):
            start = time.perf_counter()
            cpu_start = time.process_time()
            for index in range(32):
                begin = index * 1024 * 1024 + 7
                end = begin + 1024 * 1024
                client.upload_part_copy(
                    Bucket="range-benchmark",
                    Key="target",
                    UploadId=upload_id,
                    PartNumber=index + 1,
                    CopySource="range-benchmark/source",
                    CopySourceRange=f"bytes={begin}-{end - 1}",
                )
            results.append(
                {
                    "seconds": time.perf_counter() - start,
                    "cpu_seconds": time.process_time() - cpu_start,
                    **counts,
                }
            )
        multipart = backend.get_bucket("range-benchmark").multiparts[upload_id]
        for index in range(32):
            begin = index * 1024 * 1024 + 7
            assert multipart.parts[index + 1].value == data[begin : begin + 1024 * 1024]
        client.abort_multipart_upload(
            Bucket="range-benchmark", Key="target", UploadId=upload_id
        )
    print(
        json.dumps(
            {
                "python": platform.python_version(),
                "object_bytes": len(data),
                "requests": 32,
                "requested_bytes": 32 * 1024 * 1024,
                "samples": results,
            },
            indent=2,
        )
    )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant