Large uploads to S3 keep failing. Here is the reliable fix

You try to push a 12 GB export to S3, wait twenty minutes, and watch it fail at 90% because a Wi-Fi blip reset the whole transfer. Or the SDK throws EntityTooLarge before it even starts. Both have the same root cause: you are treating a large object like a small one. The fix is multipart upload, and it is built for exactly this.

Multipart upload is the standard way to move anything large into S3 reliably. It is not exotic, but the parts that bite teams are not in the happy-path tutorial: the IAM permissions, the cleanup of failed uploads, and knowing when to reach for it at all. This walks through all three for both the API and the AWS CLI.

Why a single PUT breaks on large files

A normal S3 PUT has two hard limits worth knowing.

• 5 GB ceiling. A single PutObject call cannot exceed 5 GB. Above that, the API rejects it outright.
• All or nothing. If the connection drops at 90%, there is no resume. The next attempt starts from byte zero.

On a stable data-centre link with a small file, neither matters. On a real-world connection with a multi-gigabyte backup, video master, or data export, both will eventually cost you a failed transfer and a wasted half hour. That is the problem multipart upload was designed to remove.

How multipart upload actually works

The mechanism is three steps, and every SDK and the CLI wrap them for you.

1. Initiate. You call CreateMultipartUpload and get back an UploadId that ties all the parts together.
2. Upload parts. The file is split into parts (5 MB minimum each, except the last). Each part uploads independently, in parallel, and returns an ETag. A failed part retries on its own without touching the others.
3. Complete. You call CompleteMultipartUpload with the list of part numbers and ETags, and S3 stitches them into one object.

The parallel, per-part retry is the whole point. A dropped connection costs you one 5 MB part, not the entire 12 GB. For high-throughput pipelines this is also faster, because parts move concurrently instead of one long serial stream.

The easy path: the AWS CLI

You rarely need to orchestrate the three steps by hand. The high-level CLI does multipart automatically once a file crosses the threshold:

aws s3 cp ./big-export.tar.gz s3://your-bucket/exports/

For anything over the multipart threshold (8 MB by default), the CLI splits, parallelises, and reassembles for you. You can tune the part size and concurrency with aws configure set default.s3.multipart_chunksize and max_concurrent_requests when you are pushing very large objects or saturating a fast link.

The IAM permissions teams miss

This is where most multipart setups fail with an AccessDenied that the happy-path tutorial never mentions. Multipart upload needs more than s3:PutObject. A working policy for the upload role includes:

• s3:PutObject to write the parts and the final object
• s3:ListMultipartUploadParts to track parts during the upload
• s3:AbortMultipartUpload to clean up a failed attempt
• s3:ListBucketMultipartUploads on the bucket to enumerate in-progress uploads

Grant those scoped to the specific bucket and prefix, not the whole account. Over-broad S3 access is one of the most common findings when we run a security review, and locking transfer roles to a single prefix is a quick, high-value fix. We cover the wider pattern in our piece on locking down AWS data services against leaks, and our AWS consulting team hardens these policies as standard.

The hidden cost: orphaned parts

Here is the trap nobody warns you about. When a multipart upload fails and is never completed or aborted, the parts that did upload stay in the bucket. They do not show up in a normal object listing, but they sit in storage and you are billed for them every month. On a busy ingestion pipeline with frequent failures, this quietly grows into a real line on the bill.

The fix is one bucket lifecycle rule: AbortIncompleteMultipartUpload after, say, 7 days. It sweeps up any abandoned parts automatically. Auditing for these orphaned uploads is one of the first things we check when a client asks why their S3 spend crept up. If your bill has drifted for no obvious reason, our guide on tracing an AWS billing spike walks through where to look.

When to reach for multipart, and when not to

The honest trade-off: multipart adds overhead per part, so it is the wrong tool for thousands of tiny files. Use it when the object is large or the link is unreliable, and use plain PutObject for small, quick writes. As a working rule, let the SDK or CLI default threshold decide, around 8 to 16 MB, and only tune it when you are moving very large objects or pushing a high-bandwidth link to its limit. For a managed pipeline, our managed cloud services team sets the thresholds, concurrency, and cleanup rules to match your actual traffic rather than the defaults.

FAQ

What is the minimum part size for an S3 multipart upload?

5 MB for every part except the last, which can be smaller. You can have up to 10,000 parts per object, which sets the practical ceiling on a single multipart object at 5 TB.

Do I have to write the three steps myself?

No. The AWS CLI and the SDKs perform multipart automatically once a file passes a size threshold. You only call the low-level CreateMultipartUpload, UploadPart, and CompleteMultipartUpload yourself for custom streaming or resumable logic.

Why am I getting AccessDenied on a multipart upload?

The IAM policy almost certainly lacks the multipart actions. Add s3:ListMultipartUploadParts, s3:AbortMultipartUpload, and s3:ListBucketMultipartUploads alongside s3:PutObject, scoped to the bucket and prefix.

How do I stop paying for failed uploads?

Add a bucket lifecycle rule that aborts incomplete multipart uploads after a set number of days. Orphaned parts from failed uploads do not appear in a normal object listing but still accrue storage charges until they are cleaned up.

Large file transfer is one of those problems that looks trivial until it fails in production at 90%. Multipart upload, the right IAM policy, and one lifecycle rule turn it into a non-event, which is exactly where infrastructure should sit.