How to Upload Large Files to S3 with AWS CLI Multipart Upload: Complete Guide
By Braincuber Team
Published on March 6, 2026
We had a client last month trying to upload a 47GB database backup to S3 using a single PUT request. It failed 3 times at the 90% mark because their office internet hiccupped. Each restart meant re-uploading 42GB of already-transferred data. 14 hours wasted. The fix took 20 minutes to set up: S3 multipart upload via the AWS CLI. The file gets split into manageable chunks, each chunk uploads independently, failed chunks retry without restarting the whole transfer, and S3 reassembles everything on the other end. A single PUT maxes out at 5GB anyway — anything larger requires multipart. This step by step beginner guide walks you through every command, every gotcha, and the ETag tracking step that most tutorials gloss over.
What You'll Learn:
- How multipart uploads work and why single PUT operations fail above 5GB
- Step by step guide: install AWS CLI, configure IAM credentials, and verify setup
- How to split files into chunks using the
splitcommand - The 6-step multipart upload process: split, create bucket, initiate, upload parts, compile ETags, complete
- How to verify uploaded parts and build the JSON manifest S3 needs for reassembly
How Multipart Uploads Work Under the Hood
In a multipart upload, S3 breaks the transfer into independent chunks that upload separately. Each chunk gets a distinct identifier and sequence number. When all chunks finish uploading, S3 reassembles them into the complete object. If chunk 4 of 7 fails, you retry only chunk 4 — not the entire file.
Chunk-Level Retry
If a segment fails during transfer, only that segment retries. With a 160GB file split into 1GB chunks, a failure on chunk 47 means you re-upload 1GB, not 47GB. On unreliable networks or cross-region transfers, this saves hours of bandwidth.
Pause and Resume
Multipart uploads support suspension and resumption. Start an upload at 3 PM, pause at 6 PM, resume the next morning. Each uploaded part is tracked by its UploadId and ETag. No progress is lost between sessions.
Parallel Upload Capability
Chunks can theoretically upload in parallel, saturating your available bandwidth. A 10GB file split into 100MB chunks can push multiple parts simultaneously, reducing total transfer time compared to a sequential single-stream upload.
Exceeds the 5GB Single-PUT Limit
S3's single PUT operation maxes out at 5GB per object. Database backups, video archives, ML training datasets, and large log bundles routinely exceed this. Multipart upload is the only way to get objects larger than 5GB into S3.
Prerequisites: Install AWS CLI and Configure Credentials
Before starting the multipart upload process, you need the AWS CLI installed and your IAM credentials configured. If you already have this set up, skip to Step 1.
Install the AWS CLI
Download the CLI from the official AWS CLI installation documentation for your OS (Windows, Linux, or macOS). After installation, verify it's working by running aws --version in your terminal. You should see the CLI version number in the output. Then configure your credentials by running aws configure and providing your AWS Access Key ID, Secret Access Key, default region (e.g., us-east-1), and output format (leave blank for default). To get these credentials, create an IAM user in the AWS Console with AdministratorAccess policy attached, then generate an access key under Security credentials.
# Verify CLI installation
aws --version
# Configure credentials (interactive prompts)
aws configure
# AWS Access Key ID: [paste your key]
# AWS Secret Access Key: [paste your secret]
# Default region name: us-east-1
# Default output format: [leave blank]
Never Use Root Account Credentials
Create a dedicated IAM user for CLI operations. Do not use your AWS root account access keys. If those keys leak (committed to Git, pasted in a Slack channel, saved in a plain-text file), an attacker has full control of your entire AWS account — every service, every region, every dollar. We've seen a single leaked root key result in a $23,000 EC2 mining bill in 72 hours. Use IAM. Rotate keys every 90 days. Enable MFA on the root account.
The 6-Step Multipart Upload Process
Follow these in order. Each step depends on output from the previous one. We'll use a 188MB video file as the example, but this works identically for 5GB, 50GB, or 500GB objects.
Split the Object Into Chunks
Navigate to the folder containing your file using cd. Then use the split command to divide the file into chunks. For a 188MB video file split into ~30MB parts: split -b 31457280 videoplayback.mp4. Run ls -lh to verify — you'll see new files named xaa, xab, xac, etc. Each represents one part of your original file. The chunk size is configurable — use larger chunks (100MB–500MB) for bigger files on fast connections, smaller chunks (25MB–50MB) for unreliable networks where you want finer retry granularity.
# Navigate to file location
cd /path/to/your/files
# Split file into ~30MB chunks (bytes)
split -b 31457280 videoplayback.mp4
# Verify the chunks
ls -lh
# Output: xaa (30M), xab (30M), xac (30M), ... xag (7.9M)
Create an S3 Bucket
If you don't already have an S3 bucket, create one via the AWS Console or CLI. Bucket names must be globally unique across all AWS accounts. Choose a region close to your upload source to minimize latency. For production use, enable versioning and server-side encryption (AES-256 is free, KMS adds $1/month per key). If you already have a bucket, skip to Step 3.
Initiate the Multipart Upload
Run aws s3api create-multipart-upload with your bucket name and the target key (filename in S3). This returns a JSON response containing the UploadId — a unique identifier for this upload session. Save this UploadId immediately. You need it for every subsequent step. Lose it and you'll have to abort the upload and start over. We copy it to a local text file in the same directory as the split files.
# Initiate multipart upload
aws s3api create-multipart-upload \
--bucket my-upload-bucket \
--key videoplayback.mp4
# Response (save the UploadId!):
# {
# "Bucket": "my-upload-bucket",
# "Key": "videoplayback.mp4",
# "UploadId": "exampleTUVGeKAk3Ob7qMynRKqe..."
# }
Upload Each Part to S3
Use aws s3api upload-part for each chunk. Specify the bucket, key, part number (starting at 1), body (the chunk filename like xaa), and your saved UploadId. Each successful upload returns an ETag value — save every single one. You need all ETags for Step 5. Increment the part number and body filename for each subsequent chunk: xaa with --part-number 1, xab with --part-number 2, and so on. After uploading all parts, verify with aws s3api list-parts to confirm every part registered correctly.
# Upload part 1 (xaa)
aws s3api upload-part \
--bucket my-upload-bucket \
--key videoplayback.mp4 \
--part-number 1 \
--body xaa \
--upload-id YOUR_UPLOAD_ID
# Response: { "ETag": "\"7f9b8c3e2a1d5f4e8c9b2a6d...\"" }
# Upload part 2 (xab)
aws s3api upload-part \
--bucket my-upload-bucket \
--key videoplayback.mp4 \
--part-number 2 \
--body xab \
--upload-id YOUR_UPLOAD_ID
# ... repeat for all parts ...
# Verify all parts uploaded
aws s3api list-parts \
--bucket my-upload-bucket \
--key videoplayback.mp4 \
--upload-id YOUR_UPLOAD_ID
Create the ETag JSON Manifest
This is the step most tutorials gloss over and where most people get stuck. Create a JSON file called multipart.json in the same directory as your chunks. It lists every part number and its corresponding ETag. This file tells S3 which chunks belong to this upload and in what order to reassemble them. Get one ETag wrong or miss a part number and the completion step will fail. Save it in the same folder as your split files.
{
"Parts": [
{ "ETag": "7f9b8c3e2a1d5f4e8c9b2a6d4e8f1c3a", "PartNumber": 1 },
{ "ETag": "a8e5d2c7f9b4e6a3c8d5f2e9b7c4a6d3", "PartNumber": 2 },
{ "ETag": "c4f8e2b6d9a3c7e5f8b2d6a9c3e7f4b8", "PartNumber": 3 },
{ "ETag": "e9c3f7a5d8b4e6c9f2a7d4b8c6e3f9a2", "PartNumber": 4 },
{ "ETag": "b6d4a8c7f5e9b3d6a2c8f4e7b9c5d8a6", "PartNumber": 5 },
{ "ETag": "d7e3c9f6a4b8d2e5c7f9a3b6d4e8c2f5", "PartNumber": 6 },
{ "ETag": "f2a6d8c4e7b3f6a9c2d5e8b4c7f3a6d9", "PartNumber": 7 }
]
}
Complete the Multipart Upload
Run aws s3api complete-multipart-upload with the JSON manifest file, your bucket, key, and UploadId. S3 reads the manifest, verifies every part and ETag match, then reassembles the chunks into the final object. On success, you get a JSON response with the object's Location URL, final ETag, and checksum. Go to your S3 bucket in the AWS Console and refresh — you'll see the complete file with its original filename, type, and full size. The split files on your local machine can now be deleted.
# Complete the multipart upload
aws s3api complete-multipart-upload \
--multipart-upload file://multipart.json \
--bucket my-upload-bucket \
--key videoplayback.mp4 \
--upload-id YOUR_UPLOAD_ID
# Success response:
# {
# "Location": "https://my-upload-bucket.s3.amazonaws.com/videoplayback.mp4",
# "Bucket": "my-upload-bucket",
# "Key": "videoplayback.mp4",
# "ETag": "\"78298db673a369adf33dd8054bb6bab7-7\""
# }
# Clean up local split files
rm xaa xab xac xad xae xaf xag
Incomplete Uploads Cost You Money
S3 charges storage for uploaded parts even if the multipart upload is never completed. If you initiate an upload, push 40GB of parts, then abandon it — you're paying for 40GB of storage indefinitely. Set up an S3 Lifecycle policy to automatically abort incomplete multipart uploads after 7 days. Go to your bucket → Management → Lifecycle rules → add a rule to "Delete expired object delete markers or incomplete multipart uploads." We've found $340/month in phantom storage charges on a client's AWS bill from abandoned multipart uploads nobody cleaned up.
Frequently Asked Questions
What is the maximum file size for S3 multipart upload?
S3 supports objects up to 5TB total. With multipart upload, you can upload up to 10,000 parts, each between 5MB and 5GB. For a 5TB file, you'd typically use 500MB parts (10,000 parts). The single PUT limit is only 5GB — anything larger requires multipart.
What happens if one part fails during multipart upload?
Only the failed part needs to be re-uploaded. All successfully uploaded parts remain intact. Re-run the upload-part command for the specific part number that failed, using the same UploadId. The other parts are unaffected.
Do I get charged for incomplete multipart uploads?
Yes. S3 charges standard storage rates for all uploaded parts, even if the multipart upload is never completed. Set up an S3 Lifecycle policy to automatically abort incomplete uploads after a set number of days to avoid accumulating phantom storage costs.
How do I choose the right chunk size for splitting files?
On reliable high-speed connections, use larger chunks (100MB–500MB) to reduce the number of API calls. On unreliable networks, use smaller chunks (25MB–50MB) so failed retries cost less bandwidth. The minimum part size is 5MB (except the last part). Never exceed 5GB per part.
Can I use the split command on Windows?
The split command is native to Linux and macOS. On Windows, use Git Bash (comes with Git for Windows), WSL (Windows Subsystem for Linux), or a tool like 7-Zip to split files. Alternatively, you can use PowerShell with Get-Content and Set-Content to split binary files into chunks.
Are Large File Transfers Slowing Down Your Team?
If your team is still doing single-PUT uploads and restarting 10GB transfers every time the VPN drops, you're burning hours and bandwidth every week. Multipart upload is one piece of the puzzle — but automating it into your deployment pipeline, backup rotation, and data migration workflows is where the real time savings compound. We've set up automated S3 multipart upload pipelines for 23 engineering teams in the last year. Average result: 73% reduction in failed transfer retries and $2,400/month saved in wasted bandwidth costs. Let us architect your cloud data transfer pipeline.