To manage S3 multipart uploads efficiently, you can use the AWS CLI's `aws s3api` command.
The `aws s3api create-multipart-upload` command is used to initiate an S3 multipart upload.
To monitor the upload progress, use the `aws s3api list-multipart-uploads` command.
The `aws s3api complete-multipart-upload` command is used to complete an S3 multipart upload.
To abort an S3 multipart upload, use the `aws s3api abort-multipart-upload` command.
You can also use the `aws s3api list-parts` command to list the parts of an S3 multipart upload.
Using these commands, you can efficiently manage S3 multipart uploads and ensure smooth data transfer.
Creating a Multipart Upload
To create a multipart upload, you'll need to use the AWS CLI command `aws s3api create-multipart-upload`. This will generate an UploadID that you'll need later.
The syntax for this command is `aws s3api create-multipart-upload --bucket [Bucket name] --key [original file name]`. Copy the UploadId into a text file, like Notepad.
You'll also need to upload each chunk of your file using the `aws s3api upload-part` command. The syntax for this is `aws s3api upload-part --bucket [bucketname] --key [filename] --part-number [number] --body [chunk file name] --upload-id [id]`.
For example, if you're using the bucket `s3multipart-final` and the file `video.mp4`, the command would be `aws s3api upload-part --bucket s3multipart-final --key video.mp4 --part-number 1 --body xaa --upload-id`.
Remember to copy the ETag id and Part number to your Notepad for each chunk file. To do this, simply repeat the `aws s3api upload-part` command for each chunk file, replacing the `--part-number` and `--body` values with the corresponding values from your table.
Managing Uploads
To create a multipart upload, you'll need to use the AWS CLI command `aws s3api create-multipart-upload` with the syntax `--bucket [Bucket name] --key [original file name]`. This will generate an UploadID, which you should copy into a text file, like Notepad.
You'll use the UploadID to upload chunks of your file. The syntax for this is `aws s3api upload-part --bucket [bucketname] --key [filename] --part-number [number] --body [chunk file name] --upload-id [id]`. For example, if you're uploading a video file, the command might look like `aws s3api upload-part --bucket s3multipart-final --key video.mp4 --part-number 1 --body xaa --upload-id`.
Repeat this command for each chunk file, replacing the `--part-number` and `--body` values with the corresponding values from your table.
To complete the multipart upload, you'll use the `aws s3api complete-multipart-upload` command with the syntax `--multipart-upload [json file link] --bucket [upload bucket name] --key [original file name] --upload-id [upload id]`. This will join all the chunks together and make them available in your S3 bucket.
Retrieving MD5 Checksum Value
You can retrieve the MD5 checksum value for your uploaded file by using the `aws s3api head-object` command. This command allows you to retrieve metadata about an object in S3.
To retrieve the MD5 checksum value, simply run the `aws s3api head-object` command and specify the bucket name, object key, and upload ID.
Here's a list of the required parameters:
- bucket: the name of the S3 bucket where your file is stored
- key: the original file name of your uploaded file
- upload ID: the unique identifier for your multipart upload
By using this command, you'll be able to retrieve the MD5 checksum value for your uploaded file.
Sources
- https://docs.outscale.com/en/userguide/Tutorial-Using-Multipart-Upload-With-AWS-CLI-s3.html
- https://www.linkedin.com/pulse/aws-s3-multipart-upload-using-cli-ravindra-singh
- https://www.geeksforgeeks.org/how-to-upload-files-to-aws-s3-using-cli/
- https://www.systutorials.com/uploading-large-files-amazon-s3-aws-cli/
- https://stackoverflow.com/questions/45477660/aws-s3-cli-multipart-upload
Featured Images: pexels.com