I would like to add some stuff to this question.
I got an issue while uploading a .gz file with 75GB.
The error was
An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive
I did the below changes
aws configure set default.s3.multipart_chunksize 64MB
aws configure set default.s3.max_concurrent_requests 1000
aws configure set default.s3.max_queue_size 100000
In addition to that, I contact customer support to ensure it'll work fine in future uploads.
Here is the response from AWS customer support.
To mitigate this issue, we have modified the AWS CLI S3 Configuration by executing the below "aws configure set" command to increase the multipart_chunksize value for the default profile:
$ aws configure set default.s3.multipart_chunksize 64MB
AWS CLI S3 Configuration : https://docs.aws.amazon.com/cli/latest/topic/s3-config.html
Please be informed that the parameter --expected-size (string) is to specify the expected size of a stream in terms of bytes. This argument is needed only when a stream is being uploaded to s3 and the size is larger than 50GB. Failure to include this argument under these conditions may result in a failed upload due to too many parts in upload.
Hence, I would also request you to please provide/pass the parameter --expected-size along with the command that you are executing. For example, please refer the below 'aws s3 cp' command:
$ aws s3 cp - s3://mybucket/stream.txt --expected-size 54760833024
'aws s3 cp' CLI Reference : https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html