1

I need to upload files (for a total of 100GB with several files of 10GB) from a PowerShell script to S3: http://docs.aws.amazon.com/powershell/latest/reference/Index.html (Amazon Simple Storage Service)

I have used AWS SDK for .NET and works perfectly for small number of files. I just create the bucket in previous code and try to upload all the files with Write-S3Object

Write-S3Object -Region $S3_Region -AccessKey $S3_AccessKey -SecretKey $S3_SecretKey -ServerSideEncryption "AES256" -Folder $sourceFolder -BucketName $S3_Bucket -Recurse -KeyPrefix "/" 

But I have several problems with stability. It fails very often with two types of errors. (There are 4683 files to upload)

  • Suddenly with no error it finishes just uploading part of the files

Uploaded 2664 object(s) to bucket 'bucket.test' from 'R:\temp\files' with keyprefix '/'

  • It fails with exception:

Uploaded 35 object(s) to bucket 'bucket.test' from 'R:\temp\files' with keyprefix '/' System.InvalidOperationException: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. ---> Amazon.S3.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.

So my questions are:

  • Am I doing something wrong?
  • Is there any workaround or fix for my problem?
  • Is there any other way to upload without using AWS SDK for .NET?

2 Answers 2

5

Rather than using Write-S3Object I am using S3 TransferUtility object that has the capability to upload large files using multipart upload.

I have been using the below powershell script to upload files to S3.

function UploadToAmazonUsingSDK()
{
    param([string] $sourceLocation, [string] $bucketName, [string] $versionNumber)    

    Add-Type -Path "C:\Program Files (x86)\AWS SDK for .NET\bin\Net45\AWSSDK.dll"

    $AccessKey= "your aws accesskey"
    $SecretKey ="your secret key"

    $s3Config=New-Object Amazon.S3.AmazonS3Config
    $s3Config.UseHttp = $false
    $s3Config.ServiceURL = "https://s3-eu-west-1.amazonaws.com"
    $s3Config.BufferSize = 1024 * 32

    $client=[Amazon.AWSClientFactory]::CreateAmazonS3Client($AccessKey,$SecretKey,$s3Config)

    $transferUtility = New-Object -TypeName Amazon.S3.Transfer.TransferUtility($client)   

    $files = Get-ChildItem $sourceLocation

    foreach ($fileName in $files) {        
        $amazonKey = $versionNumber + '/' + $fileName        
        Write-Host $amazonKey
        Write-Host $fileName 
        Write-Host $fileName.FullName

        $transferUtilRequest = New-Object -TypeName Amazon.S3.Transfer.TransferUtilityUploadRequest
        $transferUtilRequest.BucketName = $bucketName
        $transferUtilRequest.FilePath = $fileName.FullName
        $transferUtilRequest.Key = $amazonKey
        $transferUtilRequest.AutoCloseStream = $true
        $transferUtility.Upload($transferUtilRequest)
    }    
}
1
  • it works as a charm. The function is not exactly what I need but is ok as TransferUtility has a method for directories. Thanks :) Apr 27, 2015 at 15:29
2

If you are Golang fan you can try out https://github.com/minio/mc. We also have a precompiled binary for Windows on 64bit which can be used with powershell, command prompt etc.

https://dl.minio.io:9000/updates/2015/Sept/windows-amd64/mc.exe

To add your S3 credentials simply do

C:\> mc.exe config host add s3.amazonaws.com BKIKJAA5BMMU2RHO6IBB V7f1CwQqAcwo80UEIJEjc5gVQUSSx5ohQ9GSrr12

mc.exe implements a nice progress bar, colored console output, multipart uploads, session management etc.

We will be making a new release upcoming week as well which will have 32bit and 64bit Windows both supported.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.