7

I'm trying to upload a .zip file to S3 using boto3 for python but the .zip file in my directory is not uploaded correctly. The code downloads all emails of a given user, zips them in the same directory and uploads them to an S3 bucket. The problem is that the file that gets uploaded is not the one I intend to upload. Instead a file with 18kb only appears.

Here's the code:

import sys
import imaplib
import getpass
import email
import shutil
import boto3
import os

username = input("Enter user's first name: ")
surname = input("Enter user's surname: ")
email_address = username + "." + surname + "@gmail.com"
password = getpass.getpass()
directory = username + surname + '/'
def download_emails(server):
    result, data = server.uid('search', None, "ALL")    #search all email and return their uids
    if result == 'OK':
        for num in data[0].split():
            result, data = server.uid('fetch', num, '(RFC822)')    #RFC is a standard for the format of ARPA Internet text messages
            if result == 'OK':
                email_message = email.message_from_bytes(data[0][1])    #raw email text including headers
                file_name = email_message['Subject']       #use dates and file names(can be changed)
                if not os.path.exists(directory):
                    os.makedirs(directory)      #create a dir for user's emails
                try:
                    email_file = open(directory + file_name+'.eml', 'wb')   #open a file for each email and insert the data.
                    email_file.write(data[0][1])
                    email_file.close()
                except:
                    pass

#function to zip all the emails
def archive(zipname, directory):
    return shutil.make_archive(zipname, 'zip', root_dir=directory, base_dir=None)

#function to upload zipped emails to AWS bucket
def upload_to_s3(file_name):
    s3 = boto3.resource('s3',
                aws_access_key_id=accessKey,
                aws_secret_access_key=secretKey,
                aws_session_token=secretToken,
                )

    s3.Bucket('user-backups').put_object(Key=username.title() + " " +
                                surname.title() + "/" + file_name, Body=file_name)
    print("Uploaded")


def main():
    server = imaplib.IMAP4_SSL("imap.gmail.com", 993)   #connect to gmail's imap server
    server.login(email_address, password)   #enter creds
    result, data = server.select('"[Gmail]/All Mail"')  #get all emails(inbox, outbox etc)
    if result == 'OK':
        print("Downloading")
        download_emails(server)
        server.close()
    else:
        print("ERROR: Unable to open mailbox ", result)
    server.logout()
    archive(username + surname, directory)
    upload_to_s3(username + surname + ".zip")
    #os.remove(email_address + ".zip")
    #shutil.rmtree(email_address)
    print("Done")
if __name__ == "__main__":
    main()

5 Answers 5

3

You can check out this article for more information.

There are a number of ways to upload. Check out this boto3 document where I have the methods listed below:

The managed upload methods are exposed in both the client and resource interfaces of boto3:

S3.Client method to upload a file by name: S3.Client.upload_file()
S3.Client method to upload a readable file-like object: S3.Client.upload_fileobj()
S3.Bucket method to upload a file by name: S3.Bucket.upload_file()
S3.Bucket method to upload a readable file-like object: S3.Bucket.upload_fileobj()
S3.Object method to upload a file by name: S3.Object.upload_file()
S3.Object method to upload a readable file-like object: S3.Object.upload_fileobj()

I made it work using s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None) .
Upload a file to an S3 object.

import boto3
s3Resource = boto3.resource('s3')

try: 
    s3Resource.meta.client.upload_file('/path/to/file', 'bucketName', 'keyName')
except Exception as err:
    print(err)
2

None of the above answers worked!
The following code worked for me..

 import os 
 def upload_file_zip(local_file_path):
   s3_client = boto3.client('s3')
   s3_path = os.path.join(os.path.basename(local_file_path))
   with open(local_file_path,mode='rb') as data:
      s3_client.upload_fileobj(data, BUCKET_NAME, s3_path)

Updated code as s3_folder parameter is not required here.

2
1

The put_object function accepts Body which is either bytes object or a file object. You have currently just passed the plain filename (a string).

From documentation:

Body (bytes or seekable file-like object) -- Object data.

So the fix should be to pass the file object. Consult this to know how to do that.

3
  • The thing is it is uploaded as a .zip file. I get a zip file in my directory that's 23MB in size. When I upload it it's like a totally different file. And I'm passing the filename + extension in upload_to_s3.
    – davidb
    Mar 29, 2018 at 9:58
  • @davidb Try this and share your findings
    – Nabin
    Mar 29, 2018 at 10:33
  • @davidb can u share your code on how u use zipfile to upload? It seems that we have to know the file names within the zipped file to do it?
    – Jun
    Sep 14, 2018 at 6:54
1

Just use s3.client.upload_file.

upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)

def upload_to_s3(file_name):
  s3 = boto3.client('s3')
  Key = username.title() + " " + surname.title() + "/" + file_name
  try: 
     s3.meta.client.upload_file('/path/to/file', 'user-backups', Key)
  except Exception as e:
     print(e)
0

I managed to upload a .zip file by means of the following code:

def write_to_s3(filename, bucket, key):
    s3 = boto3.resource(service_name='s3',
                        aws_access_key_id=os.environ["AWS_USER1_ACCESS_KEY"],
                        aws_secret_access_key=os.environ["AWS_USER1_SECRET_ACCESS_KEY"])
    s3.meta.client.upload_file(filename, bucket, key)

Note : I had to use boto3.resource() instead of boto3.client() as was answered above by mootmoot, as it threw an Exception().

Exception thrown if boto3.client() is used:

AttributeError: 'ClientMeta' object has no attribute 'client'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.