0

I created an S3 bucket using boto3 in python and uploaded files to the bucket. I tried to generate the URL of that object, through the code below.

But the problem is, when I call the bucket_search method, it only returns a maximum of 1000 keys, so when the bucket has more than 1000 objects, bucket search is unable to find the desired object since it's not in the first 1000 objects. (More than 1000 objects are uploaded to the bucket alreaady)

As a result, the URL isn't generated and I get an empty string in return. Please guide me on how I can fix this issue, and properly search for the desired object and generate the URL using boto3 Python.

I would also appreciate it if somebody helped me to make the file public while being uploaded.

    def bucket_search(client_name, bucket, filepath):
    """
    Search any file/folder of S3 bucket
        Parameters:
            client_name: string
                Object name(e.g. 'client')
            bucket: string
                Bucket_name
            filepath: string
                Directory of the file or folder
        Returns: 
            True if filepath found, else False.
    """
    contents = client_name.list_objects(Bucket=bucket).get('Contents')
    if contents != None:
        for p in contents:
            if p['Key'] == filepath:
                return True
    return False


def bucket_create_folder(client_name, bucket, folder_name):
    """
    Create a folder in S3 bucket
        Parameters:
            client_name: object
                Object name(e.g. client)
            bucket: string
                Bucket name
            folder_name: string
                directory of the folder
        Returns: 
            True if folder_name is created, else False.
    """
    if not bucket_search(client_name, bucket, folder_name + '/'):
        client_name.put_object(Bucket=bucket, Key=(folder_name + '/'))
        return True
    return False


def bucket_upload_file(client_name, bucket, filepath, object_name=None):
    """Upload a file in S3 bucket
        Parameters:
            client_name: object
                Object name(e.g. client)
            bucket: string
                Bucket name
            filpath: string
                Directory of the file or folder
            object_name: string
                Name after uploaded. If None, the value of the file path will be its value.
        Returns: 
            True if the file is uploaded, else False.
    """

    # If S3 object_name was not specified, use filepath
    if object_name is None:
        object_name = filepath

    try:
        response = client_name.upload_file(filepath, bucket, object_name)
        print(type(response))
    except ClientError as e:
        logging.error(e)
        return False
    return True


def bucket_url_generator(client_name, bucket, filepath):
    """Generate URL of a file/folder of S3 bucket
        Parameters:
            client_name: object
                Object name(e.g. client)
            bucket: string
                Bucket name
            filpath: string
                Directory of the file or folder
        Returns:
            url if filepath is found, else False.
    """
    if bucket_search(client_name, bucket, filepath):
        url = client_name.generate_presigned_url(
            ClientMethod='get_object',
            Params={
                'Bucket': bucket,
                'Key': filepath
            }, ExpiresIn=24*3600
        )
        return url
    else:
        return ''
2

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.