2

I want to write a data frame column in .ann format to S3.

Right now I am using the following code to do that.

df['user_input'].to_csv(ann_file_path, header=None, index=None, sep=' ')

Where ann_file_path is the full path of the .ann file on the Server.

I am getting following error message:

[Errno 22] Invalid argument: 'https://s3-eu-west-1.amazonaws.com/bucket/sub_folder/somefile.ann'

Why am I getting that?

Also, do I need to use Boto3 to write or can I directly write the file on S3 with full path?

I can think of some authorization might be required for that but the error message seems different from something related to authorization.

2
  • 2
    Putting stuff into S3 need credential handshake, you definitely need boto3 or a boto3 wrapper.
    – mootmoot
    Commented Oct 19, 2018 at 9:25
  • 1
    Thank you, I've resolved the problem and posted my answer. I really needed AWS credential handshake :)
    – disp_name
    Commented Oct 19, 2018 at 12:45

1 Answer 1

4

I've resolved. We need AWS handshake using access_key_id and secret_key for AWS.

Get URL starting from the bucket name (not https:/...), hence get rid of whatever before it.

My URL: https://s3-eu-west-1.amazonaws.com/bucket/sub_folder/somefile.ann

Transformed to: bucket/sub_folder/somefile.ann

Code to do that: ann_file_path = ann_file_path.split('.com/', 1)[1]

Once I got ann_file_path, I used s3fs python library to upload the ann file to the server.

bytes_to_write = df['user_input'].to_csv(header=None, index=None).encode()
fs = s3fs.S3FileSystem(key=settings.AWS_ACCESS_KEY_ID, secret=settings.AWS_SECRET_ACCESS_KEY)
with fs.open(ann_file_path, 'wb') as f:
   f.write(bytes_to_write)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.