2

I am currently trying to read a .csv directly from an AWS S3 bucket. However, I am always receiving a FileNotFoundError. Weirdly after I can actually see the content of the .csv file.

Traceback (most recent call last): File "<console>", line 1, in <module> FileNotFoundError: [Errno 2] No such file or directory: b',event_id,ds,yhat,yhat_lower,yhat_upper\n0,277,2019-09-04 07:14:08.051643,0.3054256311115928,0.29750667741533227,0.31441960581142636\n'

Here my code:

BUCKET_NAME = 'fbprophet'
FORECAST_DATA_OBJECT = 'forecast.csv'
s3 = boto3.client(
    's3',
    aws_access_key_id=settings.ML_AWS_ACCESS_KEY_ID,
    aws_secret_access_key=settings.ML_AWS_SECRET_ACCESS_KEY,
)
obj = s3.get_object(Bucket=BUCKET_NAME, Key=FORECAST_DATA_OBJECT)
data = obj['Body'].read()

with open(data, newline='') as csvfile:
    spamreader = csv.reader(io.BytesIO(csvfile), delimiter=' ', quotechar='|')
    for row in spamreader:
        print(', '.join(row))

And here some content of my .csv file. I ideally I could access each row as a dictionary with row['event_id']. E.g. to access yhat I could just write row['event_id']['yhat]. But currently, that's not how it works at all.

    event_id    ds  yhat    yhat_lower  yhat_upper
0   277 2019-09-04 7:14:08  0.3054256311    0.2975066774    0.3144196058
0   178 2019-09-28  0.3454256311    0.2275066774    0.3944196058

2 Answers 2

8

Just get rid of with open(data, newline='') as csvfile:
because open expects a name of a file on your local filesystem.
You should pass data to io.BytesIO directly.

BUCKET_NAME = 'fbprophet'
FORECAST_DATA_OBJECT = 'forecast.csv'
s3 = boto3.client(
    's3',
    aws_access_key_id=settings.ML_AWS_ACCESS_KEY_ID,
    aws_secret_access_key=settings.ML_AWS_SECRET_ACCESS_KEY,
)
obj = s3.get_object(Bucket=BUCKET_NAME, Key=FORECAST_DATA_OBJECT)
data = obj['Body'].read().decode('utf-8')
spamreader = csv.reader(io.StringIO(data), delimiter=' ', quotechar='|')
for row in spamreader:
    print(', '.join(row))

Edit: Apparently csv.reader expects strings, not bytes, so you need to decode the response and wrap data in is.StringIO instead.

1
  • Hi taras, I tried that but there I receive the following error Traceback (most recent call last): File "<console>", line 1, in <module> _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?) Do you have an idea what I am doing wrong there?
    – Joey Coder
    Sep 9, 2019 at 8:17
0

Use:

spamreader = csv.reader(io.BytesIO(data), delimiter=',', quotechar='|')
for row in spamreader:
    print(', '.join(row))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.