Using node.js, with the intention of running this module as an AWS Lambda function.
Using s3.getObject()
from aws-sdk
, I am able to successfully pick up a very large CSV file from Amazon S3. The intention is to read each line in the file and emit an event with the body of each line.
In all examples I could find, it looks like the entire CSV file in S3 has to be buffered or streamed, converted to a string and then read line by line.
s3.getObject(params, function(err, data) {
var body = data.Body.toString('utf-8');
}
This operation takes a very long time, given the size of the source CSV file. Also, the CSV rows are of varying length, and I'm not certain if I can use the buffer size as an option.
Question
Is there a way to pick up the S3 file in node.js and read/transform it line by line, which avoids stringifying the entire file in memory first?
Ideally, I'd prefer to use the better capabilities of fast-csv
and/or node-csv
, instead of looping manually.