15

Using node.js, with the intention of running this module as an AWS Lambda function.

Using s3.getObject() from aws-sdk, I am able to successfully pick up a very large CSV file from Amazon S3. The intention is to read each line in the file and emit an event with the body of each line.

In all examples I could find, it looks like the entire CSV file in S3 has to be buffered or streamed, converted to a string and then read line by line.

s3.getObject(params, function(err, data) {
   var body = data.Body.toString('utf-8');
}

This operation takes a very long time, given the size of the source CSV file. Also, the CSV rows are of varying length, and I'm not certain if I can use the buffer size as an option.

Question

Is there a way to pick up the S3 file in node.js and read/transform it line by line, which avoids stringifying the entire file in memory first?

Ideally, I'd prefer to use the better capabilities of fast-csv and/or node-csv, instead of looping manually.

4 Answers 4

26

You should just be able to use the createReadStream method and pipe it into fast-csv:

const s3Stream = s3.getObject(params).createReadStream()
require('fast-csv').fromStream(s3Stream)
  .on('data', (data) => {
    // do something here
  })
4
  • This works pretty well. Just to add to it, if you are interested to know when the parsing ends, add .on('end' () => { //Your handling })
    – Deepak G M
    Commented May 3, 2019 at 12:22
  • @DeepakGM you forgot a comma .on('end', () => { }) Commented Jun 28, 2019 at 3:26
  • 2
    This method is deprecated, see my answer below for using parseStream method instead
    – Kai Durai
    Commented Jan 16, 2021 at 14:49
  • what the version of the lib contain fromStream ? parseStream throws an Error....
    – pta
    Commented Nov 17, 2021 at 20:12
17

I do not have enough reputation to comment but as of now the accepted answer method "fromStream" is deprecated on 'fast-csv'. Now you'll need to use the parseStream method:

const s3Stream = s3.getObject(params).createReadStream()
require('fast-csv').parseStream(s3Stream)
  .on('data', (data) => {
    // use rows
  })
1
  • Thanks for adding this. I was looking for it. :)
    – Hoon
    Commented Jan 13, 2021 at 21:34
6

For me, the answer that solved my issue was,

  const csv = require('@fast-csv/parse');

  const params = {
    Bucket: srcBucket,
    Key: srcKey,
  };
  const csvFile = s3.getObject(params).createReadStream();

  let parserFcn = new Promise((resolve, reject) => {
    const parser = csv
      .parseStream(csvFile, { headers: true })
      .on("data", function (data) {
        console.log('Data parsed: ', data);
      })
      .on("end", function () {
        resolve("csv parse process finished");
      })
      .on("error", function () {
        reject("csv parse process failed");
      });
  });

  try {
    await parserFcn;
  } catch (error) {
    console.log("Get Error: ", error);
  }
0
0

Not line by line but you can get S3 objects by byte range using the Range header. So you could read, say, 1000 bytes at a time and manage the new lines on your end as you read the data. Look on the GET Object documentation and search for the Range header.

1
  • 2
    with range, one might get served with broken csv data Commented Mar 7, 2023 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.