19

I manually upload the JSON file to google cloud storage by creating a new project. I am able to read the metadata for a file but I don't know how to read the JSON content.

The code I used to read the metadata is:

var Storage = require('@google-cloud/storage');
const storage = Storage({
    keyFilename: 'service-account-file-path',
    projectId: 'project-id'
});
storage
    .bucket('project-name')
    .file('file-name')
    .getMetadata()
    .then(results => {
        console.log("results is", results[0])
    })
    .catch(err => {
        console.error('ERROR:', err);
    });

Can someone guide me to the way to read the JSON file content?

2
  • never used this but it seems like you need to download it to me
    – jonatjano
    Aug 22, 2018 at 15:29
  • maybe this question can help you
    – jonatjano
    Aug 22, 2018 at 15:33

4 Answers 4

23

I've used the following code to read a json file from Cloud Storage:

    'use strict';
    const Storage = require('@google-cloud/storage');
    const storage = Storage();
    exports.readFile = (req, res) => {
            console.log('Reading File');
            var archivo = storage.bucket('your-bucket').file('your-JSON-file').createReadStream();
            console.log('Concat Data');
            var  buf = '';
            archivo.on('data', function(d) {
              buf += d;
            }).on('end', function() {
              console.log(buf);
              console.log("End");
              res.send(buf);
            });     

    };

I'm reading from a stream and concat all the data within the file to the buf variable.

Hope it helps.

UPDATE

To read multiple files:

'use strict';
const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
listFiles();

async function listFiles() {
        const bucketName = 'your-bucket'
        console.log('Listing objects in a Bucket');
        const [files] = await storage.bucket(bucketName).getFiles();
        files.forEach(file => {
            console.log('Reading: '+file.name);
            var archivo = file.createReadStream();
            console.log('Concat Data');
            var  buf = '';
            archivo.on('data', function(d) {
                buf += d;
            }).on('end', function() {
                console.log(buf);
                console.log("End");
            });    
        });
};
4
  • This works if you are reading ONE file. But what if you want to read multiple files.
    – Joakim M
    Mar 26, 2019 at 12:37
  • Thanks. Looks like what I had at first. But have you tested this? createReadStream seems to go on an async adventure and the result is ballistic :D I had to trick it with await...
    – Joakim M
    Mar 27, 2019 at 9:21
  • Yes, that will depend on how big are your files too
    – F10
    Mar 27, 2019 at 11:46
  • Surprising how long I had to look before someone finally had a working example. Thank you! Jul 27, 2019 at 2:59
16

I was using the createWriteStream method like the other answers but I had a problem with the output in that it randomly output invalid characters (�) for some characters in a string. I thought it could be some encoding problems.

I came up with my workaround that uses the download method. The download method returns a DownloadResponse that contains an array of Buffer. We then use Buffer.toString() method and give it an encoding of utf8 and parse the result with JSON.parse().

const downloadAsJson = async (bucket, path) => {
  const file = await new Storage()
    .bucket(bucket)
    .file(path)
    .download();
  return JSON.parse(file[0].toString('utf8'));
}
5
  • I'm not clear how this answer differs from the answer added a year before this one?
    – Liam
    Oct 5, 2022 at 11:12
  • @Nathan did you ever solve the stream invisible characters issue? We have this problem, but we need to use streams b/c they are large files.
    – Sean Clark
    Oct 4 at 14:44
  • Hi @Liam, I worked on this so long ago but I believe I initially tried to use the solution that you pointed to. However I faced some encoding issue so I had to make some workaround and decided to share it. So the difference would be an additional improvement on the encoding issue.
    – Nathan
    Nov 15 at 9:33
  • Hi @SeanClark, I'm no longer working on this project so unfortunately I don't have further suggestions on this :)
    – Nathan
    Nov 15 at 9:34
  • @Nathan no worries! I was able to solve this problem by using "transformHeader" and .trim() in PapaParse to strip the excel BOM character.
    – Sean Clark
    Nov 15 at 13:21
14

There exists a convenient method:'download' to download a file into memory or to a local destination. You may use download method as follows:

const bucketName='bucket name here';
const fileName='file name here';
const storage = new Storage.Storage();
const file = storage.bucket(bucketName).file(fileName);

file.download(function(err, contents) {
     console.log("file err: "+err);  
     console.log("file data: "+contents);   
}); 
10

A modern version of this:

const { Storage } = require('@google-cloud/storage')
const storage = new Storage()
const bucket = storage.bucket('my-bucket')

// The function that returns a JSON string
const readJsonFromFile = async remoteFilePath => new Promise((resolve, reject) => {
  let buf = ''
  bucket.file(remoteFilePath)
    .createReadStream()
    .on('data', d => (buf += d))
    .on('end', () => resolve(buf))
    .on('error', e => reject(e))
})

// Example usage
(async () => {
  try {
    const json = await readJsonFromFile('path/to/json-file.json')
    console.log(json)
  } catch (e) {
    console.error(e)
  }
})()

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.