2

So I've recently delved into trying to understand promises and the purpose behind them due to javascripts asynchronous behavior. While I "think" I understand, I still struggle with how to promisify something to return the future value, then execute a new block of code to do something else. Two main node modules I'm using:

  • pg-promise
  • exceljs

What I'd like to do is read a file, then once fully read, iterate of each worksheet executing DB commands. Then once all worksheets are processed, go back and delete the original file I read. Here is the code I have. I have it working to the point everything writes into the database just fine, even when there are multiple worksheets. What I don't have working is setting it up to identify when all the worksheets have been fully processed, then to go remove the file

workbook.csv.readFile(fileName)
            .then(function () {
                // this array I was going to use to somehow populate a true/false array.  
                // Then when done with each sheet, push a true into the array.  
                // When all elements were true could signify all the processing is done... 
                // but have no idea how to utilize this!  
                // So left it in to take up space because wtf...
                var arrWorksheetComplete = [];

                workbook.eachSheet(function (worksheet) {
                    console.log(worksheet.name);
                    db.tx(function (t) {
                        var insertStatements = [];
                        for (var i = 2; i <= worksheet._rows.length; i++) {
                            // here we create a new array from the worksheet, as we need a 0 index based array.
                            // the worksheet values actually begins at element 1.  We will splice to dump the undefined element at index 0.
                            // This will allow the batch promises to work correctly... otherwise everything will be offset by 1
                            var arrValues = Array.from(worksheet.getRow(i).values);
                            arrValues.splice(0, 1);

                            // these queries are upsert.  Inserts will occur first, however if they error on the constraint, an update will occur instead.
                            insertStatements.push(t.one('insert into rq_data' +
                                '(col1, col2, col3) ' +
                                'values($1, $2, $3) ' +
                                'ON CONFLICT ON CONSTRAINT key_constraint DO UPDATE SET ' +
                                '(prodname) = ' +
                                '($3) RETURNING autokey',
                                arrValues));
                        }
                        return t.batch(insertStatements);
                    })
                    .then(function (data) {
                        console.log('Success:', 'Inserted/Updated ' + data.length + ' records');
                    })
                    .catch(function (error) {
                        console.log('ERROR:', error.message || error);
                    });
                });
            });

I would like to be able to say

.then(function(){
    // everything processed!
    removeFile(fileName)
    // this probably also wouldn't work as by now fileName is out of context?
});

But I'm super confused when having a promise inside a promise.. I have the db.tx call which is essentially a promise nested inside the .eachSheet function. Please help a dumb programmer understand! Been beating head against wall for hours on this one. :)

4
  • You need to promisify workbook.eachSheet and chain it. And you should move db.tx outside of eachSheet call, because you need only a single database transaction.
    – vitaly-t
    Oct 16, 2016 at 13:40
  • Promisify ALL your async operations and then when you nest them, return a promise to chain them. Do NOT mix regular async callbacks with promises.
    – jfriend00
    Oct 16, 2016 at 18:26
  • @vitaly-t Have been looking around and I'm struggling to understand how to "promisify" the eachsheet function. Is there an example you can point me too?
    – dvsoukup
    Oct 16, 2016 at 20:36
  • You can solve this by using promises in combination with a ES6 Generator - any environmental restaints preventing you from this?
    – rabbitco
    Oct 16, 2016 at 23:03

1 Answer 1

0

If i understand correctly, you're trying to chain promises.

I suggest you to read this great article on Promises anti-pattern (see 'The Collection Kerfuffle' section)

If you need to execute promises in series, this article suggests to use reduce.

I'll rewrite your snippet to:

workbook.csv.readFile(fileName).then(function () {

  processWorksheets().then(function() {
    // all worksheets processed!
  });

});

function processWorksheets() {
    var worksheets = [];

    // first, build an array of worksheet
    workbook.eachSheet(function (worksheet) {
        worksheets.push(worksheet);
    }); 

    // then chain promises using Array.reduce
    return worksheets.reduce(function(promise, item) {
        // promise is the the value previously returned in the last invocation of the callback.
        // item is a worksheet

        // when the previous promise will be resolved, call saveWorksheet on the next worksheet
        return promise.then(function(result) {
            return saveWorksheet(item, result);
        });        

    }, Promise.resolve()); // start chain with a 'fake' promise
}

// this method returns a promise
function saveWorksheet(worksheet, result) {

    return db.tx(function (t) {

      var insertStatements = [];
      for (var i = 2; i <= worksheet._rows.length; i++) {
        // here we create a new array from the worksheet, as we need a 0 index based array.
        // the worksheet values actually begins at element 1.  We will splice to dump the undefined element at index 0.
        // This will allow the batch promises to work correctly... otherwise everything will be offset by 1
        var arrValues = Array.from(worksheet.getRow(i).values);
        arrValues.splice(0, 1);

        // these queries are upsert.  Inserts will occur first, however if they error on the constraint, an update will occur instead.
        insertStatements.push(t.one('insert into rq_data' +
                                    '(col1, col2, col3) ' +
                                    'values($1, $2, $3) ' +
                                    'ON CONFLICT ON CONSTRAINT key_constraint DO UPDATE SET ' +
                                    '(prodname) = ' +
                                    '($3) RETURNING autokey',
                                    arrValues));
      }

      return t.batch(insertStatements);
    })
    // this two below can be removed...
    .then(function (data) {
       return new Promise((resolve, reject) => { 
          console.log('Success:', 'Inserted/Updated ' + data.length + ' records');
          resolve();
       });
    })
    .catch(function (error) {
        return new Promise((resolve, reject) => {
        console.log('ERROR:', error.message || error);
        reject();
      });
    });

}

Don't forget to include the promise module:

var Promise = require('promise');

I haven't tested my code, could contains some typo errors.

3
  • One of the key ideas of working with sheets of data is to avoid overloading Node.js memory. One should not stack all worksheets into the memory, they should be read and processed one by one.
    – vitaly-t
    Oct 16, 2016 at 14:05
  • @vitaly-t you're right, my code is all about chaining promises... It can be edited, one approach could be to loop through worksheets filenames and to load the corresponding one inside the reduce callback.
    – Gab
    Oct 16, 2016 at 14:28
  • requirements changed and wound up needing to only ever work this against a single worksheet... so could remove the looping call. I did adapt some of this logic though! Thank you!
    – dvsoukup
    Oct 17, 2016 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.