17

I have a large sheet with around 30 importxml functions that obtain data from a website that updates usually twice a day.

I would like to run the importxml function on a timely basis (every 8 hours) for my Google Spreadsheet to save the data in another sheet. The saving already works, however the updating does not!

I read in Google Spreadsheet row update that it might run every 2 hours, however I do not believe that this is true, because since I added it to my sheet nothing has changed or updated, when the spreadsheet is NOT opened.

How can I "trigger" the importxml function in my Google Spreadsheet in an easy way, as I have a lot of importxml functions in it?

4 Answers 4

20

I made a couple of adjustments to Mogsdad's answer:

  • Fixed the releaseLock() call placement
  • Updates (or adds) a querystring parameter to the url in the import function (as opposed to storing, removing, waiting 5 seconds, and then restoring all relevant formulas)
  • Works on a specific sheet in your spreadsheet
  • Shows time of last update

...

function RefreshImports() {
  var lock = LockService.getScriptLock();
  if (!lock.tryLock(5000)) return;             // Wait up to 5s for previous refresh to end.

  var id = "[YOUR SPREADSHEET ID]";
  var ss = SpreadsheetApp.openById(id);
  var sheet = ss.getSheetByName("[SHEET NAME]");
  var dataRange = sheet.getDataRange();
  var formulas = dataRange.getFormulas();
  var content = "";
  var now = new Date();
  var time = now.getTime();
  var re = /.*[^a-z0-9]import(?:xml|data|feed|html|range)\(.*/gi;
  var re2 = /((\?|&)(update=[0-9]*))/gi;
  var re3 = /(",)/gi;

  for (var row=0; row<formulas.length; row++) {
    for (var col=0; col<formulas[0].length; col++) {
      content = formulas[row][col];
      if (content != "") {
        var match = content.search(re);
        if (match !== -1 ) {
          // import function is used in this cell
          var updatedContent = content.toString().replace(re2,"$2update=" + time);
          if (updatedContent == content) {
            // No querystring exists yet in url
            updatedContent = content.toString().replace(re3,"?update=" + time + "$1");
          }
          // Update url in formula with querystring param
          sheet.getRange(row+1, col+1).setFormula(updatedContent);
        }
      }
    }
  }

  // Done refresh; release the lock.
  lock.releaseLock();

  // Show last updated time on sheet somewhere
  sheet.getRange(7,2).setValue("Rates were last updated at " + now.toLocaleTimeString())
}
4
  • Your script is working great.. But I'm facing a problem. If i use =IMPORTXML("https://en.wikipedia.org/wiki/Moon_landing", "//a/@href") the script updates the data. But if I store the url in a cell, say B2 and use =IMPORTXML(B2, "//a/@href"), then the script fails to refresh. Could you please help?
    – ddas
    Feb 23, 2018 at 7:48
  • In my case started to work only after I added ?update= to the url.
    – Kintaro
    May 3, 2018 at 13:42
  • This worked GREAT for me. My only issue was a bad return on URLs I was using with importXML if that URL already had parameters in it. I have one URL with and one without parameters, and appending "?update=XXX" parameter to the end was throwing off the URL with other params. To get around this I added a dummy param to my URL withOUT params and changed the addition of "?update=XXX" to "&update=XXX", and all is well. It would be most useful if the code checked for pre-existing params and appended "update" with the appropriate "?" or "&", but I haven't taken the time to work it out yet. Apr 21, 2020 at 21:35
  • 2
    Thanks @MAKDesignLabs. I had the same problem. My feed URL already have parameters so running the update gave me the error "Imported XML content cannot be parsed.". But changing ?update to &update on line 27 fixed the issue for me. So I agree - would be great if the script checked for existing parameters and then used & instead of ?. Apart from that, it works well.
    – Josh Moore
    Jul 14, 2020 at 2:32
18

The Google Spreadsheet row update question and its answers refer to the "Old Sheets", which had different behaviour than the 2015 version of Google Sheets does. There is no automatic refresh of content with "New Sheets"; changes are only evaluated now in response to edits.

While Sheets no longer provides this capability natively, we can use a script to refresh the "import" formulas (IMPORTXML, IMPORTDATA, IMPORTHTML and IMPORTANGE).

Utility script

For periodic refresh of IMPORT formulas, set this function up as a time-driven trigger.

Caveats:

  • Import function Formula changes made to the spreadsheet by other scripts or users during the refresh period COULD BE OVERWRITTEN.
  • Overlapping refreshes might make your spreadsheet unstable. To mitigate that, the utility script uses a ScriptLock. This may conflict with other uses of that lock in your script.

 

/**
 * Go through all sheets in a spreadsheet, identify and remove all spreadsheet
 * import functions, then replace them a while later. This causes a "refresh"
 * of the "import" functions. For periodic refresh of these formulas, set this
 * function up as a time-based trigger.
 *
 * Caution: Formula changes made to the spreadsheet by other scripts or users
 * during the refresh period COULD BE OVERWRITTEN.
 *
 * From: https://stackoverflow.com/a/33875957/1677912
 */
function RefreshImports() {
  var lock = LockService.getScriptLock();
  if (!lock.tryLock(5000)) return;             // Wait up to 5s for previous refresh to end.
  // At this point, we are holding the lock.

  var id = "YOUR-SHEET-ID";
  var ss = SpreadsheetApp.openById(id);
  var sheets = ss.getSheets();

  for (var sheetNum=0; sheetNum<sheets.length; sheetNum++) {
    var sheet = sheets[sheetNum];
    var dataRange = sheet.getDataRange();
    var formulas = dataRange.getFormulas();
    var tempFormulas = [];
    for (var row=0; row<formulas.length; row++) {
      for (col=0; col<formulas[0].length; col++) {
        // Blank all formulas containing any "import" function
        // See https://regex101.com/r/bE7fJ6/2
        var re = /.*[^a-z0-9]import(?:xml|data|feed|html|range)\(.*/gi;
        if (formulas[row][col].search(re) !== -1 ) {
          tempFormulas.push({row:row+1,
                             col:col+1,
                             formula:formulas[row][col]});
          sheet.getRange(row+1, col+1).setFormula("");
        }
      }
    }

    // After a pause, replace the import functions
    Utilities.sleep(5000);
    for (var i=0; i<tempFormulas.length; i++) {
      var cell = tempFormulas[i];
      sheet.getRange( cell.row, cell.col ).setFormula(cell.formula)
    }

    // Done refresh; release the lock.
    lock.releaseLock();
  }
}
2
  • Thx for your scrip. I tried it immediately but it deleted all non import functions and data inside the sheet. Any suggestions why this might be the case? I appreciate your reply!
    – Carol.Kar
    Nov 23, 2015 at 17:06
  • 1
    I believe your releaseLock() call is inside the for loop
    – Gerbus
    May 24, 2017 at 0:09
17

To answer your question for an easy "trigger" to force the function to reload:

add an additional not used parameter to the url you are loading, while referencing a cell for the value of that parameter. Once you alter the content of that cell, the function reloads.

example:

importxml("http://www.example.com/?noop=" & $A$1,"...")

unfortunately you cannot put a date calculating function into the referenced cell, that throws an error that this is not allowed.

-1

You can also put each XML formula as a comment in the respective cells and record a macro to copy and paste it in the same cell. Later use the Scripts and then the Trigger functionality to schedule this macro. Google Sheets Comment Google Scripts Google Triggers

1
  • Do not add code as image.
    – Rubén
    Jan 6 at 1:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.