1

I am creating a chrome extension that monitors a given website. I am using XMLHttpRequest to fetch the DOM of that site, which will then parsed and the required data will be fetched.

Problem is that XMLHttpRequest does not return the fully loaded DOM with the dynamic content (for example images, dynamic text etc...).

I googled this issue and it seems there is no fix, but I might be incorrect.

Is there a way for XHR to return the fully loaded DOM.

Here is my code

window.onload = function () { chrome.runtime.sendMessage({points: document.getElementsByClassName('notification-count hidden')[1].outerText}); } //Works fine, but I need the same result with XHR

function getPoints()
{
    var link = 'example'; //placeholder

    var xhr = new XMLHttpRequest();
    xhr.open('GET', link, true);
    xhr.responseType = 'text';
    xhr.onload = function(e) {
    if (xhr.readyState== 4 && xhr.status == 200) 
    {
        DOM = xhr.response;
        console.log(DOM); //Returns the DOM of the site but missing dynamic content
    }
};

xhr.send();
}

Thank you in advance, and sorry if my question seems trivial

2
  • 2
    You need to load the content in an iframe if you want the dynamic parts to be rendered. Beware of the X-Frame-Options header, and make sure that you set the sandbox attribute on the iframe to avoid frame busting.
    – Rob W
    Commented Oct 11, 2013 at 22:28
  • @RobW I apologize but I am still a javascript newbie. Can you please elaborate on your answer? To clarify all I want is an element that is found in the DOM. Thanks for answering :)
    – r3x
    Commented Oct 12, 2013 at 0:06

1 Answer 1

2

To elaborate my comments:

You need to load the content in an iframe if you want the dynamic parts to be rendered.

This is necessary, because XMLHttpRequest just fetches the source code of the page, it does not run any embedded scripts.

Beware of the X-Frame-Options header,

The next solution only works if the X-Frame-Options is not send with the response. When this response header is set, the page cannot be embedded in a frame. You will have to remove this header, as shown in this answer.

and make sure that you set the sandbox attribute on the iframe to avoid frame busting.

You're going to insert an <iframe> with this-party content in a document. Without any counter-measures, this third-party page can replace the top document using if (top !== window) {top.location = location.href;}, also known as frame busting.

By the look of it, your script is used as a content script. Insert a frame, and use the same content script within this frame to get the desired information.

manifest.json

...
"content_scripts": [{
    "js": ["contentscript.js"],
    "all_frames": true,
    "matches": ["*://*.example.com/*"]
}],
...

contentscript.js

function getPoints() {
    var iframe = document.createElement('iframe');
    iframe.src = 'http://www.example.com/';
    // Remove iframe when the content has (not) loaded
    iframe.onerror =
    iframe.onload = function() {
        setTimeout(function() {
            iframe.parentNode.removeChild(iframe);
        }, 5000);
    };
    // Put security restrictions on the iframe
    iframe.sandbox = 'allow-scripts';
    // Make frame invisible
    iframe.style.height = '1px';
    iframe.style.width = '1px';
    iframe.style.position = 'fixed';
    iframe.style.top = '-9px';
    iframe.style.left = '-9px';
    // Insert iframe in the document, and load its content
    document.body.appendChild(iframe);
}
function sendPoints(attempts) {
    var points = document.getElementsByClassName('notification-count hidden');
    if (points.length > 1) { // <-- Example, based on your question
        // Desired element found, send the message to the background page
        chrome.runtime.sendMessage({
            points: points[1].textContent
        });
    } else if (--attempts > 0) {
        // Element not found, and some attempts left. Try again soon.
        setTimeout(sendPoints, 250, attempts);
    }
}
sendPoints(40);

This method comes with a huge latency: First the page needs to be loaded, then all scripts and resources will be loaded and parsed. If possible, try to extract the "dynamic" information with other methods. Try to reverse-engineer the page you're scraping, and try to get the desired value through other means, e.g. by using XMLHttpRequest to get data from the script that generates the element.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.