To elaborate my comments:
You need to load the content in an iframe if you want the dynamic parts to be rendered.
This is necessary, because XMLHttpRequest
just fetches the source code of the page, it does not run any embedded scripts.
Beware of the X-Frame-Options header,
The next solution only works if the X-Frame-Options
is not send with the response. When this response header is set, the page cannot be embedded in a frame. You will have to remove this header, as shown in this answer.
and make sure that you set the sandbox attribute on the iframe to avoid frame busting.
You're going to insert an <iframe>
with this-party content in a document. Without any counter-measures, this third-party page can replace the top document using if (top !== window) {top.location = location.href;}
, also known as frame busting.
By the look of it, your script is used as a content script. Insert a frame, and use the same content script within this frame to get the desired information.
manifest.json
...
"content_scripts": [{
"js": ["contentscript.js"],
"all_frames": true,
"matches": ["*://*.example.com/*"]
}],
...
contentscript.js
function getPoints() {
var iframe = document.createElement('iframe');
iframe.src = 'http://www.example.com/';
// Remove iframe when the content has (not) loaded
iframe.onerror =
iframe.onload = function() {
setTimeout(function() {
iframe.parentNode.removeChild(iframe);
}, 5000);
};
// Put security restrictions on the iframe
iframe.sandbox = 'allow-scripts';
// Make frame invisible
iframe.style.height = '1px';
iframe.style.width = '1px';
iframe.style.position = 'fixed';
iframe.style.top = '-9px';
iframe.style.left = '-9px';
// Insert iframe in the document, and load its content
document.body.appendChild(iframe);
}
function sendPoints(attempts) {
var points = document.getElementsByClassName('notification-count hidden');
if (points.length > 1) { // <-- Example, based on your question
// Desired element found, send the message to the background page
chrome.runtime.sendMessage({
points: points[1].textContent
});
} else if (--attempts > 0) {
// Element not found, and some attempts left. Try again soon.
setTimeout(sendPoints, 250, attempts);
}
}
sendPoints(40);
This method comes with a huge latency: First the page needs to be loaded, then all scripts and resources will be loaded and parsed. If possible, try to extract the "dynamic" information with other methods. Try to reverse-engineer the page you're scraping, and try to get the desired value through other means, e.g. by using XMLHttpRequest
to get data from the script that generates the element.