24

How can I make requests to other server(s) (i.e. get a page from any desired server) with a JavaScript within the user's browser? There are limitations in place to prevent this for methods like XMLHttpRequest, are there ways to bypass them or other methods?

That is a general question, specifically I want to check a series of random websites and see if they contain a certain element, so I need the HTML content of a website without downloading any additional files; all that in a JavaScript file, without any forwarding or proxy mechanism on a server.

(Note: one way is using Greasemonkey and its GM_xmlhttpRequest.)

1
  • 3
    You'll need server side code to do this, cross-domain requests are a security hole and not only will it be prohibitively difficult to get around this situation you really shouldn't be doing it at all. Use a simple backend like PHP or even something like Sinatra.
    – rfunduk
    Feb 23, 2009 at 17:31

8 Answers 8

25

You should check out jQuery. It has a rich base of AJAX functionality that can give you the power to do all of this. You can load in an external page, and parse it's HTML content with intuitive CSS-like selectors.

An example using $.get();

$.get("anotherPage.html", {}, function(results){
  alert(results); // will show the HTML from anotherPage.html
  alert($(results).find("div.scores").html()); // show "scores" div in results
});

For external domains I've had to author a local PHP script that will act as a middle-man. jQuery will call the local PHP script passing in another server's URL as an argument, the local PHP script will gather the data, and jQuery will read the data from the local PHP script.

$.get("middleman.php", {"site":"http://www.google.com"}, function(results){
  alert(results); // middleman gives Google's HTML to jQuery
});

Giving middleman.php something along the lines of

<?php

  // Do not use as-is, this is only an example.
  // $_GET["site"] set by jQuery as "http://www.google.com"
  print file_get_contents($_GET["site"]);

?>
9
  • @mmattax, check my last paragraph.
    – Sampson
    Feb 23, 2009 at 15:54
  • You mentioned "in some cases" - what are the other cases? (Note: I've updated the question to exclude middle-men server scripts.)
    – user46665
    Feb 23, 2009 at 16:21
  • 2
    @petersidor, Some websites will not allow javascript from another domain have any access. So you'll need to do incorporate file_get_contents() or some other similar method to get their HTML.
    – Sampson
    Feb 23, 2009 at 16:59
  • @Jonathan: correction, no websites will allow javascript from another [top-level] domain to have access. Feb 23, 2009 at 17:02
  • This looks like an acceptable solution for the general problem, so I have voted this up; unfortunately it doesn't quite cover my specific issue, therefore I can't mark it as the final solution.
    – user46665
    Feb 24, 2009 at 13:21
5

update 2018:

You can only access cross domain with the following 4 condition

  • in response header has Access-Control-Allow-Origin: *

Demo

$.ajax({
  url: 'https://api.myjson.com/bins/bq6eu',
  success: function(response){
    console.log(response.string);
  },
  error: function(response){
    console.log('server error');
  }
})
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>

  • use server as bridge or proxy to the target

Demo:

$.ajax({
  url: 'https://cors-anywhere.herokuapp.com/http://whatismyip.akamai.com/',
  success: function(response){
    console.log('server IP: ' + response);
  },
  error: function(response){
    console.log('bridge server error');
  }
})
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>

  • using browser addon to enable Allow-Control-Allow-Origin: *
  • disable browser web security

Chrome

chrome.exe --args --disable-web-security

Firefox

about:config -> security.fileuri.strict_origin_policy -> false

end


noob old answer 2011

$.get(); can get data from jsbin.com but i don't know why it can't get data from another site like google.com

$.get('http://jsbin.com/ufotu5', {},
  function(results){  alert(results); 
});

demo: http://jsfiddle.net/Xj234/ tested with firefox, chrome and safari.

0
3

Write a proxy script that forwards along the http request from your domain, this will bypass the XMLHttpRequest restrictions.

If your using PHP, simply use cURL to request and read the page, then simply spit out the html as if it was from you domain.

3

This is rather easy... if you know the 'secret' trick almost nobody shares..

It's called Yahoo yql...

So in order to regain 'power to the user' (and returning to the convenient mantra: 'never accept no'), just use http://query.yahooapis.com/ (instead of a php? proxy serverside script).
jQuery would not be strictly needed.

EXAMPLE 1:
Using the SQL-like command:

select * from html 
where url="http://stackoverflow.com" 
and xpath='//div/h3/a'

The following link will scrape SO for the newest questions (bypassing cross-domain security bull$#!7):
http://query.yahooapis.com/v1/public/yql?q=select%20title%20from%20html%20where%20url%3D%22http%3A%2F%2Fstackoverflow.com%22%20and%0A%20%20%20%20%20%20xpath%3D%27%2F%2Fdiv%2Fh3%2Fa%27%0A%20%20%20%20&format=json&callback=cbfunc

As you can see this will return a JSON array (one can also choose xml) and calling the callback-function: cbfunc.

Indeed, as a 'bonus' you also save a kitten every time you did not need to regex data out of 'tag-soup'.

Do you hear your little mad scientist inside yourself starting to giggle?

Then see this answer for more info (and don't forget it's comments for more examples).

Good Luck!

1
1

You can also use a iframe to emulate an ajax request. This saves you the mess of having to code a Backend solution for a Frontend problem. Here is an example:

function setUploadEvent(typeComponet){
       var eventType = "";
       var iframe = document.getElementById("iframeId");
       try{
           /* for Mozilla / Opera9 */
           if (/(?!.*?compatible|.*?webkit)^mozilla|opera/i.test(navigator.userAgent)) {
                eventType = "onload";
           }else{
           /* IE  */
                eventType = "onreadystatechange";
           }

           iframe[eventType] = function(){
                var doc = iframe.contentDocument || iframe.contentWindow.document;
                var response = doc.body.innerHTML; /* or what ever content you are looking for */
             }
           }
       catch(e){
           alert("Error loading content")}
       } 

That should do the trick. Please note that the Browser detection line is not the cleanest, you should absolutely use the ones provided in all the most common JS frameworks (Prototype, JQuery, etc)

2
  • Now, this looks more like it - will try it out and report back, have voted it up already. It may appear crude, but sometimes, simpler is really better. :)
    – user46665
    Feb 24, 2009 at 13:23
  • Tried out; unfortunately the browsers don't grant access to the properties of an iframe with a website from a different server. So it's basically the same problem as with XMLHttpRequest and it seems my problem is, as such, impossible to do. :|
    – user46665
    Mar 2, 2009 at 10:08
0

You will need to write a proxy on the server to do this. And all requests will be to your server and then your server will load html and send it back to client. And there are no good way to implement this via javascript only.
jQuery contains functionality to load JSON data or external scripts using XmlHttpRequest but this functionality can not be used for html pages. Also you may check this thread of jQuery mailing list.

0

<script language="JavaScript" type="text/javascript" src="http://www.example.com/hello.js"></script>

You add the data into hello.js in as an array, JSON or similar. Example: var daysInMonth = new Array(31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31);

Getting a JavaScript from another server doesn't much simpler.. :-)

3
  • :) Wanted was a way to get data from another server with a JavaScript script, but it was funny. :)
    – user46665
    Jan 21, 2010 at 12:24
  • Please see my update on how to add data to the external JavaScript. Jan 22, 2010 at 14:57
  • Sorry, my question was unclear, I have updated it - the purpose is to get data - like a whole HTML page - from a server, and access it with a JavaScript running in the user's browser.
    – user46665
    Jan 25, 2010 at 10:57
0

Thanks a lot, this is really a good trick. I did in this way:

test.html

<!DOCTYPE html>
<html>
<head>
<script>
function loadXMLDoc()
{
if (window.XMLHttpRequest)
  {// code for IE7+, Firefox, Chrome, Opera, Safari
  xmlhttp=new XMLHttpRequest();
  }
else
  {// code for IE6, IE5
  xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
xmlhttp.onreadystatechange=function()
  {
  if (xmlhttp.readyState==4 && xmlhttp.status==200)
    {
    document.getElementById("myDiv").innerHTML=xmlhttp.responseText;
    }
  }
xmlhttp.open("GET","sp.php",true);
xmlhttp.send();
}
</script>
</head>
<body>

<h2>Using the XMLHttpRequest object</h2>
<div id="myDiv"></div>
<button type="button" onclick="loadXMLDoc()">Change Content</button>

</body>
</html>

sp.php

<?php
  print file_get_contents("http://your.url.com/you-can-use-cross-domain");
?>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.