0

Nowadays many websites contain some content loaded by ajax(e.g,comments in some video websites). Normally we can't crawl these data and what we get is just some js source code. So here is the question: in what ways can we execute the javascript code after we get the html response and get to the final page we want?

I know that HtmlUnit has the ability to execute background js,yet some many bugs and errors are there. Are there any else tools can help me with it?

Some people tell me that I can crawl the ajax request url, analyze its parameters and send request again so as to gain the data. If things can't work out according to the way I mention above, can anyone tell me how to extract the ajax url and send the request in correct format?

By the way,if the language is java,it would be the best

3
  • Modify the browser/write an add-on to the browser to do the job for you. Is the hard way...
    – nhahtdh
    Aug 11, 2012 at 13:11
  • Using the webkit core to obtain the final dom tree may technically be feasible, but it's too hard=.=are there any other solutions? Aug 12, 2012 at 2:19
  • I think so. Can you use selenium? Apr 17, 2014 at 19:10

2 Answers 2

0

Yes, Netwoof can crawl Ajax easily. Its API and bot builder let you do it without a line of code.

0

Thats the great thing about HTTP you don't even need java. My goto tool for debugging AJAX is the chrome extension Postman. I start by looking at the request in the chrome debugger and identifying the salient bits(url or form encoded params etc.)

Then it can be as simple as opening a tab and launch requests at the server with Postman. As long as its all in the same browser context all of your cookies(for authentication, etc.) will be shipped along too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.