16

Can you use Jsoup to submit a search to Google, but instead of sending your request via "Google Search" use "I'm Feeling Lucky"? I would like to capture the name of the site that would be returned.

I see lots of examples of submitting forms, but never a way to specify a specific button to perform the search or form submission.

If Jsoup won't work, what would?

4
  • Do you mean JSONP? Google's search API is code.google.com/apis/customsearch/v1/overview.html
    – ceejayoz
    Sep 22, 2011 at 2:22
  • @Ceejayoz: put your mouse above the [jsoup] tag below the question until a popbox shows and then click therin the info link to learn about it.
    – BalusC
    Sep 23, 2011 at 1:51
  • @BalusC I'd say it was a fair enough request for clarification, considering the content of the question and the all-caps, one-letter-off JSOUP.
    – ceejayoz
    Sep 23, 2011 at 1:56
  • @Ceejayoz: Agree that. Jsoup shouldn't be in all-caps.
    – BalusC
    Sep 23, 2011 at 1:57

3 Answers 3

13

According to the HTML source of http://google.com the "I am feeling lucky" button has a name of btnI:

<input value="I'm Feeling Lucky" name="btnI" type="submit" onclick="..." />

So, just adding the btnI parameter to the query string should do (the value doesn't matter):

http://www.google.com/search?hl=en&btnI=1&q=your+search+term

So, this Jsoup should do:

String url = "http://www.google.com/search?hl=en&btnI=1&q=balusc";
Document document = Jsoup.connect(url).get();
System.out.println(document.title());

However, this gave a 403 (Forbidden) error.

Exception in thread "main" java.io.IOException: 403 error loading URL http://www.google.com/search?hl=en&btnI=1&q=balusc
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:387)
    at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:364)
    at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:143)
    at org.jsoup.helper.HttpConnection.get(HttpConnection.java:132)
    at test.Test.main(Test.java:17)

Perhaps Google was sniffing the user agent and discovering it to be Java. So, I changed it:

String url = "http://www.google.com/search?hl=en&btnI=1&q=balusc";
Document document = Jsoup.connect(url).userAgent("Mozilla").get();
System.out.println(document.title());

This yields (as expected):

The BalusC Code

The 403 is however an indication that Google isn't necessarily happy with bots like that. You might get (temporarily) IP-banned when you do this too often.

1
  • Thanks BalusC. That answered the question perfectly. I see how that URL request is unique to Google's site. I was trying to figure out if you can literally program Jsoup to click a specific button using Google as an example. I better clarify what I am trying to do in my other question on this site. That question hasn't been answered in a month and got frustrated so asked a different question in hopes it would help me with the first. Unfortunately my plan backfired and I apologize for taking your time to answer a question that doesn't solve my problem.
    – Brian
    Sep 23, 2011 at 3:47
2

I'd try HtmlUnit for navigating trough a site, and JSOUP for scraping

1
  • so how can htmlUnit and jSoup be used in combination? edit: got it.. jSoup.parse can parse the html string given by htmlUnit
    – tObi
    Jan 5, 2015 at 21:29
0

Yes it can, if you are able to figure out how Google search queries are made. But this is not allowed by Google, even if you would success with that. You should use their official API to make automated search queries.

http://code.google.com/intl/en-US/apis/customsearch/v1/overview.html

1
  • Thanks for the response. I was only using google as an example however. I have a site that I am trying to use JSOUP with to login and get some information back. I don't think I am actually logging in. I am trying to find out how to simulate pressing the "login" or "submit" button generically to make sure I am actually logging in to the site. Sorry for the confusion.
    – Brian
    Sep 22, 2011 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.