347

Google Web Search API has been deprecated and replaced with Custom Search API (see http://code.google.com/apis/websearch/).

I wanted to search the whole web but it looks like with the new API only custom sites can be searched.

Is there a way to search the whole web programmatically? I was able to query the old API using JSON from a Java program.

2
  • 3
    There is also SerpApi. That's a solid solution for Google search and other engines.
    – Hartator
    Commented Jun 9, 2021 at 16:00
  • 1
    serphouse.com is a great solution to get data using API.
    – Mehul V.
    Commented Jun 17, 2022 at 9:16

10 Answers 10

503

Yes, Google Custom Search has now replaced the old Search API, but you can still use Google Custom Search to search the entire web, although the steps are not obvious from the Custom Search setup.

To create a Google Custom Search engine that searches the entire web:

  1. From the Google Custom Search homepage ( http://www.google.com/cse/ ), click Create a Custom Search Engine.
  2. Type a name and description for your search engine.
  3. Under Define your search engine, in the Sites to Search box, enter at least one valid URL (For now, just put www.anyurl.com to get past this screen. More on this later ).
  4. Select the CSE edition you want and accept the Terms of Service, then click Next. Select the layout option you want, and then click Next.
  5. Click any of the links under the Next steps section to navigate to your Control panel.
  6. In the left-hand menu, under Control Panel, click Basics.
  7. In the Search Preferences section, select Search the entire web but emphasize included sites.
  8. Click Save Changes.
  9. In the left-hand menu, under Control Panel, click Sites.
  10. Delete the site you entered during the initial setup process.

Now your custom search engine will search the entire web.

Pricing

  • Google Custom Search gives you 100 queries per day for free.
  • After that you pay $5 per 1000 queries.
  • There is a maximum of 10,000 queries per day.

Source: https://developers.google.com/custom-search/json-api/v1/overview#Pricing


  • The search quality is much lower than normal Google search (no synonyms, "intelligence" etc.)
  • It seems that Google is even planning to shut down this service completely.
27
  • 28
    Thanks for this. Hopefully this is a valid procedure and not a loophole waiting to be plugged by Google!
    – Mazatec
    Commented Jul 7, 2012 at 14:02
  • 9
    Thank you! This is possibly the only answer on the Internet that addressed my question. It's mind boggling why Google would end direct API support for their core service.
    – spamguy
    Commented Jan 24, 2013 at 4:16
  • 7
    but how to use it with json ?
    – nawara
    Commented Apr 19, 2013 at 19:32
  • 71
    WARNING: we did development using the free version, but to upgrade to the paid version (to do more than 100 searches), google forces you to turn off the "search the entire web but emphasize included sites" Commented Aug 11, 2015 at 14:50
  • 5
    "On April 1, 2017, Google will discontinue sales of the Google Site Search. All new purchases and renewals must take place before this date. The product will be completely shut down by April 1, 2018."
    – Gajus
    Commented Mar 7, 2017 at 15:52
59

Google Custom Search (as advocated in the top rated answers) works well, but is very expensive, compared to its competitors (below) or compared to other Google API's. It has a small free tier (100 queries/day) and a very high price of $5 per 1000 query.

They offer the option to upgrade to Site Search, which has slightly better prices, but that is meant for searching one site (your own), so it is really something quite different - not an upgrade.

The main alternatives seem to be:

Bing Search API
https://datamarket.azure.com/dataset/5BA839F1-12CE-4CCE-BF57-A49D98D29A44
Which has a free tier of 5000q/month, and prices starting at 5 query per penny, and no hard limit.

UPDATE: At the end of 2016 this API was shutdown in favour of its Azure counterpart "Cognitive Services Bing Search API":
https://azure.microsoft.com/en-us/services/cognitive-services/search/

See here for a pricing chart, which starts at US$3/m for 1,000 transactions. Unless I'm missing something it is quite expensive.

Yahoo BOSS Search API
UPDATE: Was discontinued on March 31, 2016. http://developer.yahoo.com/boss/search/
With prices starting at about 12 queries/penny for whole web searches.

And some I haven't heard of before:

http://www.gigablast.com/searchfeed.html

http://www.faroo.com/hp/api/api.html

http://www.commoncrawl.org/

http://www.entireweb.com/search_api/implementation/
[discontinued - as pointed out below]

There is a bit of discussion of some of these on this SO post.
[got closed for being off-topic and is now gone]

7
40

You could just send them through like a browser does, and then parse the html, that is what I have always done, even for things like Youtube.

33
  • 68
    i'm told that googles terms of service forbid spidering... Commented Nov 21, 2010 at 5:06
  • 56
    From the TOS: "You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers)..."
    – ændrük
    Commented Mar 6, 2011 at 17:53
  • 21
    "Violate the terms of service with a service provider" is never a good advice. Parsing webpages is something that breaks from one day to the next without warning, this is awful advice - that's the reason it was downvotes more that it was upvoted.
    – WhyNotHugo
    Commented Dec 27, 2012 at 16:35
  • 166
    Yes it breaks the terms of service but personally I wouldn't worry about that. Google can handle a little bit of scraping, after all they have made a fortune scraping other peoples sites.
    – jimbo2087
    Commented Feb 13, 2013 at 0:29
  • 119
    Come on people. Don't be so naive. Google cannot force that ToS down your throat. In order to violate a ToS you must first agree with it (in writing, or by clicking a button like 'Yes, I accept the terms'). Think at this: I put a ToS on my web page that every person that visits that page has to give me $10000. Can I enforce this ToS on my visitors? Will the have to may me immediately.
    – IceCold
    Commented Nov 25, 2013 at 22:13
26

Here is an option at the bottom of the Custom Search Control Panel: "Sites to search", you can choose "Search the entire web but emphasize included sites"

Custom Search Control Panel - Sites to search

5
  • 1
    does it still work for you?
    – Dejell
    Commented Feb 11, 2015 at 20:10
  • Yep, it still works.
    – user3477312
    Commented Mar 5, 2015 at 22:36
  • 5
    Google forces you to turn that option off when you upgrade to paid search. And free has a limit of 100 searches. Commented Aug 11, 2015 at 14:51
  • 1
    @Yishu, Why does the page https://support.google.com/customsearch/answer/141877?hl=en states "You cannot configure Google Site Search to search the entire web"?
    – Pacerier
    Commented Sep 18, 2015 at 4:28
  • @Pacerier, I have no idea about it. Maybe the policy have changed?
    – Yishu Fang
    Commented Sep 18, 2015 at 14:43
14

Faroo has a free Web Search API

7
  • Their results seam limited but a good starting point.
    – afro360
    Commented Oct 20, 2014 at 8:02
  • @Jack, Not heard of this before. Where do they get their search results from?
    – Pacerier
    Commented Sep 18, 2015 at 4:24
  • 7
    Possible deal breaker for Faroo is that your API key is restricted to the IP address you specify during registration. Commented Nov 3, 2015 at 4:40
  • 6
    Are these guys still operational? I've requested API keys and heard nothing. Commented Feb 1, 2017 at 14:23
  • Page has a "Coming Soon" banner now...
    – Miriam
    Commented Aug 14, 2020 at 8:50
7

I have just come across this from Common Crawl.

http://www.commoncrawl.org/

Might be the answer we are all looking for!!

5
  • 3
    It has a limited index, refreshed about once a year. And it is finally quite expensive, as you have to plug into Amazon S3. Commented Oct 23, 2013 at 11:25
  • @GuillaumeLebourgeois, Expensive? I don't think that's true. It's a nonprofit. The entire 102 TB of data is free for download.
    – Pacerier
    Commented Aug 23, 2016 at 19:01
  • The cost is for connecting to AWS where you can access this. If you are a student, you are eligible for their free tier, but there could still be transfer costs etc; and if you are not in the free tier, there are running costs.
    – tripleee
    Commented May 16, 2018 at 9:07
  • 1
    Looks like common crawl is updated monthly now
    – Jack
    Commented Mar 18, 2019 at 19:56
  • At least currently (february 2022) the data can be downloaded from S3 for free. HTTP-links can be found on the commoncrawl website.
    – x-ray
    Commented Feb 20, 2022 at 11:48
4

There's a note on top of the docs:

Note: The Google Web Search API has been officially deprecated as of November 1, 2010. It will continue to work as per our deprecation policy, but the number of requests you may make per day will be limited. Therefore, we encourage you to move to the new Custom Search API.

The deprecation policy says that they will continue to run the API for 3 years. So if you already have an application that uses the old API, you don't have to rush to change things just yet. If you're writing a new application, use the Custom Search API. See my answer here for how to do this in Python, but the idea's the same for any language.

3
  • 8
    And it's not free.... "$5 per 1000 queries"... very much not free!
    – UpTheCreek
    Commented Jun 23, 2012 at 18:34
  • 1
    This answer is now obsolete as the three years are up and 2014/09/29 has passed.
    – nanofarad
    Commented Oct 22, 2014 at 23:42
  • 1
    CustomeSearchAPI is not in all the websites - it's for the user websites
    – Dejell
    Commented Feb 11, 2015 at 19:06
3

There's a free Java API called JFreeWebSearch which uses the already mentioned Faroo: http://www.ke.tu-darmstadt.de/resources/jfreewebsearch

1
1

Gigablast offers a cheap web search API: http://www.gigablast.com/searchfeed.html

0

You can create "everywhere" custom search engine right from the Google Custom Search homepage ( http://www.google.com/cse/ ). You should just click 'advanced', during adding new engine. There you can provide Schema.org site type. 'Thing' is most generic type, which covers all the web.

1
  • I didn't get it. Does it work for you?
    – Dejell
    Commented Feb 11, 2015 at 20:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.