7

What I'm trying to do

For a list of websites, I want to get the pages indexed by year, if they were archived at any point that year. So if I'm looking at example1.com and example2.com, I want to be able to get:

2010: example1.com, example2.com (the html from these archived pages)
2011: example1.com (example2.com, say, was not archived in 2011)
2012: example2.com
2013: example1.com, example2.com

and so on.

Question

Is this possible to get using the Wayback Machine API? I looked at their API listing and it didn't seem like I could do what I was trying to do. Maybe I'm missing something, but it seems like a fairly plausible use case. Any other suggestions?

2 Answers 2

18

They key thing to understand about the Wayback Machine APIs is that there are (from what I can tell) three different ways to work with them.

Wayback Availability JSON API

The first is the API which is documented near the top of the Wayback Machine API page you already mentioned.

That API gives the date-wise nearest result for an archive on a given page. So you can check the Wayback Machine for copies of the Google homepage archived around New Year's Day like so:

http://archive.org/wayback/available?url=google.com&timestamp=20080101 http://archive.org/wayback/available?url=google.com&timestamp=20090101 http://archive.org/wayback/available?url=google.com&timestamp=20100101 etc..

Using the information returned in those URLs, you can easily download the content programmatically.

Wayback CDX Server API

Next we have the Wayback Machine CDX Server API which reveals a much richer series of interfaces. Most notably, you can quickly download every snapshot of a URL that you are interested in:

http://web.archive.org/cdx/search/cdx?url=www.fredtrotter.com

Memento API

Lastly we have the deep and mysterious resource that is the Wayback Machine Memento API. That link is to a blog post about the functionality, but from what I can garner, this is about working with the Wayback Machine at a protocol level, where the Mememnto Protocol is a well-thought out version of the way an archive site should operate.

Final thoughts

In all cases, please be gentle and respectful with your scripting. The Wayback Machine API does not currently require credentials, which is a very generous and open posture in general keeping with the Internet Archive's role as a "Wonder of the Virtual World". So do not abuse it, because that is how we ensure that we have nice things.

Thanks to Greg, and the rest of the Wayback Machine team, for the excellent work you do to keep the Internet a source of personal freedom and expression.

8

Our CDX API allows you to make 2 separate calls get a list of all captures for the url or domain example1.com and the url or domain example2.com. You can then produce whatever summary you like.

3
  • This answer is somewhat confusing. On whose behalf was it written? (I.e. who is "our" in this context?) Also, any chance you could state the URL for the API that you referred to, to avoid ambiguity and to eliminate needless web searches? (Cf. "Brevity is acceptable, but fuller explanations are better.")
    – user82216
    Nov 22, 2018 at 12:19
  • 1
    I was working at the Internet Archive on the Wayback when I wrote my answer, and the API I refer to is linked in the question and the other answer. Here it is again: archive.org/help/wayback_api.php (scroll to the bottom to the section labeled "Wayback CDX Server API") Nov 23, 2018 at 17:09
  • Upvoted. Thanks for clarifying, and thanks for working on Wayback :)
    – user82216
    Nov 27, 2018 at 1:21

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.