2

I'm doing some research I want to crawl youtube video links(and then get video IDs)

for example, if I type in keyword "obama" on youtube search bar then youtube.com will return the resulted page

http://www.youtube.com/results?search_query=htc&page=2 scroll the page down to the bottom we can see the page number 1 2 3 4 5 6 7 click on the page number from 2 to 3 to 4 to 5.... then we can reach the maximum page number :50

so I want to download all the pages from 1 to 50 there is a url trick,like

http://www.youtube.com/results?search_query=obama&page=5

however, if I use wget to download each page, sometimes, the downloading will be blocked or sometimes, when the page number is big, the returned page is not a abnormal page.

so a correct way is to manually click on the page number button and use the web browser to download each page, is it possible to automate the mannual procedures? I searched on INternet, some poeple mentioned chrome javascript extention, anyway, are there any ways to write some automation scripts to mimic the mannual operation? thanks!

1 Answer 1

1

Why not just use their API? Crawling is generally bad practise.

https://developers.google.com/youtube/2.0/developers_guide_protocol

https://developers.google.com/youtube/2.0/developers_guide_protocol_api_query_parameters

https://gdata.youtube.com/feeds/api/videos?
    q=football+-soccer
    &orderby=published
    &start-index=11
    &max-results=10
    &v=2
5
  • these api can't get all the video ids related to a search keyword Jan 3, 2013 at 17:43
  • If there is some limitation in the API it's probably there for a reason. Can you give examples?
    – Halcyon
    Jan 3, 2013 at 17:44
  • ah, it seems OK, but I all so want to collect the comments, like youtube.com/all_comments?v=FL7yD-0pqZg&page=6 , so I still hope to use automation scripts to mimic mannual operation Jan 3, 2013 at 17:54
  • This? developers.google.com/youtube/2.0/…
    – Halcyon
    Jan 3, 2013 at 17:55
  • thanks, but this only provides limited number of comments, I want to collect a lot of comments Jan 3, 2013 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.