3

I am new to Python and have learned basis of Web Scraping with bs4. here i tried to extract all the links of Youtube search results , it doesn't work like on other sites. i analyzed the search result html data and the links of search results were in anchor tag with id "video title" , but the tag doesn't appear in my bs4 parsed html document

from bs4 import BeautifulSoup as bs
import requests
name=input("Enter video name ")
url='https://www.youtube.com/results?search_query='+name
searched=requests.get(url)
soup=bs(searched.text,'html.parser')
aid=soup.find_all('a',{'id':'video-title'})
print(aid)

i expect the output contain all the search result. i have not learned other packages , i want to do this in bs4 if possible.

2
  • 1
    this is a pretty common question on StackOverflow (and in the tags you have) so there should be lots of info to help you with this.
    – QHarr
    May 9, 2019 at 7:23
  • the only answers out there is of selenium or api. selenium makes everything slow and pops up and i dont like it.is there any selenium like browser that works completely in background invisible.?? i was just asking if bs4 could do these things May 9, 2019 at 8:00

1 Answer 1

5

All this trouble to get YouTube search result data is just a waste of time and effort.

Why not try out these options

That being said, for the first 20 results, you can get the data from the JavaScript content in the source. The answer to that is given below.

After about 1 hr of making sense of the resulting json, it still fails for some queries.YouTube is a very complicated site. The response may vary based on location, browser, search query etc.

We are extracting the data from this script tag in the source. enter image description here

Code:

from bs4 import BeautifulSoup as bs
import requests
import re
import json
headers={
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}
name=input("Enter video name: ")
url='https://www.youtube.com/results?search_query=hello'+name
searched=requests.get(url,headers=headers)
soup=bs(searched.text,'html.parser')
aid=soup.find('script',string=re.compile('ytInitialData'))
extracted_josn_text=aid.text.split(';')[0].replace('window["ytInitialData"] =','').strip()
video_results=json.loads(extracted_josn_text)
#print(item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][1])
item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][0]["itemSectionRenderer"]["contents"]

for item in item_section:
    try:
        video_info=item["videoRenderer"]
        title=video_info["title"]["simpleText"]
        url=video_info["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"]
        print('Title:',title)
        print('Url:',url, end="\n----------\n")
    except KeyError:
            pass

Output:

Enter video name: hello
Title: New Punjabi Songs 2017-Hello Hello(Ful Song)-Prince Narula-Yuvika Chaudhary-Latest Punjabi Song 2017
Url: /watch?v=mv326-zVpAQ
----------
Title: Alan Walker - The Spectre
Url: /watch?v=wJnBTPUQS5A
----------
Title: Hello Hello (Full HD) - Rajvir Jawanda | MixSingh | Josan Bros | New Punjabi Songs 2018
Url: /watch?v=xydupjQSj44
----------
Title: Bachchan - Hello Hello - Kannada Movie Full Song Video | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=oLMMgoug4Uk
----------
Title: Hello Hello latest 2017 16 june punjabi song
Url: /watch?v=MqCSsPXw8QU
----------
Title: 👋  Hello Hello 👋  | + More Kids Songs | Super Simple Songs
Url: /watch?v=saDkICxEdgY
----------
Title: 'Gallan Goodiyaan' Full VIDEO Song | Dil Dhadakne Do | T-Series
Url: /watch?v=jCEdTq3j-0U
----------
Title: Hello Hello Gippy Grewal Feat. Dr. Zeus Full Song HD | Latest Punjabi Song 2013
Url: /watch?v=IRW2O4QZhgs
----------
Title: Hello Hello | Pataakha | Malaika Arora | Vishal Bhardwaj & Rekha Bhardwaj | Gulzar | Ganesh Acharya
Url: /watch?v=RxBAitQLSLA
----------
Title: Hello Hello (Lyrical Audio) Prince Narula ft. Yuvika Chaudhary | Punjabi Lyrical Audio 2017 | WHM
Url: /watch?v=v8VIsIvhDoQ
----------
Title: Hello Hello Full Video Song || Bhale Bhale Magadivoi || Nani, Lavanya Tripathi
Url: /watch?v=y3FI02OO_kU
----------
Title: Hello hello gaad bahe dhufee na egaa  (new comedy hhhhhh)
Url: /watch?v=DuRrcTo4rgg
----------
Title: Proper Patola - Official Video | Namaste England | Arjun | Parineeti | Badshah | Diljit | Aastha
Url: /watch?v=YmXJp4RtBCM
----------
Title: Official Video: Nikle Currant Song | Jassi Gill | Neha Kakkar | Sukh-E Muzical Doctorz | Jaani
Url: /watch?v=uBaqgt5V0mU
----------
Title: Insane (Full Song)  Sukhe - Jaani - Arvindr Khaira - White Hill Music - Latest Punjabi Song 2018
Url: /watch?v=mKpPhVVF8So
----------
Title: Radha bole HELLO HELLO-cartoon song mix with step up 2
Url: /watch?v=TFCTgNCzrck
----------
Title: Hello Song | CoCoMelon Nursery Rhymes & Kids Songs
Url: /watch?v=fxVMqaViVaA
----------
Title: Bachchan - Hello Hello Unplugged Version | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=lvH3kTGJeEQ
----------
Title: Hello Hello! Can You Clap Your Hands? | Original  Kids Song | Super Simple Songs
Url: /watch?v=fN1Cyr0ZK9M
----------

One last thing that you can try out is to emulate the API used by youtube itself

ie. POST request to

https://www.youtube.com/results?search_query=yoursearchtext

It has a lot of cookie and session values being sent as parameters. You may need to emulate all of them. You may need to use Requests session objects to do that.

2
  • thanks mate, you actually did it without using selenium or API just as i wanted, but this makes things difficult and maybe not possible in every dynamic sites , so i will be using Selenium even though it is RAM and time consuming..thanks again May 9, 2019 at 15:10
  • very nice, how can i use this for a .html or .php website? @Bitto
    – Sebastian
    Nov 23, 2022 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.