I have the list of the URL which i want to use to fetch the data. I can do for one url like this:
URL list = ['https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/4722/',
'https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/6318/'
from bs4 import BeautifulSoup
import requests
url = "https://www2.daad.de/deutschland/studienangebote/international-programmes/en/detail/4479/"
page = requests.get(url)
soup = BeautifulSoup(page.text, "html.parser")
info = soup.find_all("dl", {'class':'c-description-list c-description-list--striped'})
comp_info = pd.DataFrame()
cleaned_id_text = []
for i in info[0].find_all('dt'):
cleaned_id_text.append(i.text)
cleaned_id__attrb_text = []
for i in info[0].find_all('dd'):
cleaned_id__attrb_text.append(i.text)
df = pd.DataFrame([cleaned_id__attrb_text], column = cleaned_id_text)
But i am not sure how to do this for several urls and append the data into dataframe. Each URL describe the course decription so i want to create the dataframe which will include all the data from the all URL's... And would be nice if i can able to add URL also as separate column in dataframe.