1

I am trying to scrape this website enter link description here. we have almost ten different opportunities on each page. each one has its own title and details. I want to get all this information. I have written a python code that can locate other required tags and information but I can't locate the paragraphs that contain a description in it.

here is my code.

    base_url = "https://www.enabel.be/content/enabel-tenders"
    driver.get(base_url)
    WebDriverWait(driver , 10).until(EC.visibility_of_element_located(
            (By.XPATH , "//*[@id='block-views-tenders-block']/div/div/div[@class='view-content']/div")))

    current_page_tag = driver.find_element(By.XPATH ,
                                               "//*[@id='block-views-tenders-block']/div/div/div[3]/ul/li[2]").text.strip()
    all_divs = driver.find_elements(By.XPATH ,
                                        "//*[@id='block-views-tenders-block']/div/div/div[@class ='view-content' "
                                        "]/div")


      for each_div in all_divs :

            singleData = {
                # could not detect
                "language" : 107 ,
                # means open
                "status" : 0 ,
                "op_link" : "" ,
                "website" : website_name ,
                "close_date" : '' ,
                # means not available
                "organization" : website_name ,
                "description" : "" ,
                "title" : '' ,
                "checksum" : "" ,
                # means not available
                "country" : '' ,
                "published_date" : ''
            }

            singleData['title'] = each_div.find_element(By.XPATH ,
                                                        ".//span[@class='title-accr no-transform']").text.strip()
    
                singleData['country'] = each_div.find_element(By.XPATH ,
                                                              ".//div[1]/div/div/div[@class ='field-items']/div").text.strip()
                close_date = each_div.find_element(By.XPATH , ".//div//div[1]/div").text.strip()
    
                 #description always returns me empty text.
                description = each_div.find_element(By.XPATH, ".//div/div[2]/div[3]/div[2]/div/p").text.strip()
                download = each_div.find_elements_by_xpath('.//div//div[2]/div[4]/div[2]//a')
                download_file_link = []
                for eachfile in download :
                    download_file_link.append(eachfile.get_attribute('href'))

my code can get the title, country, deadline, and its attachment but can't get the description part. it returns me an empty text but when I see it on the website it has text in it.

can anyone help me with the issue and solution. thanks in advance

enter image description here

3
  • Not every single one seems to have a description section. Dec 14, 2020 at 7:23
  • I tried looking at your page and noticed that the description field doesn't appear by default. Are you setting specific values into the filter dropdown at the top to get these values which contain description?
    – Mugen
    Dec 14, 2020 at 7:24
  • Please provide a link to the page which contains the description or provide the HTML code.
    – Mugen
    Dec 14, 2020 at 7:35

1 Answer 1

2

Use a try except to catch it if it's there.There's some   so might need to remove it.

for each_div in all_divs :
     #description always returns me empty text.
    try:
        description = each_div.find_element(By.XPATH, ".//div[contains(text(),'Description')]/parent::div/div[2]//p[1]").get_attribute('innerHTML')
        print(description)
    except:
        print('none')

Outputs

This is the annual publication of information on recipients of funds for the TVET Project. 
none
At the latest 14 calendar days before the final date for receipt of tenders (up to 4th January 2021), tenderers may ask questions about the tender documents and the contract in accordance with Art. 64 of the Law of 17 June 2016. Questions shall be addressed in writing to:
Pour tout besoin d'information complémentaire, veuillez contacter: <a href="mailto:adama.dianda@enabel.be">adama.dianda@enabel.be</a>
none
none
none
Marché relatif &nbsp;à &nbsp;la&nbsp;fourniture, &nbsp;l’installation, &nbsp;la &nbsp;mise &nbsp;en &nbsp;marche &nbsp;et&nbsp;formation des utilisateurs et techniciens chargé de la&nbsp;maintenance &nbsp;des &nbsp;équipements &nbsp;de &nbsp;Laboratoire&nbsp;destinés au CERMES.&nbsp;
Pour tout besoin d'information complémentaire, veuillez contacter: <a href="mailto:adama.dianda@enabel.be">adama.dianda@enabel.be</a>
Tenders should request the price schedule in xls from Ms. Eva Matovu. email: <a href="mailto:eva.matovu@enabel.be">eva.matovu@enabel.be</a>

You could use

for each_div in all_divs :
     #description always returns me empty text.
    try:
        description = each_div.find_elements(By.XPATH, ".//div[contains(text(),'Description')]/parent::div/div[2]//p")
        for desc in description:
            print(desc.get_attribute('textContent'))
    except:
        print('none')

Outputs

This is the annual publication of information on recipients of funds for the TVET Project.
At the latest 14 calendar days before the final date for receipt of tenders (up to 4th January 2021), tenderers may ask questions about the tender documents and the contract in accordance with Art. 64 of the Law of 17 June 2016. Questions shall be addressed in writing to:
Françoise MUSHIMIYIMANA, National Expert in Contractualization & Administration _National ECA                                    (francoise.mushimiyimana@enabel.be ), with copy to
denise.nsanga@enabel.be
evariste.sibomana@enabel.be

They shall be answered in the order received. The complete overview of questions asked shall be available as of at the latest 7 calendar days before the final date for receipt of tenders at the address mentioned above.
Pour tout besoin d'information complémentaire, veuillez contacter: adama.dianda@enabel.be
Marché relatif  à  la fourniture,  l’installation,  la  mise  en  marche  et formation des utilisateurs et techniciens chargé de la maintenance  des  équipements  de  Laboratoire destinés au CERMES.
Pour tout besoin d'information complémentaire, veuillez contacter: adama.dianda@enabel.be
Tenders should request the price schedule in xls from Ms. Eva Matovu. email: eva.matovu@enabel.be
5
  • hi arundeep, some of each_div have many p tags and each p has some text in it. I want to get all the text of all p present in each_div. your code uses p[1] , if i remove1 from p[1], the code does not give the required output. could you please edit the code in such a way that gives me all p tags text. Dec 14, 2020 at 10:45
  • In fact I tried, it didn't work. I am trying again now. Dec 14, 2020 at 10:48
  • It would become a list you'd have to do a loop. Dec 14, 2020 at 10:48
  • yeah,I looped through eachp in description and tried to used full_text = full_text + eachp..get_attribute('innerHTML') but it does not work. full_text is defined above the loop, am I doing right? Dec 14, 2020 at 11:00
  • Try something like this if you want it without the tags. Dec 14, 2020 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.