0

I'm trying to get the text "Former United States Secretary Of State" out of this tag. I've tried many ways but cannot seem to get it.

<div class="tag"><a href="en/profession/748/former-united-states-secretary-of-state" class="">Former United States Secretary Of State</a></div>

This is my code:

site_content = etree.HTML(result)
selection = site_content.xpath(xpath_select)
content = [item.strip() for item in selection]

Every other xpath is working. This is the xpath I'm using as there are multiple of this one tag on the page "/html/body/div[5]/div[4]/div[5]/div[*]"

Any help in right direction would be greatly appreciated.

Working url = https://www.blackandwhitequotes.com/en/quotes/william-jennings-bryan_1182154_1&key=2OP8jfJC1D

3
  • Can you edit your question and add the url you're working with? Aug 31, 2022 at 1:53
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
    – Community Bot
    Aug 31, 2022 at 5:15
  • I added it in bottom now, apologies. What I'm after is all the text of the 'tag' in on this page. Preferably as a list.
    – Py47
    Aug 31, 2022 at 8:15

1 Answer 1

1

Your XPath doesn't seem to be valid for your HTML example.

In general when building XPaths it's best to rely on classes and identifiers rather than tree structure. So, we should write //div[contains(@class,"tag")] instead of //div/div/div[0] etc.

In your case you can also use //text() XPath function to select all of the inner text of your node:

from lxml import etree

html = """<div class="tag"><a href="en/profession/748/former-united-states-secretary-of-state" class="">Former United States Secretary Of State</a></div>"""
tree = etree.HTML(html)
print(tree.xpath("//div[contains(@class,'tag')]//text()")[0])
#'Former United States Secretary Of State'

Looking for a div with class of tag will be much more reliable way of parsing this HTML than /html/body/div[5]/div[4]/div[5]/div[*]

5
  • On this website the tag website would give me 55 'tags'. While the selection I want is only 26 'tags'. (I do realise I added the working website post-answer)
    – Py47
    Aug 31, 2022 at 8:36
  • 1
    I found the reason why this was happening. Apperently on the website there are more classes containing the name 'tag' that's why it was giving me move. Fixed it by changing the contains to the actual class name => '//div[@class="tag"]//text()' thank you for pointing me in the right direction.
    – Py47
    Aug 31, 2022 at 8:46
  • Could you explain to me why the straight xpath I was using does not work.
    – Py47
    Aug 31, 2022 at 9:49
  • @Py47 it's unreliable because it's too structure specific. Modern websites don't always have exact same html tree structure and your xpath is way too specific. What are the chances that the detail will always be in body -> 5th div -> 4th div ->5th div? If the page gets updated with a minor detail the div you are looking for will no longer be 5th or 4th. That's why relying on class names is more consistent since class names are used to apply styling, so if the web master is applying blue color to all nodes with class tag then the same way you can find all tags :) Aug 31, 2022 at 10:31
  • Thanks for taking the time to explain it to me. Greatly appreciated!
    – Py47
    Aug 31, 2022 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.