1

Actually, the situation is a little more complex.

I'm trying to get data from this example html:

<li itemprop="itemListElement">
    <h4>
        <a href="/one" title="page one">one</a>
    </h4>
</li>

<li itemprop="itemListElement">
    <h4>
        <a href="/two" title="page two">two</a>
    </h4>
</li>

<li itemprop="itemListElement">
    <h4>
        <a href="/three" title="page three">three</a>
    </h4>
</li>

<li itemprop="itemListElement">
    <h4>
        <a href="/four" title="page four">four</a>
    </h4>
</li>

For now, I'm using Python 3 with urllib and lxml. For some reason, the following code doesn't work as expected (Please read the comments)

scan = []

example_url = "path/to/html"
page = html.fromstring(urllib.request.urlopen(example_url).read())

# Extracting the li elements from the html
for item in page.xpath("//li[@itemprop='itemListElement']"):
    scan.append(item)

# At this point, the list 'scan' length is 4 (Nothing wrong)

for list_item in scan:
    # This is supposed to print '1' since there's only one match
    # Yet, this actually prints '4' (This is wrong)
    print(len(list_item.xpath("//h4/a")))

So as you can see, the first move is to extract the 4 li elements and append them to a list, then scan each li element for a element, but the problem is that each li element in scan is actually all the four elements.

...Or so I thought.

Doing a quick debugging, I found that the scan list contains the four li elements correctly, so I came to one possible conclusion: There's something wrong with the for loop aforementioned above.

for list_item in scan:
    # This is supposed to print '1' since there's only one match
    # Yet, this actually prints '4' (This is wrong)
    print(len(list_item.xpath("//h4/a")))

    # Something is wrong here...

The only real problem is that I can't pinpoint the bug. What causes that?

PS: I know, there's an easier way to get the a elements from the list, but this is just an example html, the real one contains many more... things.

2 Answers 2

1

In your example, when the XPath starts with //, it will start searching from the root of the document (which is why it was matching all four of the anchor elements). If you want to search relative to the li element, then you would omit the leading slashes:

for item in page.xpath("//li[@itemprop='itemListElement']"):
    scan.append(item)

for list_item in scan:
    print(len(list_item.xpath("h4/a")))

Of course you can also replace // with .// so that the search is relative as well:

for item in page.xpath("//li[@itemprop='itemListElement']"):
    scan.append(item)

for list_item in scan:
    print(len(list_item.xpath(".//h4/a")))

Here is a relevant quote taken from the specification:

2.5 Abbreviated Syntax

// is short for /descendant-or-self::node()/. For example, //para is short for /descendant-or-self::node()/child::para and so will select any para element in the document (even a para element that is a document element will be selected by //para since the document element node is a child of the root node); div//para is short for div/descendant-or-self::node()/child::para and so will select all para descendants of div children.

3
  • .// solved the problem, thank you for your answer. But why it did? First, we load a page and get its html, then extract li tags and put each one in a list. Why using // would make any difference? Since at the second for loop we iterate over each one of the li tags, there should be one and only h4 and therefore a tag. EDIT: Could it be that even after extracting the li tags, we still have the whole html? This could be the real culprit.
    – Eekan
    Feb 13, 2017 at 18:03
  • @Eekan - Correct, even after extracting the li tags, the XPath query still has access to the whole HTML. In your example, list_item is a reference to the li elements. I believe the reason it's done like this is because XPath allows you to traverse up the tree and select a parent element. This means that the li would have to be a reference so that the other elements up the tree are still available for more complex querying. Feb 13, 2017 at 18:54
  • Thanks, mate. I guess I've grasped XPath better.
    – Eekan
    Feb 13, 2017 at 19:05
0
print(len(list_item.xpath(".//h4/a")))

// means /descendant-or-self::node() it starts with /, so it will search from root node of the document.

use . to point the current context node is list_item, not the whole document

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.