Actually, the situation is a little more complex.
I'm trying to get data from this example html:
<li itemprop="itemListElement">
<h4>
<a href="/one" title="page one">one</a>
</h4>
</li>
<li itemprop="itemListElement">
<h4>
<a href="/two" title="page two">two</a>
</h4>
</li>
<li itemprop="itemListElement">
<h4>
<a href="/three" title="page three">three</a>
</h4>
</li>
<li itemprop="itemListElement">
<h4>
<a href="/four" title="page four">four</a>
</h4>
</li>
For now, I'm using Python 3 with urllib
and lxml
.
For some reason, the following code doesn't work as expected (Please read the comments)
scan = []
example_url = "path/to/html"
page = html.fromstring(urllib.request.urlopen(example_url).read())
# Extracting the li elements from the html
for item in page.xpath("//li[@itemprop='itemListElement']"):
scan.append(item)
# At this point, the list 'scan' length is 4 (Nothing wrong)
for list_item in scan:
# This is supposed to print '1' since there's only one match
# Yet, this actually prints '4' (This is wrong)
print(len(list_item.xpath("//h4/a")))
So as you can see, the first move is to extract the 4 li
elements and append them to a list, then scan each li
element for a
element, but the problem is that each li
element in scan
is actually all the four elements.
...Or so I thought.
Doing a quick debugging, I found that the scan
list contains the four li
elements correctly, so I came to one possible conclusion: There's something wrong with the for
loop aforementioned above.
for list_item in scan:
# This is supposed to print '1' since there's only one match
# Yet, this actually prints '4' (This is wrong)
print(len(list_item.xpath("//h4/a")))
# Something is wrong here...
The only real problem is that I can't pinpoint the bug. What causes that?
PS: I know, there's an easier way to get the a
elements from the list, but this is just an example html, the real one contains many more... things.