1

I'm trying to get products for a project I'm working on from this page:lazada, page ispection using :

from bs4 import BeautifulSoup
import urllib
import re
r = urllib.urlopen("http://www.lazada.co.id/catalog/?q=note+2").read()
soup = BeautifulSoup(r,"lxml")
letters = soup.findAll("span",class_=re.compile("product-card__name"))
print type(letters) 
print letters[0]

When I do this I am getting the following error:

Traceback (most recent call last):
  File "C:/Python27/project/testaja.py", line 9, in 
    print letters[0]
IndexError: list index out of range

Any thoughts on this?

2
  • 1
    What does print(r) output? The code works fine for me. Apr 6, 2016 at 16:35
  • This code also works completely fine for me. I suspect that you either do not have access (maybe you scraped it too often and the server blocked you - check if header code is 403) or for some reason the HTML was never sent along. Apr 6, 2016 at 17:31

1 Answer 1

0

I think you may have hit their page too much, navigate there in a browser and see what the page returns on your network.

Also, you can modify your code so you can check the page response header to make sure that the page returned properly before trying to scrape it. I modified your code to show an example of this below:

from bs4 import BeautifulSoup
import urllib
import re

r = urllib.urlopen("http://www.lazada.co.id/catalog/?q=note+2")
header_code = r.getcode()

if header_code == 200:
    html = r.read()
    soup = BeautifulSoup(html, "lxml")
    letters = soup.findAll("span", {"class" : re.compile("product-card__name")})

    for letter in letters:
        print letter
else:
    print("oops, something went wonky. Page response was: %s"% header_code)
6
  • i try your code and get header code 200 but still cant find the element span it return nothing any other soution sir ? btw im using python 2.7.1.1 in windows 8<br> thankyou in advance
    – Rio Arfani
    Apr 7, 2016 at 3:00
  • Check to see how I modified the soup findAll in my example, see if that works for you.
    – bmcculley
    Apr 7, 2016 at 3:02
  • Alright, give it a try now. I didn't realize beautifulsoup 4 uses regex differently than version 3.
    – bmcculley
    Apr 7, 2016 at 3:31
  • Still no Luck sir, :( but when i tray to do this to other web ex link, its work fine but it still return nothing with lazada
    – Rio Arfani
    Apr 7, 2016 at 4:09
  • Can you make sure the HTML is loading properly for you and it's not some captcha or something similar? I've tested it as is against this page and it returns a whole list of items.
    – bmcculley
    Apr 7, 2016 at 4:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.