1

I am trying to scrape a travel website agoda.com. I am using selenium and beautifulsoup. I am able to reach the page from where I need to scrape the hotel names and there prices. I have scraped that too. But the problem is I am getting all values with tags Output :The Taj Mahal Palace

How to get only text between anchor tags

Also I have the prices scraped but that too in tags. But I dont know how to print together the hotel name and prices like The Taj Mahal Palace, USD 219.

Please Help

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re
import time
from bs4 import BeautifulSoup
import urllib2
import sys;
reload(sys);
sys.setdefaultencoding("utf8")


class Agoda(CrawlSpider):
   name = 'agoda'
   allowed_domains = ["agoda.com"]
   start_urls = ["http://www.agoda.com"] 
   driver = webdriver.Firefox()
driver.get("http://www.agoda.com")
driver.find_element_by_id("ctl00_ctl00_MainContent_area_promo_HomeSearchBox1_TextSearch1_searchText").clear()
driver.find_element_by_id("ctl00_ctl00_MainContent_area_promo_HomeSearchBox1_TextSearch1_searchText").send_keys("Mumbai")
driver.find_element_by_xpath("//select[contains(@id,'ddlCheckInDay')]")
driver.find_element_by_xpath("//option[contains(.,'Mon 09')]").click()
driver.find_element_by_id("ctl00_ctl00_MainContent_area_promo_HomeSearchBox1_SearchButton").click()
   driver.find_element_by_id("ctl00_ContentMain_rptAB1936_ctl01_rptSearchResultAB1936_ctl01_lnkResult1936").click() 
time.sleep(20);
#print driver.page_source
TotalResults = driver.find_element_by_xpath("//span[@class='blue ssr_search_text']")
print TotalResults.text

html_source = driver.page_source
soup = BeautifulSoup(html_source) 


names = soup("a", {"class":"hot_name"})

#comments = soup("div", {"class":"mbluebold col_hotelinfo_name"}, text = True)  
#comments[0].Contents()
#print comments 
#tags = soup.find_all("a")
for name in enumerate(names): 
    print name

prices = soup("span", {"class":"fontxlargeb purple"})
for price in enumerate(prices): 
    print price 

1 Answer 1

1
try the get_text() method on the 'a' tags(or any tags)

for instance if html is simply "<a href="alisejflai">hello</a>"

soup = BeautifulSoup(html)

soup.get_text() is 'hello'`

EDIT:

Regarding your comment: enumerate(names) will produce tuples of the form:

(0, <a class="hot_name"> howdy pardner</a>)
(1, <a class="hot_name">againagain</a>)

Since you only want to call get_text() on the actual 'a' tags, you either need to do:

for name in names:
    name.get_text() # no tuple involved

or if you must use enumerate for some reason:

for name in enumerate(names):
    name[1].get_text() # accessing just the a tag within the tuple.

EDIT:

If you want to put the hotel names and prices in 'pairs' you could replace my first edit above with the following:

These list comprehensions are more pythonic and I believe faster than the for loops:

hotel_names = [name.get_text() for name in names] #or [name[1].get_text() for name in enumerate(names)]
prices = [price.get_text() for prices in prices] #[price[1].get_text() for price in enumerate(prices)]

name_price_list = zip(hotel_names, prices)

for name, price in name_price_list:
    print name, price

prints out:

name price
name price
name price etc.

let me know if this works for you

9
  • Thank you so much Totem but when I am adding this names = soup("a", {"class":"hot_name"}) for name in enumerate(names): name.get_text() I am getting an error AttributeError: 'tuple' object has no attribute 'get_text' . Thanks
    – Tarun
    Dec 7, 2013 at 18:54
  • Thanks Totem , its working now. One more thing , how can i get together both name and prices as [hotel name any separator price] As now I am first printing names and then prices . How can they be clubbed . Thanks Tarun
    – Tarun
    Dec 8, 2013 at 19:06
  • I have edited the post again. If you find this answer helpful, please vote up and accept. Thanks.
    – Totem
    Dec 8, 2013 at 19:19
  • Thanks a lot totem. I have already accepted the answer but vote up requires 15 reputation , I have 13 so need 2 more to give a vote up. But really thanks man .
    – Tarun
    Dec 8, 2013 at 19:27
  • Hey Totem I am also curious to know that how can i go to all the next pages of particular results and repeat the same steps to crawl name and prices.As in this case there are more than 200 results but I am getting 20 of the page I scraped.So how to go to next pages and scrape this same info.Any help appreciated.
    – Tarun
    Dec 8, 2013 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.