1

I created a script with selenium to get given GST NUMBER information. I completed that program and it's giving me required details on output without any problem.

Now I do not want to interact it with chrome browser anymore so I'm trying to do this with BeautilfulSoup.

BeautifulSoup is new for me so I have not much idea to find element and I searched a lot about how to send keys with BeautifulSoup but I'm not getting it.

Now my script is stuck here.

from bs4 import BeautifulSoup
import requests
import urllib.request as urllib2

quote_page = 'https://my.gstzen.in/p/search-taxpayer'
page = urllib2.urlopen(quote_page)
soup = BeautifulSoup(page, 'html.parser')

Now even if I manage to find the gst input element then I'm wondering how do I send keys to it? like the 15 digit gst number with sending enter button code or clicking on "Search GST Details".

If possible, then let me know the solution so I can start my research on it.

Actually, I need to do complete this tonight.

Plus, Here is my script which do the same thing with selenium easily and I want to do the same thing with BeautilfulSoup because I do not want chrome to be run every time while checking the GST and BeautilfulSoup seems interesting.

import selenium
import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
import csv
import requests
#import pyvirtualdisplay            
#from pyvirtualdisplay import display

#display = Display(visible=0, size=(800, 600))
#display.start()


browser = webdriver.Chrome('E:\\Chrome Driver\\chromedriver_win32\\chromedriver.exe')
browser.set_window_position(-10000,0)
browser.get('https://my.gstzen.in/p/search-taxpayer/')


with open ('product.csv') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    next(csv_reader)
    for row in csv_reader:
        name, phone = row
        time.sleep(1)
        gst = browser.find_element_by_name('gstin')
        gst.click()
        gst.send_keys(name)
        time.sleep(1)
        Details = browser.find_element_by_xpath("//*[contains(text(), '                Search GSTIN Details')]")
        Details.click()
        info = browser.find_element_by_class_name('col-sm-4')
        print(info.text)
        info2 = browser.find_element_by_xpath('/html/body/div[4]/div/div/div[1]/div[2]/div[2]/div[1]/div[2]')
        print(info2.text)
        input('Press Enter to quit')
        browser.quit()

1 Answer 1

0

BeautifulSoup is a library for parsing and formatting, not interacting with web pages. For the latter, if that page requires JavaScript to work, you're stuck using a headless browser.

If it doesn't, you have at least two options:

  1. Watch the Network tab in your browser's developer tools and see if you can recreate the request for the page you want using requests or urllib2
  2. Use mechanize, which is built specifically to work with forms on sites that don't depend on JavaScript

mechanize is a little more work if there's no CSRF token or similar mechanism (though, again, it'll fail if JavaScript is required) and a little less work if there is.

3
  • Thank you for taking time to answer this. I'm going to try and see if it works. Commented Oct 3, 2018 at 12:44
  • You're welcome. Both options are things I've done myself recently while automating tasks at work, so they're definitely viable; hopefully at least one of them works for your target site.
    – kungphu
    Commented Oct 3, 2018 at 21:45
  • 1
    Awesome. Again thanks for sharing your knowledge with me. Have a good day! Commented Oct 4, 2018 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.