I am trying to figure out what I'm doing wrong here, but I keep getting lost...
In python 2.7, I'm running following code:
>>> import requests
>>> req = requests.request('GET', 'https://www.zomato.com/praha/caf%C3%A9-a-restaurant-z%C3%A1ti%C5%A1%C3%AD-kunratice-praha-4/daily-menu')
>>> req.content
'<html><body><h1>500 Server Error</h1>\nAn internal server error occured.\n</body></html>\n'
If I open this one in browser, it responds properly. I was digging around and found similar one with urllib library (500 error with urllib.request.urlopen), however I am not able to adapt it, even more I would like to use requests here.
I might be hitting here some missing proxy setting, as suggested for example here (Perl File::Fetch Failed HTTP response: 500 Internal Server Error), but can someone explain me, what is the proper workaround with this one?
requests
.requests
is great for fetching stuff with HTTP(S) but if you need to see what's in a browser you need a browser. I have had good luck with Selenium when you simply have to scrape a page that's mostly rendered by Javascript. It gives you an API that lets you drive and query a running browser. If you can find the underlying API endpoints that the page is pulling its data from you can userequests
and you'll be better off.