Questions tagged [urllib]
Python module providing a high-level interface for fetching data across the World Wide Web. Predecessor to urllib2. In Python 3, urllib2 and urllib have been reorganized and merged into urllib.
3,942
questions
-1
votes
0
answers
9
views
I have 3m URLs for images. What is the fastest way of downloading into blob and how long will it take?
I have 3 million URL's in CSV. I want to use Azure Databricks and download into blob storage. I really need to know long it will take and any advice on the flow, steps and code.
Previously downloaded ...
-1
votes
0
answers
14
views
urllib_error / HTTPError 302
I am try to scrape a website using python==3.11 and Beautifulsoap, but it throws an error as following:
urllib.error.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would ...
0
votes
0
answers
11
views
Why do I get "urllib.error.URLError: <urlopen error [Errno -3] Try again>" when I use the urlopen() function?
I am trying to open a URL using the following code:
import urllib.request
with urllib.request.urlopen("http://www.example.com/") as doc:
html = doc.read()
I've tried with many ...
0
votes
0
answers
14
views
How can I change the urllib requests proxies?
I have a function that makes requests on a website, when I'm not specifying any proxies it works fine. But when I make requests through proxies, it doesn't work.
I've tried this code:
def parseCookie(...
-2
votes
0
answers
20
views
how to I download csv file from github [duplicate]
code
import os
import urllib.request
DOWNLOAD_URL = "https://github.com/ageron/handson-ml2/tree/master/datasets/titanic/"
TITANIC_PATH = os.path.join("datasets&...
-2
votes
0
answers
17
views
urllib.request.urlopen is being executed like forever
I'm trying to use the following piece of code, but it is being executed like forever, neither it ends nor it gives an error, and I am never be able to run my program as it is supposed to be:
import ...
0
votes
2
answers
28
views
Reading a File in Pandas with quote symbol in Filename
I have a file name which has say the following (with apostrophe in file name)
Path = 'C:\xxx\yyy-testing\my_inner_path'
FileName = 'ABC'01myFile.xlsx'
so define File = Path + FileName
I am trying to ...
0
votes
0
answers
10
views
urllib.error.HTTPError: HTTP Error 404: Not Found traying to webscrape
why does this gives me an error?:
la_url = 'https://www.tradingview.com/symbols/'
news_tables = {}
news = '/news/'
for ticker in tickers:
url = la_url + ticker +news
req = Request(url=url,...
0
votes
0
answers
28
views
WinError 10061 Running Selenium+Chromedriver
I have a Python GUI application that uses Selenium and Chromedriver to crawl sites, interact with elements, download files, etc. The application has been packaged as a standalone .exe (produced using ...
0
votes
1
answer
34
views
Printing list item above tqdm progress bar fails to replace the previous item instead adds a new line
I am attempting to read urls from a text file "url.com", append them to a list, then iterate the list to check if the url is an active website.
I am having problems with tqdm, I don't know ...
1
vote
0
answers
65
views
How to access SharePoint site in Linux using proxies in Python? I am successful using curl command
I am try to establish connection to a SharePoint site from Python. To get the access token I m using "requests" module from python. I have set proxies in Linux Shell
export http_proxy=http://...
0
votes
1
answer
66
views
Cyrillic Encoding in Urllib Python with lower cases
My goal is to encode this dict with cyrilic text:
target_dict = {"Animal": "Cat", "city": "Москва"}
To this (cyrilic lettesrs with lower case encoded):
Animal=...
0
votes
1
answer
41
views
Python http request and utf-8 encoding
I use urllib to perform http request to a REST API. All data are in JSON with utf-8 encoding.
The problem is that when i read some special charater via REST API i get the correct encoding (i.e. if i ...
0
votes
3
answers
52
views
Word count script in Python [duplicate]
Can someone please explain me why there is 'b' in front of each word and how to get read of it? Script returns something like this:
word= b'yesterday,' , count = 3
current_word = {}
current_count = ...
0
votes
1
answer
31
views
Can somebody help me whith urllib request in python code? [duplicate]
I'm trying to make a program that requests a meteorology message to one free API made by the Brazilian air force and shows it to users. This program it works great at three first hours, but after, it ...
0
votes
1
answer
58
views
How to get a style value inside a span tag with a python scraper?
I downloaded the code of this website: https://www.ilmakiage.co.il/mineral-lip-pencil-4040 with the response.get method. In the inspectors mode I could see that there was a style value "display: ...
0
votes
0
answers
25
views
urllib.error.HTTPError: HTTP Error 404: Not Found Issue when Trying to read url from Yahoo Finance given Stock Symbol
I am trying to write a program that will help me with accounting work. I am trying to pull information by using stock prices for companies over time. I am utilizing urllib.request to read and open ...
0
votes
0
answers
23
views
i have created an insta image scaper using urllib but the files cannot be opened
so i have tried shutil also, as far as i can figure the file gets generated and downloaded but at the end it refuses to be opened by any software. can someone tell me why??
This is for private use for ...
1
vote
1
answer
16
views
How to continue other link when Urllib 404 error
I'm trying to download images from a csv with lot of links. Works fine until some link is broken (urllib.error.HTTPError: HTTP Error 404: Not Found) .
import pandas as pd
import urllib.request
import ...
0
votes
0
answers
29
views
Implement the parse_cookie function python
Implemented the test function from parsing, the tests passed. Now it is necessary to make a second function that uses cookies, receives as input a string with values, and at the output we have a key\...
0
votes
0
answers
65
views
How to crawl webpages with titles in URL?
I would like to crawl news from a webpage. They use URLs like this:
https://www.eqs-news.com/de/news/adhoc/vectron-systems-ag-vectron-uebernimmt-acardo/1724329
The last number in the URL is a unique ...
0
votes
0
answers
20
views
Freezes at accessing Mouser.com using Urllib
I want to access mouser.com using urllib. When I try to fetch the data from the URL, it hangs indefinitely.
Here is code:
import urllib.error
import urllib.request
try:
htmls = urllib.request....
0
votes
2
answers
29
views
Python - Download zip files with requests package but get unknown file format
I am using Python 3.8.12. I tried the following code to download files from URLs with the requests package, but got 'Unkown file format' message when opening the zip file. I tested on different zip ...
0
votes
2
answers
88
views
How to convert from urllib to scrapy?
I am webscraping an output of url amazon links, each of these links contains the price of a given book. So, the idea is to get the link and the price of the book of that link.
I have created code ...
0
votes
2
answers
50
views
urllib.requiest.urlopen error: certificate verify failed on python Virtual Environment
Im building a Web Scapper, when testing on venv -> [SSL: CERTIFICATE_VERIFY_FAILED]
But, when I'm testing on ipython shell -> Perfectly good
I wondering what the root problem is?
Thanks for your ...
1
vote
1
answer
52
views
Url with umlaut (non ascii), dashes and brackets is not parseable by requests
I am trying to get the HTML content of a page with requests, but it results in UnicodeDecodeError. The reproducible code:
import requests
import urllib
url = "https://www.unique.nl/vacature/...
0
votes
1
answer
49
views
Read compressed GRIB file from URL into xarray dataframe entirely in memory in Python
I am trying to read the gzipped grib2 files at tis URL: https://mtarchive.geol.iastate.edu/2022/12/24/mrms/ncep/SeamlessHSR/
I want to read the grib file into an xarray DataFrame. I know I could write ...
0
votes
1
answer
23
views
urllib: specifying download path makes url invalid
I am trying to write a function that takes a url and a path and downloads a file to that path IF it's a text file.
import urllib
import re
import os
mcBethURL = 'https://ia802707.us.archive.org/1/...
-1
votes
4
answers
63
views
How to get specific part of any url using urlparse()?
I have an url like this
url = 'https://grabagun.com/firearms/handguns/semi-automatic-handguns/glock-19-gen-5-polished-nickel-9mm-4-02-inch-barrel-15-rounds-exclusive.html'
When I use urlparse() ...
1
vote
1
answer
47
views
UnicodeEncodeError: 'ascii' codec can't encode character '\xf8' in position 66: ordinal not in range(128)
I have a excel-sheet with a bunch of unique url's which will trigger a download of a file.
I've automated the process using Python and pandas, but I'm struggling with finding a solution for when ...
0
votes
1
answer
63
views
AWS Lambda Chalice app not receiving request payload
I have a small lambda function that I am creating with AWS Chalice in Python.
When testing locally all works as intended and my test request works perfectly.
However when running chalice deploy and ...
0
votes
0
answers
22
views
Urlopen can't decode bytes
My code is very simple:
from urllib.request import urlopen
quest = input("question link: ").strip()
page = urlopen(quest).read().decode()
I want the user to input an url and to store the ...
0
votes
0
answers
35
views
How can I stop API returning URL escape code of the characters from URL I am sending through Python code?
I am trying to query GitLab's API to fetch users.
URL I have passed to the API:
https://gitlab.com/api/scim/v2/groups/cimpress-technology/Users/waad|5cmnhgd@NVJ4bHFYnhfv_mvbj4KGjguJGka_:
The response ...
0
votes
3
answers
122
views
How to pass a dictionary to an url in POST request in Python
I have to get information about product stocks from marketplace's API via POST request.
API requires to send products IDs (sku) in url
Example:
https://api.market.com/campaigns/{campaignId}/warehouse/{...
-1
votes
3
answers
45
views
When accessing a Rest API via requests.get it times out. Same URL gets the response in browser
I am trying to access a Rest API that supports GET, POST, PUT, MOVE functions. When I try the URL using the browser to get list of files I get a response from the REST API. However, when I try to call ...
0
votes
1
answer
178
views
Stop urllib.request from raising exceptions on HTTP errors
Python's urllib.request.urlopen() will raise an exception if the HTTP status code of the request is not OK (e.g., 404).
This is because the default opener uses the HTTPDefaultErrorHandler class:
A ...
0
votes
0
answers
147
views
urllib.error.URLError: <urlopen error [WinError 10060]
I am not able to execute my script, when I run my script these errors show in the command prompt :
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did
not ...
1
vote
1
answer
36
views
wget Python package downloads/saves XML without issue, but not text or html files
Have been using this basic code to download and store updated sitemaps from a hosting/crawling service, and it works fine for all the XML files. However, the text and HTML files appear to be in the ...
1
vote
0
answers
55
views
Issue with BingX Swap Api Python HTTP Error 405: Method Not Allowed
I'm using urllib.request.
I'm trying to use the BingX Swap Api with Python and I'm getting this error:
I'm trying to use the BingX Swap Api with Python and I'm getting this error:
I'm trying to use ...
1
vote
1
answer
49
views
Can't find hrefs of interest with BeautifulSoup
I am trying to collect a list of hrefs from the Netflix careers site: https://jobs.netflix.com/search. Each job listing on this site has an anchor and a class: <a class=css-2y5mtm essqqm81>. To ...
0
votes
0
answers
184
views
Deactivate SSL verification globally in Python's requests or urllib package
I know that I can disable SSL verification in Python's request package as follows:
import requests
response = requests.put("some.host/RESTfulService/My/Endpoint/", verify=False)
The ...
1
vote
2
answers
84
views
pb avec web scraping
import requests
import pandas as pd
from urllib.request import urlopen
from bs4 import BeautifulSoup
df = []
for x in range(1,31):
url_allocine= 'https://www.allocine.fr/film/meilleurs/?page='
...
1
vote
1
answer
38
views
a data collection with web scraping
I'am trying to extract data from a site and then to create a DataFrame out of it. the program doesnt work properly. I'am new in web scraping. Hope somoene help me out and find the problem.
from urllib....
0
votes
0
answers
26
views
chrome user agent doesn't work for a scrapper
I have the following code to scrape images:
import os, requests, lxml, re, json, urllib.request
from bs4 import BeautifulSoup
from os.path import expanduser
headers = {
"User-Agent": &...
0
votes
1
answer
42
views
Downloading an image from the web and saving
I am trying to download an image from Wikipedia and save it to a file locally (using Python 3.9.x). Following this link I tried:
import urllib.request
http = 'https://en.wikipedia.org/wiki/Abacus#/...
0
votes
0
answers
23
views
connection error while trying to plt.show()
I am using a remote interpreter in PyCharm to run code that plots heat maps. yesterday it worked fine, but today it stopped working without me changing the code or my conda environment.
I am trying to ...
-1
votes
1
answer
36
views
How to get full html when requesting page
I am working on scrapping web page which requires authentication and need full page html to scrap the data. I am using cookies to authenticate request.
I am not in favor of using selenium driver as it ...
0
votes
2
answers
67
views
How to get rid of the rest of the text after getting the results I want?
import urllib.request
import json
from collections import Counter
def count_coauthors(author_id):
coauthors_dict = {}
url_str = ('https://api.semanticscholar.org/graph/v1/author/47490276?...
0
votes
1
answer
87
views
How can I get more than 20 youtube search results using urllib?
I am trying to make a python script that fetches youtube channel URLs with a keyword input.
I'm using urlib to request the html of the search result page and then filtering out the channel IDs using ...
0
votes
1
answer
31
views
Bypass SSL certificate in Python using urllib
I am trying to bypass the SSL Certificate and use the URL in Python,
All HTTPS sites are getting the same error
Kindly suggest how can it get resolved. Thanks in Advance.
Code:
import urllib.request ...