Questions tagged [python-requests]
USE ONLY FOR THE PYTHON REQUESTS LIBRARY. Requests is a full-featured Python HTTP library with an easy-to-use, logical API.
19,487
questions
0
votes
1
answer
24
views
I am trying to monitor a website but I'm facing some trouble
This is my code:
import requests,time
websiteURL = input("Enter The Website's URL: ")
response = requests.get(websiteURL)
while True:
if 100<= response.status_code <= 199:
...
0
votes
0
answers
14
views
App Store Connect returns 401 Status Code for all my requests
What am I doing wrong here? I get 401 status code for all my requests although something meaningful is printed for the serialized token.
from jwcrypto import jwk, jwt
import time
import os
import ...
0
votes
0
answers
12
views
I need to send a request.post() to the site with the js file uploading
I have a problem. To work with one of the sites must be loaded necessarily js file (in a normal browser is loaded automatically). Otherwise it will not be possible to log into the account.
How can I ...
0
votes
0
answers
11
views
Python requests by session id
I have 2 different urls, one for getting an session id, other one is for requesting (post/get) with the session id obtained via first url. When I use postman for that, everthing works fine but in case ...
0
votes
0
answers
19
views
Python requests - correct way to send query-string for POST
Here is what I want to be sent:
https://dev.xxx.com/employee-service/login?email=apitester@xxx.com&password=xxx!xxx
Body:
{
<JSON>
}
Here is how I send it.
Why I didn't use a dictionary ...
0
votes
0
answers
9
views
Python Webscraping with HTTP/S proxies shows my own ip
I really appreciate you reading this, any answer helps :)
When I use my proxies with selenium (that support http and https), to connect to a https page, it successfully loads, with the ip of the proxy....
0
votes
1
answer
28
views
Web Scraping a list of links from Tripadvisor
I'm trying to create a webscraper that will return a list of links to individual objects from the website example website.
The code I wrote takes the list of pages and returns the list of links to ...
1
vote
1
answer
42
views
How to only download updated CSV files via Python requests?
I am trying to download CSV files from:
Download sports fixtures, schedules and results as CSV, XLSX, ICS and JSON.
I have a python program that downloads the files I am looking for. The problem is ...
0
votes
1
answer
19
views
How do I do a beforetimestamp scroll request
I want to scroll on a followers list and to do that, I need to send a timestamp request to scrape users. How do I get the exact timestamp?
import requests
headers = {
'authority': 'tise.com',
...
0
votes
0
answers
16
views
AWS API Gateway 301 when call Instagram
I'm trying to fetch some data from Instagram and to prevent IP limitation I want to use AWS API Gateway proxies. I'm using a python lib requests_ip_rotator to manage my gateway and it works well, my ...
1
vote
1
answer
46
views
PowerShell Get request with body
I am trying this api endpoint.
I can call this in python, no problem, like the below
get_model_versions={
"filter":"name='model_name'",
"order_by":["version ...
2
votes
2
answers
37
views
I get status code 200 with postman but with request.get I get status code 500
When I make the request in postman to this api https://api.bigauto.solutionslion.com/api/store/showApp sending the parameters = { "part_number": "10-659"}
https://ibb.co/QvHvxPw ...
-3
votes
0
answers
20
views
Beautifulsoup Error : 'bytes' object has no attribute 'find_all' [closed]
I have created a script months back using Beautifulsoup, which was working fine without any issue. 7 days back I started to get this error even on printing soup
Traceback (most recent call last):
...
1
vote
1
answer
22
views
Requests: Proxy not working in people_also_ask module
I am scraping search results from google using people_also_ask module. The module itself dont have method to use proxies but I manually added proxies in the module. When I got blocked from google I ...
0
votes
1
answer
18
views
Paginating API error (Nested 'next' link)
I am trying to build a class to automate requests to our suplier, but i am having trouble with pagination.
This is the working snippet so far, but the api limits the request to 1000 records, and if ...
0
votes
0
answers
9
views
Keep more then one openned HTTP sessions
I wonder if it is possible by using python to establish more than one opened HTTP keep-alive session by one user simultaneously.
For example, could I use multiprocessing to create more sessions ...
0
votes
1
answer
24
views
Successful API call but no data returned
I'm trying to request data from api.travelpayouts to find cheap flights, but even after trying exact examples from their documentation I'm not returning any data.
The code I'm using (with example ...
-2
votes
1
answer
35
views
Is it possible to use a vpn without affecting the whole computer? [closed]
I have bought a vpn (Surfshark vpn) for my web scraper (I have a proxy so I am already aware that I should use a proxy, so there's no need to write an answer that tells me to buy a proxy instead). I ...
0
votes
1
answer
29
views
Download image with python requests
I'm new to python. I have to download some images from the web and save it to my local file system. I've noticed that the response content does not contain any image data.
The problem only occurs with ...
3
votes
0
answers
47
views
Python ImportError: sys.meta_path is None, Python is likely shutting down
When using __del__
datetime.date.today() throws ImportError: sys.meta_path is None, Python is likely shutting down
import datetime
import time
class Bug(object):
def __del__(self):
try:
...
1
vote
3
answers
30
views
Python Web Scrape Request resulting in a 406 Error
I am trying to scrape https://registry.verra.org/app/search/VCS/All%20Projects for a school project. I am trying to send a request to the "download excel" button by replicating the POST ...
1
vote
0
answers
13
views
Scrape Website that is running meteor, using python requests
You see, there is this website called edabit. All you need to know is that I want to scrape the website without using selenium.
I want to learn how.
What does selenium do under the hood that makes it ...
0
votes
1
answer
21
views
Getting a 400 Response Spotify API Python
I am trying to use requests.get with the Spotify API and I can't figure out why I keep getting a 400 response. I am working in an .ipynb file in Jupyter Lab running locally. I read these documentation ...
0
votes
2
answers
31
views
Unable to connect to a url with python request module
I'm unable to connect to the URL using requests module, but it works fine when browsed in a browser. Could it be some robots.txt issue Allowed/Disallowed issue ?
Below is the codebase.
import requests
...
0
votes
1
answer
36
views
Parallelize requests call with asyncio in Python
I'm trying to create an interface to an API, and I want to have the option to easily run the requests sync or asynchronously, and I came up with the following code.
import asyncio
import requests
def ...
0
votes
1
answer
26
views
Proper syntax of parameters using function run_in_executor()
To make a POST call to the API, I use the following script:
r = requests.post(
url,
headers={
"Content-Type": "application/json"
},
json={
"...
0
votes
2
answers
36
views
Why is the Python Requests Module not returning links?
So I had created a python web scraper for my college capstone project that scraped around the web and followed links based on a random selection from the page. I utilized Python's request module to ...
0
votes
0
answers
15
views
How to get the actual download link embedded with any "Download" button
I have this code to download a big file in chunks:
import requests
from tqdm import tqdm
def get_size(url):
response = requests.head(url)
size = int(response.headers['Content-Length'])
...
0
votes
0
answers
20
views
The problem when working with requests using a proxy in Python
Sorry if some of the suggestions are going to be weird. I used a translator, because this question was not answered on my language's stackoverflow.
import threading
import requests
import random
...
0
votes
2
answers
36
views
Calling a REST API with Api Id and key using Python
I am trying to send a request to get a token to an API.
curl -X POST "https://base_url/token"
-H "accept: application/json"
-H "Content-Type: application/json" -d ...
1
vote
1
answer
23
views
positional argument error in sheduling task in python
I got a positional argument error when i tried running my code and the do() passes extra arguments to the job function
import schedule
from schedule import Scheduler
import threading
import warnings
...
-3
votes
0
answers
23
views
Why Python is printing strange characters? [closed]
I've tried making a web crawler that stores an unique list of links using requests, but i keep receiving as a link these strange chars, does someone know why is that?
Link to full code: https://github....
0
votes
0
answers
11
views
Python Requests Passing Array to GET API query
I'm using Amazon's selling partner API to query my inventory. I only want a few SKUs, so I'm trying to send a list using the sellerSkus parameter. Please help me specify the "sellerSkus" ...
0
votes
0
answers
14
views
Python Adding Images to specific folders
I have a list of 4000 picture urls and id's and trying to pull them into separate folders based on the id. I am able to create the folders and insert the files into the directories but I seem to have ...
0
votes
0
answers
15
views
Python script that makes request to router gets stuck on recvfrom() after a month of activity
I have a Python script that makes requests every minute to the router to see the active users.
after a month and a half of working it just hanged
when I run:
ubuntu@ubuntu:/db/$ sudo strace -p 478419
...
-3
votes
0
answers
29
views
How to tell code which variable to pick on web request [closed]
I'll try to explain. I am requesting weather updates from an open source website. Now this website gives me all the info on a separate link which I give to the code. The real problem comes here. I ...
0
votes
0
answers
6
views
How to pass multiple proxies as a list but stop when it works?
I have "implemented" a way to work with proxies, and python's requests library. I'm scraping a website and after a while even though I wait like 20 sec after making the next request it ...
0
votes
0
answers
15
views
How to get instant response (one of many) from API
I need to make such an architecture:
The first block is some application that sends five POST requests to the global API - which is second block. This API gets all five requests, which have data and ...
0
votes
0
answers
17
views
How to bypass Google recaptcha in a form using Mechanize
I need to fill a form on a webpage and bypass its captcha.
using this library I'm able to solve the captcha and get g-recaptcha-response stored in a variable. now the question is, how to pass the ...
1
vote
2
answers
33
views
Python pulling html table from webpage
The table on this page needs to be scraped daily. We are trying to keep the scraping as simple (robust) as possible so there are no issues with the code running on our server. Would like to steer ...
0
votes
0
answers
9
views
zip file upload with Python Requests Not working
Hi am attempting to upload a file to a server when making a post with requests but everytime I get errors. If do the same with cURL it goes through but Im not familiar with cURL or uploading files ...
-1
votes
1
answer
15
views
How to set a destination for shutil.copyfileobj?
This code saves a discord image to the folder which it is in. I tried to set a destination for the save file, but I haven't found anything on the shutil website which sets the destination. I tried to ...
0
votes
0
answers
11
views
Python Requests: PHP-Form login is not being sent
I'am trying to scrape this website: https://gb4.typewriter.at/. I'am struggling with logging in. The weird thing is: the form is being filled out but it is not sent, and therefore I don't get any ...
0
votes
0
answers
27
views
Getting error when running a http request to microsoft azure
I am sending a post request to Microsoft Azure server for face identification
foo = {
"PersonGroupId": "596366",
...
-2
votes
1
answer
30
views
How can i find all channel ids?
def SpamRequest():
channel_id = random.choice(channel_ids)
r = requests.post(f"https://discord.com/api/v9/channels/{channel_id}/messages", data=payload, headers=header)
How can i ...
0
votes
0
answers
20
views
Scraping With Python - Works Locally But Not on Server
I'm sending a GET request to a URL which returns a JSON as a response.
Locally this works perfectly (I had to add headers to bypass).
However, when running the very same script on my server (ubuntu ...
0
votes
0
answers
19
views
Python Proxies Returning Connection Error
I've seen hundreds of posts with this same question but none of the answers ever work. I've tried doing everything on the post answers and they all don't work.
import requests
import random
def ...
0
votes
1
answer
15
views
Log into WSJ using Python requests
I am trying to log into the Wall Street Journal's website using the Python module requests. I know that other people on these boards here have successfully used Selenium to accomplish the same. ...
0
votes
0
answers
16
views
Cookies that come from Python Requests lib do not work when using rest API
I have a problem, maybe is my lack of knowledge on the subject.
I'm building a script to connect to the Jira API via cookie auth.
When I use on my script:
session = requests.Session()
a_session.get('...
-1
votes
1
answer
27
views
How to convert yaml file to json in Flask application? [duplicate]
Is there a way to convert yaml textual data(from python requests on Flask) to json format, or at least exract key value pairs.
config.yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: confmap
...