22,118
questions
0
votes
0
answers
21
views
Where do I look to find a domain's policy on web scraping? [closed]
I often use bs4 and requests in python to gather data from a website.
Where can I find the policy on data collection? (Keywords, Sections, shortcuts, etc.)
I'm hesitant to post any of my code on the ...
0
votes
1
answer
23
views
How to Fetch GraphQL JSON Data from a URL in Python?
I am trying to fetch GraphQL data from the URL:
https://URL
I’ve tried using Python’s requests library to make a POST request, but I’m not sure how to structure the request to get the desired JSON ...
0
votes
1
answer
43
views
Why is Python httpx.get or requests.get much slower than cURL for this API?
For a home automation project I am trying to pull train delay data. An API wrapper exists, with cURL examples. These work fine, but both Python's requests.get and httpx.get are slow to pull data (up ...
0
votes
0
answers
16
views
Python requests.get method stuck on audio streams [duplicate]
requests.get gets stuck on streams, like, for example,
https://centova.radios.com.pt/proxy/476?mp=/stream
https://nl.digitalrm.pt:8216/stream
Is there any way to detect this automatically and abort? ...
0
votes
0
answers
27
views
How to download a image using requests
I have to download and image using Python requests. I logged into the website and followed each session when I clicked the image link from the network section, I got a post session with URL.
("...
-4
votes
0
answers
31
views
Efficient Strategies for Scraping Multiple Pages from a Website [closed]
I am developing a web scraper to extract data (such as titles, authors, and prices) from a website that has potentially up to 2100 pages. I’m seeking advice on the most efficient strategy for handling ...
-3
votes
0
answers
16
views
Got_Error_403_after_few_requests [closed]
Request failed:
HTTPSConnectionPool(host='www.lowes.com', port=443): Read timed out.
(read timeout=10)
Please help me with that issue, website block me after few request even i use proxies and ...
-2
votes
0
answers
63
views
ModuleNotFoundError: No module named 'requests' in vs code
I have installed the latest version of requests as well as of pip. I also checked the pip list and in that, I found request, requests-file and requests-toolbelt all three with the latest version, but ...
-2
votes
2
answers
149
views
+50
A script using the requests module produces a 428 status code
I've created a script to issue a POST HTTP request with appropriate parameters, generated upon complying with the fields shown in this image, to this website. The script is supposed to produce a 200 ...
-1
votes
0
answers
55
views
SSL: CERTIFICATE_VERIFY_FAILED on Python requests call to local server
I am attempting to make an API call to an application on a local machine. It works perfectly when I set verify=False in my test environment, however my understanding is that this is not best practice ...
0
votes
1
answer
49
views
Is this the correct way of using sessions with multiprocessing in a process safe way in Python?
I am going to implement requests with sessions in my application, but I have read some threads concerning issues with thread and process safety, but those arent exactlty recent.
My application run 800+...
-2
votes
1
answer
48
views
How to get new cookies everytime from a password protected website?
I am trying to scrape a website that has APIs. One of the API gives json only if we are logged in.
I got logged in cookies and then send requests, I am successfully sending the request.
But the ...
0
votes
2
answers
44
views
How do I create a Python API client which executes *some* methods asynchronously in the background?
I have this (example) REST API client:
import requests
class FakeHttpClient:
base_url = "https://api.example.com"
def __init__(self) -> None:
self.session = requests....
-1
votes
1
answer
68
views
Translating IP to Domain name [closed]
Sorry if my question is kinda broad as I myself don't know the problem.
Long story short, I am given an IP adress and want to translate it to Domain name (if possible) so I thought about reverse DNS, ...
0
votes
0
answers
13
views
How do I pull data from GtR API using search term + python get request instead of search term in UI?
Can anyone help me understand why my code does not work? If you type a search term into the UI at https://gtr.ukri.org/ you get back a list of relevant projects, but using the exact same search term ...
0
votes
1
answer
43
views
Django REST Framework MultiPartParser parsing JSON as string
I am making a django functional based view where the request contains some JSON and a file.
Using DRF's MultiPartParser is parsing the JSON section of the request as a string.
what can I do to fix ...
0
votes
1
answer
28
views
HTTP Error 404 when scraping first table using BeautifulSoup, but second table works fine
I’m working on a Python script to scrape historical CDS data from Investing.com using BeautifulSoup. The goal is to extract data from a specific table on the page and compile it into a DataFrame.
Here’...
0
votes
0
answers
21
views
Scrapy returning 400 when request returns 200
I'm trying to scrape a website using Scrapy and I'm stuck because no matter what I do I cannot get a valid response. I need a proxy to access this website, and when I use Python's requests library, it ...
1
vote
0
answers
16
views
Solr delete multiple synonyms at once in managed resources
How do I do a bulk delete of some synonyms in the Managed Resource Api.
I find it strange that the documentation only talk about about how to delete
synonyms one at a time. It will be quite sad if we ...
0
votes
0
answers
71
views
Request exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I am making a Python app to fetch data from my website (which is written in PHP) as a stream.
main.py
import requests
from sseclient import SSEClient
def get_sse(url):
try:
# Make the ...
2
votes
1
answer
40
views
Failed to produce a JSON response containing a phone number based on a license number from a webpage using the requests module
I've created a script to fetch a phone number based on a license number from this webpage, using Python with the requests module. The script is supposed to produce a JSON response containing the phone ...
0
votes
0
answers
29
views
How to solve verification lbank API failed and use server timestamp in requests?
I'm trying use lbank API:
I set API and secret key
when I use API to get public data ( historical price of trading pairs , timstamp ...) it truly works.
but when I want to use authorized actions (...
0
votes
0
answers
18
views
Python binance-us doesn't return most recent klines / candles
I am trying to get recent candles from binance us. However, it doesn't return the most recent data and only returns from July of 2023.
First, here is my code:
def get_candles(self, symbol = "...
-2
votes
1
answer
52
views
downloading a file using python without knowing the file type (file name + file type)
I have been trying to download a file from the American Gut Project in qiita (study number 10317).
I can download directly from the web, however I want to create a python code to download the files ...
0
votes
0
answers
19
views
Python requests: No module named 'werkzeug.wrappers.json'
I'm trying to use the requests library in Python, but when I do so, I get an error
from werkzeug.wrappers.json import JSONMixin
ModuleNotFoundError: No module named 'werkzeug.wrappers.json'
Below ...
0
votes
0
answers
26
views
How can I make my llm process the whole JSON instead of only the last four entries?
I´m currently working on a local Chatbot using Langchain and Ollama.
The Chatbot should answer questions based on the data on my FROST-Server.
My implementation so far is able to send out a GET-...
-1
votes
0
answers
19
views
Are response headers still used if headers argument is sent for a requests session call?
I know the proper way to use a session is
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'}
s = requests....
-1
votes
0
answers
19
views
decoding data with ocecet-stream encryption
I was checking the web requests of a site and it had octet-stream in Content/Type the payload sent was encrypted I wanna know what that payload has it is probably in json format
U tried searching and ...
-1
votes
0
answers
32
views
How can I get the debt of Germany using the worldbank API?
I tried this, but I only gives a value from 1990:
import requests
def get_world_bank_data(indicator_code):
url = f"https://api.worldbank.org/v2/country/DEU/indicator/{indicator_code}?format=...
0
votes
1
answer
58
views
Trying to run a python script that uses the 'requests' module in on windows 10
I'm having the following error produced every time I try to run the grabber.py script. Any help would be greatly appreciated.
Traceback (most recent call last):
File "C:\Users\dolgi\...
0
votes
1
answer
36
views
TelegramBotAPI meaning of “attach://<file_attach_name>”
I need clarification for the TelegramBotAPI section about sending files : https://core.telegram.org/bots/api#inputmediaphoto
More specifically, I'm trying to use python and requests to perform a ...
3
votes
1
answer
30
views
Accessing pokeapi - taking a long time
I want to get a list of names of pokemon from the first 150 that have a hp less than a particular value.
Here's what I've got so far:
def get_pokemon_with_similar_hp(max_hp):
pokemon_names = []
...
0
votes
1
answer
100
views
Making an Azure AI API request with the content being a big dictionary
I'm a newbie and I need to make a request like in the example below, but my issue is that 'my_dict' far to big and I keep getting '400 Bad Request' errors, and when I try to split it and make multiple ...
0
votes
0
answers
35
views
Making an api for tornado.com stock borker
I want to create an API for logging in and viewing things in the tornado.com stock exchange. I am using Python and trying to set the CSRF and send the data but I am getting a 404 error which I think ...
1
vote
0
answers
65
views
Has this Roblox API been discontinued, or why did my code break?
I'm trying to write an application in python that has the purpose of handling multiple Roblox instances on Android emulators including cookie validation, login, joining games etc... I have the ...
0
votes
1
answer
64
views
get response from the HTTP triggered azure function without part processing
i have a time triggered function and another function with http trigger who are hosted in the same azure function. I want to raise a http request to the http triggered function from the time triggered ...
0
votes
2
answers
79
views
How do I download a json file with 50K+ lines faster? [closed]
I am 100% sure this is not my internet connection/speed issue. I want to download a text/json file with, let's say, 10K+ lines (24 MB) from GitHub, using Python, but the download takes too long, ...
-1
votes
0
answers
40
views
401 Unauthorized Error, but can POST with curl
hoping someone can help here.
This curl command works with no problem:
curl -v http://website.com \
-b "token1=BUNCHOFNUMBERS; token2=BUNCHOFOTHERNUMBERS" \
-H "Content-Type: ...
-1
votes
0
answers
19
views
how to login using Requests in python, on this site https://pocketoption.com/pt/login
I've tried several ways to log in using requests, but I couldn't.
Can anyone help me with the correct way?
the website is https://pocketoption.com/pt/login
I need to log in, then get the token to ...
0
votes
0
answers
25
views
What is the difference in aiohttp default Content-Type vs requests module's defaults?
I am working with an API that requires RSA256 public key POSTed as bytes. I was using requests (at first for testing) and aiohttp (to scale it up for many requests) and noticed a difference that I am ...
0
votes
0
answers
40
views
Is there an issue with parsing UUIDs for GraphQL requests?
I am trying to make some automatic GraphQL API tests using the Starlette TestClient in Python. My issue is that my GraphQL schema has a query that needs a UUID! as an input, say
testQuery(id: UUID!): ...
-1
votes
0
answers
17
views
LinkedIn Community Management API - Competitor's Analysis
is it possible to get insights about your competitors from the API the very same insights you get from the Analytics Dashboard Competitors?
enter image description here
LinkedIn API endpoints
...
-2
votes
0
answers
36
views
I cannot resolve recaptchav3 when logging in to the site
I am writing a code that is logged in to a site, I am using the solver service, but although it solves it, I get a not solved error why
import requests
import time
s = requests.Session()
site_key =&...
0
votes
1
answer
52
views
Getting errors trying to use python requests connection
trying to make a connection using 4 arguments.
headers = {
'Content-Type': 'application/json;charset=UTF-8',
'Cookie': 'A'}
body = {
"grant_type": "client_credentials",
"...
0
votes
2
answers
41
views
Downloading image via a blob:https
I'm trying to download images which are at an url such as blob:https://, however using requests and urllib does not work. Both of them crash when making the request. urllib gives the error code "&...
-3
votes
0
answers
37
views
Troubleshooting Cloudflare Interception and API Request Failures in Python
I've encountered an issue when making POST requests to certain endpoints using Python's requests library. Cloudflare is intercepting these requests, resulting in a 403 Forbidden error. For endpoint A, ...
0
votes
1
answer
47
views
Scraping Youtube video
I want to scrape a YouTube video using the requests module, but after I obtain the video and audio links, the request's status code is always 403 or I get no response. If I don't add 'Content-Length' ...
0
votes
1
answer
26
views
Azure function requests call to CPanel hosted python Flask API giving an "Verifying that you are not a robot"
I am running a Python service in Azure Functions that makes a request using requests module to a Flask API hosted on cPanel. However, I receive a response with the text "Verifying that you are ...
1
vote
1
answer
41
views
Zip file is only partially downloaded
I am trying to download a zip file with Python 3 requests:
try:
r = requests.get(BASEURL, stream=True)
with open(localfiletitle, 'wb') as fd:
shutil.copyfileobj(r.raw, fd)
except ...
1
vote
1
answer
47
views
Adding a default image when json returns null / None
Is it possible to add a default image to this code when the result returned for avatarUrl from JSON is null?
@staticmethod
def score_to_mod(score):
return floor((score - 10) / 2)
def __init__(...