Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

Where do I look to find a domain's policy on web scraping? [closed]

I often use bs4 and requests in python to gather data from a website. Where can I find the policy on data collection? (Keywords, Sections, shortcuts, etc.) I'm hesitant to post any of my code on the ...
Duby's user avatar
  • 45
0 votes
1 answer
23 views

How to Fetch GraphQL JSON Data from a URL in Python?

I am trying to fetch GraphQL data from the URL: https://URL I’ve tried using Python’s requests library to make a POST request, but I’m not sure how to structure the request to get the desired JSON ...
Mary's user avatar
  • 797
0 votes
1 answer
43 views

Why is Python httpx.get or requests.get much slower than cURL for this API?

For a home automation project I am trying to pull train delay data. An API wrapper exists, with cURL examples. These work fine, but both Python's requests.get and httpx.get are slow to pull data (up ...
0xDEADBEEF's user avatar
0 votes
0 answers
16 views

Python requests.get method stuck on audio streams [duplicate]

requests.get gets stuck on streams, like, for example, https://centova.radios.com.pt/proxy/476?mp=/stream https://nl.digitalrm.pt:8216/stream Is there any way to detect this automatically and abort? ...
user27333936's user avatar
0 votes
0 answers
27 views

How to download a image using requests

I have to download and image using Python requests. I logged into the website and followed each session when I clicked the image link from the network section, I got a post session with URL. ("...
Tonin thomas's user avatar
-4 votes
0 answers
31 views

Efficient Strategies for Scraping Multiple Pages from a Website [closed]

I am developing a web scraper to extract data (such as titles, authors, and prices) from a website that has potentially up to 2100 pages. I’m seeking advice on the most efficient strategy for handling ...
Shaimoom Shahriar's user avatar
-3 votes
0 answers
16 views

Got_Error_403_after_few_requests [closed]

Request failed: HTTPSConnectionPool(host='www.lowes.com', port=443): Read timed out. (read timeout=10) Please help me with that issue, website block me after few request even i use proxies and ...
Abdullah Zubair's user avatar
-2 votes
0 answers
63 views

ModuleNotFoundError: No module named 'requests' in vs code

I have installed the latest version of requests as well as of pip. I also checked the pip list and in that, I found request, requests-file and requests-toolbelt all three with the latest version, but ...
Jaskaran Singh's user avatar
-2 votes
2 answers
149 views
+50

A script using the requests module produces a 428 status code

I've created a script to issue a POST HTTP request with appropriate parameters, generated upon complying with the fields shown in this image, to this website. The script is supposed to produce a 200 ...
robots.txt's user avatar
-1 votes
0 answers
55 views

SSL: CERTIFICATE_VERIFY_FAILED on Python requests call to local server

I am attempting to make an API call to an application on a local machine. It works perfectly when I set verify=False in my test environment, however my understanding is that this is not best practice ...
Sam Cohen-Devries's user avatar
0 votes
1 answer
49 views

Is this the correct way of using sessions with multiprocessing in a process safe way in Python?

I am going to implement requests with sessions in my application, but I have read some threads concerning issues with thread and process safety, but those arent exactlty recent. My application run 800+...
Welsige's user avatar
  • 151
-2 votes
1 answer
48 views

How to get new cookies everytime from a password protected website?

I am trying to scrape a website that has APIs. One of the API gives json only if we are logged in. I got logged in cookies and then send requests, I am successfully sending the request. But the ...
Ali Ahsen's user avatar
0 votes
2 answers
44 views

How do I create a Python API client which executes *some* methods asynchronously in the background?

I have this (example) REST API client: import requests class FakeHttpClient: base_url = "https://api.example.com" def __init__(self) -> None: self.session = requests....
g_t_m's user avatar
  • 714
-1 votes
1 answer
68 views

Translating IP to Domain name [closed]

Sorry if my question is kinda broad as I myself don't know the problem. Long story short, I am given an IP adress and want to translate it to Domain name (if possible) so I thought about reverse DNS, ...
Moran's user avatar
  • 15
0 votes
0 answers
13 views

How do I pull data from GtR API using search term + python get request instead of search term in UI?

Can anyone help me understand why my code does not work? If you type a search term into the UI at https://gtr.ukri.org/ you get back a list of relevant projects, but using the exact same search term ...
Harriet Downing's user avatar
0 votes
1 answer
43 views

Django REST Framework MultiPartParser parsing JSON as string

I am making a django functional based view where the request contains some JSON and a file. Using DRF's MultiPartParser is parsing the JSON section of the request as a string. what can I do to fix ...
Abdullah Shekfeh's user avatar
0 votes
1 answer
28 views

HTTP Error 404 when scraping first table using BeautifulSoup, but second table works fine

I’m working on a Python script to scrape historical CDS data from Investing.com using BeautifulSoup. The goal is to extract data from a specific table on the page and compile it into a DataFrame. Here’...
jaokz's user avatar
  • 5
0 votes
0 answers
21 views

Scrapy returning 400 when request returns 200

I'm trying to scrape a website using Scrapy and I'm stuck because no matter what I do I cannot get a valid response. I need a proxy to access this website, and when I use Python's requests library, it ...
Ariel's user avatar
  • 3,591
1 vote
0 answers
16 views

Solr delete multiple synonyms at once in managed resources

How do I do a bulk delete of some synonyms in the Managed Resource Api. I find it strange that the documentation only talk about about how to delete synonyms one at a time. It will be quite sad if we ...
Baama's user avatar
  • 2,600
0 votes
0 answers
71 views

Request exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I am making a Python app to fetch data from my website (which is written in PHP) as a stream. main.py import requests from sseclient import SSEClient def get_sse(url): try: # Make the ...
blazgocompany's user avatar
2 votes
1 answer
40 views

Failed to produce a JSON response containing a phone number based on a license number from a webpage using the requests module

I've created a script to fetch a phone number based on a license number from this webpage, using Python with the requests module. The script is supposed to produce a JSON response containing the phone ...
robots.txt's user avatar
0 votes
0 answers
29 views

How to solve verification lbank API failed and use server timestamp in requests?

I'm trying use lbank API: I set API and secret key when I use API to get public data ( historical price of trading pairs , timstamp ...) it truly works. but when I want to use authorized actions (...
mohammad's user avatar
0 votes
0 answers
18 views

Python binance-us doesn't return most recent klines / candles

I am trying to get recent candles from binance us. However, it doesn't return the most recent data and only returns from July of 2023. First, here is my code: def get_candles(self, symbol = "...
finny's user avatar
  • 1
-2 votes
1 answer
52 views

downloading a file using python without knowing the file type (file name + file type)

I have been trying to download a file from the American Gut Project in qiita (study number 10317). I can download directly from the web, however I want to create a python code to download the files ...
נדב לישע's user avatar
0 votes
0 answers
19 views

Python requests: No module named 'werkzeug.wrappers.json'

I'm trying to use the requests library in Python, but when I do so, I get an error from werkzeug.wrappers.json import JSONMixin ModuleNotFoundError: No module named 'werkzeug.wrappers.json' Below ...
crobar's user avatar
  • 2,927
0 votes
0 answers
26 views

How can I make my llm process the whole JSON instead of only the last four entries?

I´m currently working on a local Chatbot using Langchain and Ollama. The Chatbot should answer questions based on the data on my FROST-Server. My implementation so far is able to send out a GET-...
user27159834's user avatar
-1 votes
0 answers
19 views

Are response headers still used if headers argument is sent for a requests session call?

I know the proper way to use a session is headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'} s = requests....
Brian's user avatar
  • 1
-1 votes
0 answers
19 views

decoding data with ocecet-stream encryption

I was checking the web requests of a site and it had octet-stream in Content/Type the payload sent was encrypted I wanna know what that payload has it is probably in json format U tried searching and ...
Op Game's user avatar
-1 votes
0 answers
32 views

How can I get the debt of Germany using the worldbank API?

I tried this, but I only gives a value from 1990: import requests def get_world_bank_data(indicator_code): url = f"https://api.worldbank.org/v2/country/DEU/indicator/{indicator_code}?format=...
Ray Hulha's user avatar
  • 11.1k
0 votes
1 answer
58 views

Trying to run a python script that uses the 'requests' module in on windows 10

I'm having the following error produced every time I try to run the grabber.py script. Any help would be greatly appreciated. Traceback (most recent call last): File "C:\Users\dolgi\...
Aaron Dolgin's user avatar
0 votes
1 answer
36 views

TelegramBotAPI meaning of “attach://<file_attach_name>”

I need clarification for the TelegramBotAPI section about sending files : https://core.telegram.org/bots/api#inputmediaphoto More specifically, I'm trying to use python and requests to perform a ...
Hugo's user avatar
  • 1,284
3 votes
1 answer
30 views

Accessing pokeapi - taking a long time

I want to get a list of names of pokemon from the first 150 that have a hp less than a particular value. Here's what I've got so far: def get_pokemon_with_similar_hp(max_hp): pokemon_names = [] ...
NucyLoodle's user avatar
0 votes
1 answer
100 views

Making an Azure AI API request with the content being a big dictionary

I'm a newbie and I need to make a request like in the example below, but my issue is that 'my_dict' far to big and I keep getting '400 Bad Request' errors, and when I try to split it and make multiple ...
Iluvatar Bombadil's user avatar
0 votes
0 answers
35 views

Making an api for tornado.com stock borker

I want to create an API for logging in and viewing things in the tornado.com stock exchange. I am using Python and trying to set the CSRF and send the data but I am getting a 404 error which I think ...
Solomon P Byer's user avatar
1 vote
0 answers
65 views

Has this Roblox API been discontinued, or why did my code break?

I'm trying to write an application in python that has the purpose of handling multiple Roblox instances on Android emulators including cookie validation, login, joining games etc... I have the ...
Jannis's user avatar
  • 11
0 votes
1 answer
64 views

get response from the HTTP triggered azure function without part processing

i have a time triggered function and another function with http trigger who are hosted in the same azure function. I want to raise a http request to the http triggered function from the time triggered ...
Akash Singh's user avatar
0 votes
2 answers
79 views

How do I download a json file with 50K+ lines faster? [closed]

I am 100% sure this is not my internet connection/speed issue. I want to download a text/json file with, let's say, 10K+ lines (24 MB) from GitHub, using Python, but the download takes too long, ...
Jonathan Levi's user avatar
-1 votes
0 answers
40 views

401 Unauthorized Error, but can POST with curl

hoping someone can help here. This curl command works with no problem: curl -v http://website.com \ -b "token1=BUNCHOFNUMBERS; token2=BUNCHOFOTHERNUMBERS" \ -H "Content-Type: ...
Ashley Rice's user avatar
-1 votes
0 answers
19 views

how to login using Requests in python, on this site https://pocketoption.com/pt/login

I've tried several ways to log in using requests, but I couldn't. Can anyone help me with the correct way? the website is https://pocketoption.com/pt/login I need to log in, then get the token to ...
Lucas Feix's user avatar
0 votes
0 answers
25 views

What is the difference in aiohttp default Content-Type vs requests module's defaults?

I am working with an API that requires RSA256 public key POSTed as bytes. I was using requests (at first for testing) and aiohttp (to scale it up for many requests) and noticed a difference that I am ...
Coldchain9's user avatar
  • 1,665
0 votes
0 answers
40 views

Is there an issue with parsing UUIDs for GraphQL requests?

I am trying to make some automatic GraphQL API tests using the Starlette TestClient in Python. My issue is that my GraphQL schema has a query that needs a UUID! as an input, say testQuery(id: UUID!): ...
Jonas's user avatar
  • 53
-1 votes
0 answers
17 views

LinkedIn Community Management API - Competitor's Analysis

is it possible to get insights about your competitors from the API the very same insights you get from the Analytics Dashboard Competitors? enter image description here LinkedIn API endpoints ...
Shubham Sharan's user avatar
-2 votes
0 answers
36 views

I cannot resolve recaptchav3 when logging in to the site

I am writing a code that is logged in to a site, I am using the solver service, but although it solves it, I get a not solved error why import requests import time s = requests.Session() site_key =&...
muhahahaha's user avatar
0 votes
1 answer
52 views

Getting errors trying to use python requests connection

trying to make a connection using 4 arguments. headers = { 'Content-Type': 'application/json;charset=UTF-8', 'Cookie': 'A'} body = { "grant_type": "client_credentials", "...
girish's user avatar
  • 1
0 votes
2 answers
41 views

Downloading image via a blob:https

I'm trying to download images which are at an url such as blob:https://, however using requests and urllib does not work. Both of them crash when making the request. urllib gives the error code "&...
Taxenmaxen's user avatar
-3 votes
0 answers
37 views

Troubleshooting Cloudflare Interception and API Request Failures in Python

I've encountered an issue when making POST requests to certain endpoints using Python's requests library. Cloudflare is intercepting these requests, resulting in a 403 Forbidden error. For endpoint A, ...
Not Sul's user avatar
0 votes
1 answer
47 views

Scraping Youtube video

I want to scrape a YouTube video using the requests module, but after I obtain the video and audio links, the request's status code is always 403 or I get no response. If I don't add 'Content-Length' ...
zaber8787利巴's user avatar
0 votes
1 answer
26 views

Azure function requests call to CPanel hosted python Flask API giving an "Verifying that you are not a robot"

I am running a Python service in Azure Functions that makes a request using requests module to a Flask API hosted on cPanel. However, I receive a response with the text "Verifying that you are ...
Hakuna N's user avatar
  • 181
1 vote
1 answer
41 views

Zip file is only partially downloaded

I am trying to download a zip file with Python 3 requests: try: r = requests.get(BASEURL, stream=True) with open(localfiletitle, 'wb') as fd: shutil.copyfileobj(r.raw, fd) except ...
Fabio Marzocca's user avatar
1 vote
1 answer
47 views

Adding a default image when json returns null / None

Is it possible to add a default image to this code when the result returned for avatarUrl from JSON is null? @staticmethod def score_to_mod(score): return floor((score - 10) / 2) def __init__(...
Patrick's user avatar
  • 13

1
2 3 4 5
443