Questions tagged [httpx]
HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
161
questions
0
votes
1
answer
56
views
How to stop your source IP address being seen by a site while using proxies?
I'm trying to scrape some data off a site using proxies, I won't put any code here since it is a bit long and maybe not important for the question. I can just list these facts:
The code I have does ...
0
votes
0
answers
28
views
How can this scraping structure be improved to get a better control of the proxies used and the simultaneous tasks running?
I'm a web scraping enthusiast that just learnt about asyncio and proxies in web scraping to improve performance. The program I wrote did its job of downloading a big number of sites pretty fast but ...
0
votes
0
answers
92
views
Making requests asynchronously using httpx not much faster when getting big zip files
I'm having an interesting problem where using httpx to make requests asynchronously doesn't improve runtimes over making requests synchronously when getting big-ish zip files.
This is my current test ...
0
votes
1
answer
70
views
Handling POST Request in expedia.com
I came across this post, the answer to this question shows how to handle POST request for hotels.com
here is the code made by αԋɱҽԃ αмєяιcαη
import trio
import httpx
import pandas as pd
async def ...
0
votes
0
answers
24
views
DigestAuthentication doesn't work with httpx
It seems to be an issue with Digest Authentication when using httpx. Even with a basic HTTP server, I can't manage to correctly authenticate a client with httpx.
Here's the server implementation with ...
0
votes
0
answers
33
views
Clean httpclient httpx class
I would like to create a custom asynchronous Httpclient class while implementing the above code in a fastapi api which needs to type 2 or 3 other api.
As mentioned and explained in the answer in the ...
0
votes
1
answer
37
views
'Task got bad yield' when wrapping streaming response to use as a normal generator
I have a FastAPI backend with a POST endpoint that response with a streaming response and I need to wrap the call to the endpoint in a way that I can use the response as a normal (not async) iterator. ...
0
votes
1
answer
49
views
async httpx requests fails when using redis cache backend
I'm running a FastAPI that makes some external calls to fetch data and it use the cache decorator to store the auth tokens; The app works with in-memory and Redis Backend (usually in-memory for local ...
-1
votes
1
answer
77
views
Python's requests triggers Cloudflare
import requests , json
from collections import OrderedDict
def Core():
response = requests.Session()
headers = OrderedDict({'Authorization': 'bearer 0',
...
0
votes
0
answers
46
views
httpx version 0.13.3 [Errno 13] Permission denied: '/root/.netrc' in production server AWS EC2
I have a simple GET request to an endpoint on a Flask server using httpx version 0.13.3.
async with httpx.AsyncClient(auth=auth) as client:
response = await client.get(
url = f&...
0
votes
1
answer
211
views
"Runtime Error: Event loop is closed" during testing using Pytest
Please help me. How to fix it? The error appears only when using cookies in test_logout()
conftest.py
...............
@pytest_asyncio.fixture(autouse=True, scope='session')
async def prepare_database(...
0
votes
1
answer
31
views
How to getting the certificate directly from the WinStore in bytes to send to the request?
I have an API which accepts certificate signed requests.
At the moment I do it this way
response = requests.post(some_url, data=some_data, cert=path_to_cert)
For security reasons, I was obliged not to ...
1
vote
1
answer
39
views
Why the response of httpx is different compared to inspect element network?
I am trying to scrape this site
https://shopee.co.id/S-B-Golden-Curry-Bumbu-Kari-Japanese-Curry-Mix-Medium-Hot-220-gr-i.65323877.3343602079?sp_atk=0b471505-cfca-491b-b7a2-453d455eecf6&xptdk=...
1
vote
2
answers
55
views
Selenium delay when python webscraping
I am running a web scraping task on Invest's website, the code below has always worked and started to give an error for a reason I am not identifying:
from bs4 import BeautifulSoup
from lxml import ...
1
vote
0
answers
78
views
Python HTTPX post Request timeout
We have a python 3.8 application that monitor others application and generate metrics for grafana
Configuration:
python 3.8,
httpx 0.24.1
httpcore 0.17.2
Sometimes a check on applications failed with ...
0
votes
1
answer
121
views
Cannot import name 'SOCKET_OPTION' from 'httpcore.backends.base
I am trying to make an httpx request from an API, this is my code. I have upgraded httpcore, but still same error. I also uninstalled and installed httpcore but to no avail, it still returns the same ...
0
votes
1
answer
62
views
Python Async function not working. Just keep running
The problem is When running the code it doesn't finish
it just print the index and stops at the end
What I am doing wrong?
limits = httpx.Limits(max_keepalive_connections=5, max_connections=10)
...
0
votes
1
answer
98
views
httpx.RemoteProtocolError while processign 1Gb+ file
My code downloads huge file with httpx and process it's chunks on the fly
async with httpx.AsyncClient() as client:
async with client.stream("GET", self.url, follow_redirects=True, ...
0
votes
0
answers
101
views
How to asynchronously query dynamodb table using python
I am using the aiodynamo library to query a dynamodb table, but the current code I have is slow. Can someone please suggest modifications or point out errors?
The table is queried with partitionKey=...
1
vote
1
answer
31
views
Iterating through requests until return is none (Python)
I am working with ticket inventory for various events that have an unknown total number of listings. For each listing, I get a return of some data such as price, quantity, owning user, etc. To ...
2
votes
1
answer
228
views
HTTPX RESPX Pytest TypeError: Invalid type for url. Expected str or httpx.URL, got <class 'tuple'>:
I have a function in my Python class that works fine when I use it in my other .py file.
@exception_handler
def get_all_workspaces(self) -> Union[WorkspacesModel, GSResponse]:
Client = self....
0
votes
3
answers
482
views
How do I make sure httpx calls are run in parallel?
I was recommended httpx as a way to perform api requests in parallel, with a nice api like requests.
my code
import asyncio
import time
import httpx
async def main():
t0 = time.time()
...
0
votes
1
answer
49
views
httpx/URL is changing original URL
If I have %20 in URL before, it ends up with + in the code.
url='http://www.example.com?param=a%20b'
url=URL(url)
url=url.copy_merge_params(params={})
In QueryParams.str it's invoking urlencode() ...
0
votes
0
answers
31
views
How to reuse class in async Django view
I've made a class with requests inherited from httpx:
class URLRequest:
"""URL request class."""
def __init__(self, client):
self.client = httpx....
1
vote
1
answer
172
views
HTTPX returns 403, but requests 200. Python
I'm trying to parse one site. I'm sending a naked http request, just by setting a User-Agent.
It's surprising how a request using "requests" returns a 200 response. But the same query using ...
0
votes
1
answer
224
views
httpx async http2 errorhandling slow down and retry on error
When server gets overloaded it stats dropping, but my program then drops everything left in jobs. 2-300 lines of the below error is common (same last_stream_id):
20230428143422.761|ERROR|gitlab.py:293|...
1
vote
0
answers
86
views
How to use csv.reader in an async context?
I query multiple services with asynchronously using httpx. These services return csv data that could be very large so I'm using streams.
So far so good.
The problem I'm having is that the Python ...
1
vote
0
answers
140
views
HTTPx Async Client not working when uploading file to FastAPI endpoint [duplicate]
I am using a HTTPx client.
See this documentation for wrapping a FastAPI app in a HTTPx Async Client.
https://fastapi.tiangolo.com/advanced/async-tests/#example
It is just, not with block, I return ...
1
vote
1
answer
475
views
How to test a FastAPI route that retries a SQLAlchemy insert after a rollback?
I have a route where I want to retry an insert if it failed due to an IntegrityError. I’m trying to test it using pytest and httpx but I get an error when I reuse the session to retry the insert after ...
0
votes
1
answer
92
views
How to check call count in HTTPX library
Using requests_mock I can check if it was called with call_count attribute like in the example below:
def test_foo(
self,
response,
mock_service,
):
assert mock_service.call_count == 1
...
0
votes
1
answer
292
views
What exceptions can be raised by Python HTTPX's json() method?
The excellent Python HTTPX package has a .json() method for conveniently decoding resposnes that are in JSON format. But the documentation does not mention what exceptions can be raised by .json(), ...
0
votes
1
answer
888
views
Disable ssl verification on a third party module using HTTPX
I'm using a module that uses httpx to make a request. Because of my company's network policy, I get a CERTIFICATE_VERIFY_FAILED every time I try to run my code. The data I'm sending and receiving in ...
0
votes
0
answers
243
views
Response failing in httpx but not in requests
I am making a POST request to a URL using 'httpx' library. However, I get a 401 unauthorized error with my below request
cert = os.path.realpath('./certs/certificate.pem')
key = os.path.realpath('./...
1
vote
2
answers
361
views
python httpx use --compressed like curl
How can I use httpx library to do something similar to
curl --compressed "http://example.com"
Meaning I want the library to check if server supports compression, if so send the right header ...
1
vote
1
answer
179
views
How to make requests in parallel using FastAPI [duplicate]
In FastAPI, I have this route:
for id in ids: #get projects from list of ids
p = await gitlab.project(id)
if p and 'error' not in p:
projects[int(id)] = p
But it takes around 2sec per ...
0
votes
0
answers
129
views
How to do async python with max concurrency, sequential processing, and error handling?
I am working on a hook that will process a batch of http requests asynchronously.
The hook should be able to do the following:
Send requests concurrently up to a max concurrency
Process requests in ...
0
votes
1
answer
1k
views
Can't test Post request with FastAPI & Pytest [duplicate]
I'm trying to test my /login API with FastAPI's Testclient.
But when I pass data to the post api. It shows, 422 error with content username and password fields are required.
API:
@router.post('/...
1
vote
1
answer
220
views
httpx POST request always gives "401 Unauthorized" before working fine the second attempt
I'm using async with httpx in a python script to make an HTTP POST request using DigestAuth.
async with httpx.AsyncClient() as client:
try:
r = await client.post(f"http://{str(...
1
vote
1
answer
556
views
using httpx to send 100K get requests
I'm using the httpx library and asyncio to try and send about 100K of get requests.
I ran the code and received httpx.ConnectError so I opened wireshark and saw that I was getting a lot of messages ...
0
votes
1
answer
169
views
proxies+verify kwargs break HTTP/2 in httpx
Versions: httpx 0.23.3 and httpcore 0.16.3.
import httpx
s=httpx.Client(
http2=True, proxies='http://127.0.0.1:9000', verify=False
)
r=s.get('https://in.indeed.com')
print(r.http_version)
...
0
votes
0
answers
21
views
I want to send a POST request with really long headers. Is there an easy way to do this for a string until the entire sting has been sent in python?
params = {
"description": description
}
The description variable above is about 9000 characters, which is too long for the API I want to POST to.
I think sending multiple requests in small ...
1
vote
0
answers
159
views
SqlAlchemy select hangs in FastAPI test setup
I developed a FastAPI app with WebSockets and I'm trying to test it. I need async tests in order to check data in the database during tests. My setup looks like this: I have a fixture that generates a ...
0
votes
1
answer
253
views
Handle Request payload using HTTPX for post data and capcha key to google recapchaV2
I'm trying to send the key I got from anti captcha, an example of the key is like this
{
"errorId":0,
"status":"ready",
"solution":
{
&...
0
votes
0
answers
185
views
Python-Authlib: setting an Oauth2 Async client with (service principal) and (user, password)
I come here for help after being stuck a few hours creating an OAuth2 Async httpx client.
It's one of those situations that feels so close, and still doesn't come through in the end.
I'm using this ...
0
votes
0
answers
99
views
How to write unit test case for fastapi for mysql
app.py
def get_db(request: Request):
return request.state.db
@router.get("/data")
async def endpoint(
db:Session = Depends(get_db),
):
try:
data = await crud....
0
votes
1
answer
78
views
Scrape website using httpx and requests returns a timeout
I am trying to scrape this website
https://www.blibli.com/p/facial-tissue-tisu-wajah-250-s-paseo/is--LO1-70001-00049-00003?seller_id=LO1-70001&sku_id=LO1-70001-00049-00001&sclid=...
3
votes
1
answer
1k
views
Fastapi Testclient not able to send POST request using form-data
Currently I am doing Unit Testing in Fastapi using from fastapi.testclient import TestClient
def test_login_api_returns_token(session,client):
form_data = {
"username": "...
1
vote
1
answer
583
views
Deprecation error in venv/lib/python3.8/site-packages/httpx/_content.py:201:DeprecationWarning: Use 'content=<...>' to upload raw bytes/text content
I installed a project that has httpx in the code and it seems that there is a method deprecated in its code. I looked for a solution but I don't find any solution for it. ( Also, I have run it before ...
0
votes
0
answers
32
views
Why is the response sometimes passing through and sometimes not when using requests httpx in python?
I am trying to scrape this website items, however when I used httpx or even requests sometimes it passes and gets the response sometimes it doesn't. It seems random, that's why I tried doing a rerun ...
2
votes
1
answer
587
views
Monkeypatching/mocking the HTTPX external requests
I'm trying to monkeypatch the external request. Here is the code of a web endpoint:
import httpx, json
...
@app.get('/test')
async def view_test(request):
async with httpx.AsyncClient() as client:
...