Questions tagged [httpx]

HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.

Filter by
Sorted by
Tagged with
0 votes
1 answer
56 views

How to stop your source IP address being seen by a site while using proxies?

I'm trying to scrape some data off a site using proxies, I won't put any code here since it is a bit long and maybe not important for the question. I can just list these facts: The code I have does ...
DoctorEvil's user avatar
0 votes
0 answers
28 views

How can this scraping structure be improved to get a better control of the proxies used and the simultaneous tasks running?

I'm a web scraping enthusiast that just learnt about asyncio and proxies in web scraping to improve performance. The program I wrote did its job of downloading a big number of sites pretty fast but ...
sushiwithoutsushi's user avatar
0 votes
0 answers
92 views

Making requests asynchronously using httpx not much faster when getting big zip files

I'm having an interesting problem where using httpx to make requests asynchronously doesn't improve runtimes over making requests synchronously when getting big-ish zip files. This is my current test ...
Ethan Singer's user avatar
0 votes
1 answer
70 views

Handling POST Request in expedia.com

I came across this post, the answer to this question shows how to handle POST request for hotels.com here is the code made by αԋɱҽԃ αмєяιcαη import trio import httpx import pandas as pd async def ...
Dev Burman's user avatar
0 votes
0 answers
24 views

DigestAuthentication doesn't work with httpx

It seems to be an issue with Digest Authentication when using httpx. Even with a basic HTTP server, I can't manage to correctly authenticate a client with httpx. Here's the server implementation with ...
RobBlanchard's user avatar
0 votes
0 answers
33 views

Clean httpclient httpx class

I would like to create a custom asynchronous Httpclient class while implementing the above code in a fastapi api which needs to type 2 or 3 other api. As mentioned and explained in the answer in the ...
Walid's user avatar
  • 1
0 votes
1 answer
37 views

'Task got bad yield' when wrapping streaming response to use as a normal generator

I have a FastAPI backend with a POST endpoint that response with a streaming response and I need to wrap the call to the endpoint in a way that I can use the response as a normal (not async) iterator. ...
thomas's user avatar
  • 624
0 votes
1 answer
49 views

async httpx requests fails when using redis cache backend

I'm running a FastAPI that makes some external calls to fetch data and it use the cache decorator to store the auth tokens; The app works with in-memory and Redis Backend (usually in-memory for local ...
Carlos Rojas's user avatar
-1 votes
1 answer
77 views

Python's requests triggers Cloudflare

import requests , json from collections import OrderedDict def Core(): response = requests.Session() headers = OrderedDict({'Authorization': 'bearer 0', ...
John Duo's user avatar
0 votes
0 answers
46 views

httpx version 0.13.3 [Errno 13] Permission denied: '/root/.netrc' in production server AWS EC2

I have a simple GET request to an endpoint on a Flask server using httpx version 0.13.3. async with httpx.AsyncClient(auth=auth) as client: response = await client.get( url = f&...
Vidiskiu Fortino K's user avatar
0 votes
1 answer
211 views

"Runtime Error: Event loop is closed" during testing using Pytest

Please help me. How to fix it? The error appears only when using cookies in test_logout() conftest.py ............... @pytest_asyncio.fixture(autouse=True, scope='session') async def prepare_database(...
GADJIIAVOV's user avatar
0 votes
1 answer
31 views

How to getting the certificate directly from the WinStore in bytes to send to the request?

I have an API which accepts certificate signed requests. At the moment I do it this way response = requests.post(some_url, data=some_data, cert=path_to_cert) For security reasons, I was obliged not to ...
NonniSnon's user avatar
1 vote
1 answer
39 views

Why the response of httpx is different compared to inspect element network?

I am trying to scrape this site https://shopee.co.id/S-B-Golden-Curry-Bumbu-Kari-Japanese-Curry-Mix-Medium-Hot-220-gr-i.65323877.3343602079?sp_atk=0b471505-cfca-491b-b7a2-453d455eecf6&xptdk=...
Hal's user avatar
  • 123
1 vote
2 answers
55 views

Selenium delay when python webscraping

I am running a web scraping task on Invest's website, the code below has always worked and started to give an error for a reason I am not identifying: from bs4 import BeautifulSoup from lxml import ...
Leonardo Silva's user avatar
1 vote
0 answers
78 views

Python HTTPX post Request timeout

We have a python 3.8 application that monitor others application and generate metrics for grafana Configuration: python 3.8, httpx 0.24.1 httpcore 0.17.2 Sometimes a check on applications failed with ...
Vincent BILLARD's user avatar
0 votes
1 answer
121 views

Cannot import name 'SOCKET_OPTION' from 'httpcore.backends.base

I am trying to make an httpx request from an API, this is my code. I have upgraded httpcore, but still same error. I also uninstalled and installed httpcore but to no avail, it still returns the same ...
Ridwan Adejumo Suleiman's user avatar
0 votes
1 answer
62 views

Python Async function not working. Just keep running

The problem is When running the code it doesn't finish it just print the index and stops at the end What I am doing wrong? limits = httpx.Limits(max_keepalive_connections=5, max_connections=10) ...
Coding Dog's user avatar
0 votes
1 answer
98 views

httpx.RemoteProtocolError while processign 1Gb+ file

My code downloads huge file with httpx and process it's chunks on the fly async with httpx.AsyncClient() as client: async with client.stream("GET", self.url, follow_redirects=True, ...
Serge's user avatar
  • 1,416
0 votes
0 answers
101 views

How to asynchronously query dynamodb table using python

I am using the aiodynamo library to query a dynamodb table, but the current code I have is slow. Can someone please suggest modifications or point out errors? The table is queried with partitionKey=...
Alex's user avatar
  • 125
1 vote
1 answer
31 views

Iterating through requests until return is none (Python)

I am working with ticket inventory for various events that have an unknown total number of listings. For each listing, I get a return of some data such as price, quantity, owning user, etc. To ...
johne518's user avatar
2 votes
1 answer
228 views

HTTPX RESPX Pytest TypeError: Invalid type for url. Expected str or httpx.URL, got <class 'tuple'>:

I have a function in my Python class that works fine when I use it in my other .py file. @exception_handler def get_all_workspaces(self) -> Union[WorkspacesModel, GSResponse]: Client = self....
krishna lodha's user avatar
0 votes
3 answers
482 views

How do I make sure httpx calls are run in parallel?

I was recommended httpx as a way to perform api requests in parallel, with a nice api like requests. my code import asyncio import time import httpx async def main(): t0 = time.time() ...
Harry Moreno's user avatar
  • 10.3k
0 votes
1 answer
49 views

httpx/URL is changing original URL

If I have %20 in URL before, it ends up with + in the code. url='http://www.example.com?param=a%20b' url=URL(url) url=url.copy_merge_params(params={}) In QueryParams.str it's invoking urlencode() ...
aikipooh's user avatar
  • 137
0 votes
0 answers
31 views

How to reuse class in async Django view

I've made a class with requests inherited from httpx: class URLRequest: """URL request class.""" def __init__(self, client): self.client = httpx....
Vitalii Mytenko's user avatar
1 vote
1 answer
172 views

HTTPX returns 403, but requests 200. Python

I'm trying to parse one site. I'm sending a naked http request, just by setting a User-Agent. It's surprising how a request using "requests" returns a 200 response. But the same query using ...
Ivan's user avatar
  • 35
0 votes
1 answer
224 views

httpx async http2 errorhandling slow down and retry on error

When server gets overloaded it stats dropping, but my program then drops everything left in jobs. 2-300 lines of the below error is common (same last_stream_id): 20230428143422.761|ERROR|gitlab.py:293|...
MortenB's user avatar
  • 2,816
1 vote
0 answers
86 views

How to use csv.reader in an async context?

I query multiple services with asynchronously using httpx. These services return csv data that could be very large so I'm using streams. So far so good. The problem I'm having is that the Python ...
fanta fles's user avatar
1 vote
0 answers
140 views

HTTPx Async Client not working when uploading file to FastAPI endpoint [duplicate]

I am using a HTTPx client. See this documentation for wrapping a FastAPI app in a HTTPx Async Client. https://fastapi.tiangolo.com/advanced/async-tests/#example It is just, not with block, I return ...
cansusargin's user avatar
1 vote
1 answer
475 views

How to test a FastAPI route that retries a SQLAlchemy insert after a rollback?

I have a route where I want to retry an insert if it failed due to an IntegrityError. I’m trying to test it using pytest and httpx but I get an error when I reuse the session to retry the insert after ...
bfontaine's user avatar
  • 18.3k
0 votes
1 answer
92 views

How to check call count in HTTPX library

Using requests_mock I can check if it was called with call_count attribute like in the example below: def test_foo( self, response, mock_service, ): assert mock_service.call_count == 1 ...
Tony's user avatar
  • 3
0 votes
1 answer
292 views

What exceptions can be raised by Python HTTPX's json() method?

The excellent Python HTTPX package has a .json() method for conveniently decoding resposnes that are in JSON format. But the documentation does not mention what exceptions can be raised by .json(), ...
mhucka's user avatar
  • 2,149
0 votes
1 answer
888 views

Disable ssl verification on a third party module using HTTPX

I'm using a module that uses httpx to make a request. Because of my company's network policy, I get a CERTIFICATE_VERIFY_FAILED every time I try to run my code. The data I'm sending and receiving in ...
João Areias's user avatar
  • 1,212
0 votes
0 answers
243 views

Response failing in httpx but not in requests

I am making a POST request to a URL using 'httpx' library. However, I get a 401 unauthorized error with my below request cert = os.path.realpath('./certs/certificate.pem') key = os.path.realpath('./...
Vishnukk's user avatar
  • 524
1 vote
2 answers
361 views

python httpx use --compressed like curl

How can I use httpx library to do something similar to curl --compressed "http://example.com" Meaning I want the library to check if server supports compression, if so send the right header ...
Ehud Lev's user avatar
  • 2,491
1 vote
1 answer
179 views

How to make requests in parallel using FastAPI [duplicate]

In FastAPI, I have this route: for id in ids: #get projects from list of ids p = await gitlab.project(id) if p and 'error' not in p: projects[int(id)] = p But it takes around 2sec per ...
MortenB's user avatar
  • 2,816
0 votes
0 answers
129 views

How to do async python with max concurrency, sequential processing, and error handling?

I am working on a hook that will process a batch of http requests asynchronously. The hook should be able to do the following: Send requests concurrently up to a max concurrency Process requests in ...
JTa's user avatar
  • 181
0 votes
1 answer
1k views

Can't test Post request with FastAPI & Pytest [duplicate]

I'm trying to test my /login API with FastAPI's Testclient. But when I pass data to the post api. It shows, 422 error with content username and password fields are required. API: @router.post('/...
Fahad Md Kamal's user avatar
1 vote
1 answer
220 views

httpx POST request always gives "401 Unauthorized" before working fine the second attempt

I'm using async with httpx in a python script to make an HTTP POST request using DigestAuth. async with httpx.AsyncClient() as client: try: r = await client.post(f"http://{str(...
prime_number's user avatar
1 vote
1 answer
556 views

using httpx to send 100K get requests

I'm using the httpx library and asyncio to try and send about 100K of get requests. I ran the code and received httpx.ConnectError so I opened wireshark and saw that I was getting a lot of messages ...
Omer Dagry's user avatar
0 votes
1 answer
169 views

proxies+verify kwargs break HTTP/2 in httpx

Versions: httpx 0.23.3 and httpcore 0.16.3. import httpx s=httpx.Client( http2=True, proxies='http://127.0.0.1:9000', verify=False ) r=s.get('https://in.indeed.com') print(r.http_version) ...
aikipooh's user avatar
  • 137
0 votes
0 answers
21 views

I want to send a POST request with really long headers. Is there an easy way to do this for a string until the entire sting has been sent in python?

params = { "description": description } The description variable above is about 9000 characters, which is too long for the API I want to POST to. I think sending multiple requests in small ...
user20706867's user avatar
1 vote
0 answers
159 views

SqlAlchemy select hangs in FastAPI test setup

I developed a FastAPI app with WebSockets and I'm trying to test it. I need async tests in order to check data in the database during tests. My setup looks like this: I have a fixture that generates a ...
Artem Shalagin's user avatar
0 votes
1 answer
253 views

Handle Request payload using HTTPX for post data and capcha key to google recapchaV2

I'm trying to send the key I got from anti captcha, an example of the key is like this { "errorId":0, "status":"ready", "solution": { &...
perymerdeka's user avatar
0 votes
0 answers
185 views

Python-Authlib: setting an Oauth2 Async client with (service principal) and (user, password)

I come here for help after being stuck a few hours creating an OAuth2 Async httpx client. It's one of those situations that feels so close, and still doesn't come through in the end. I'm using this ...
Diego-MX's user avatar
  • 2,280
0 votes
0 answers
99 views

How to write unit test case for fastapi for mysql

app.py def get_db(request: Request): return request.state.db @router.get("/data") async def endpoint( db:Session = Depends(get_db), ): try: data = await crud....
putta's user avatar
  • 45
0 votes
1 answer
78 views

Scrape website using httpx and requests returns a timeout

I am trying to scrape this website https://www.blibli.com/p/facial-tissue-tisu-wajah-250-s-paseo/is--LO1-70001-00049-00003?seller_id=LO1-70001&sku_id=LO1-70001-00049-00001&sclid=...
Hal's user avatar
  • 193
3 votes
1 answer
1k views

Fastapi Testclient not able to send POST request using form-data

Currently I am doing Unit Testing in Fastapi using from fastapi.testclient import TestClient def test_login_api_returns_token(session,client): form_data = { "username": "...
code_10's user avatar
  • 165
1 vote
1 answer
583 views

Deprecation error in venv/lib/python3.8/site-packages/httpx/_content.py:201:DeprecationWarning: Use 'content=<...>' to upload raw bytes/text content

I installed a project that has httpx in the code and it seems that there is a method deprecated in its code. I looked for a solution but I don't find any solution for it. ( Also, I have run it before ...
Pain's user avatar
  • 113
0 votes
0 answers
32 views

Why is the response sometimes passing through and sometimes not when using requests httpx in python?

I am trying to scrape this website items, however when I used httpx or even requests sometimes it passes and gets the response sometimes it doesn't. It seems random, that's why I tried doing a rerun ...
Hal's user avatar
  • 193
2 votes
1 answer
587 views

Monkeypatching/mocking the HTTPX external requests

I'm trying to monkeypatch the external request. Here is the code of a web endpoint: import httpx, json ... @app.get('/test') async def view_test(request): async with httpx.AsyncClient() as client: ...
srbssv's user avatar
  • 23