I second the suggestion above to use HTTPX, but I often use it in a different way so am adding my answer.
I personally use asyncio.run
(introduced in Python 3.7) rather than asyncio.gather
and also prefer the aiostream
approach, which can be used in combination with asyncio and httpx.
As in this example I just posted, this style is helpful for processing a set of URLs asynchronously even despite the (common) occurrence of errors. I particularly like how that style clarifies where the response processing occurs and for ease of error handling (which I find async calls tend to give more of).
It's easier to post a simple example of just firing off a bunch of requests asynchronously, but often you also want to handle the response content (compute something with it, perhaps with reference to the original object that the URL you requested was to do with).
The core of that approach looks like:
async with httpx.AsyncClient(timeout=timeout) as session:
ws = stream.repeat(session)
xs = stream.zip(ws, stream.iterate(urls))
ys = stream.starmap(xs, fetch, ordered=False, task_limit=20)
process = partial(process_thing, things=things, pbar=pbar, verbose=verbose)
zs = stream.map(ys, process)
return await zs
where:
process_thing
is an async response content handling function
things
is the input list (which the urls
generator of URL strings came from), e.g. a list of objects/dictionaries
pbar
is a progress bar (e.g. tqdm.tqdm
) [optional but useful]
All of that goes in an async function async_fetch_urlset
which is then run by calling a synchronous 'top-level' function named e.g. fetch_things
which runs the coroutine [this is what's returned by an async function] and manages the event loop:
def fetch_things(urls, things, pbar=None, verbose=False):
return asyncio.run(async_fetch_urlset(urls, things, pbar, verbose))
Since a list passed as input (here it's things
) can be modified in-place, you can effectively get output back (as we're used to from synchronous function calls)
requests-async
provides async/await functionality forrequests
- github.com/encode/requests-async