30

I can perform single requests using Guzzle and I'm very pleased with Guzzle's performance so far however, I read in the Guzzle API something about MultiCurl and Batching.

Could someone explain to me how to make multiple requests at the same time? Async if possible. I don't know if that is what they mean with MultiCurl. Sync would also be not a problem. I just want to do multiple requests at the same time or very close (short space of time).

1
  • There's a demo of this in the docs. This is still a synchronous call from your perspective, but will be parallel internally - thus the total time for the call will just be the time for the single longest fetch.
    – halfer
    Oct 23, 2013 at 3:34

3 Answers 3

29

From the docs: http://guzzle3.readthedocs.org/http-client/client.html#sending-requests-in-parallel

For an easy to use solution that returns a hash of request objects mapping to a response or error, see http://guzzle3.readthedocs.org/batching/batching.html#batching

Short example:

<?php

$client->send(array(
    $client->get('http://www.example.com/foo'),
    $client->get('http://www.example.com/baz'),
    $client->get('http://www.example.com/bar')
));
1
  • can we send different data to same url in guzzel ? Jan 14, 2019 at 7:15
29

An update related to the new GuzzleHttp guzzlehttp/guzzle

Concurrent/parallel calls are now run through a few different methods including Promises.. Concurrent Requests

The old way of passing a array of RequestInterfaces will not work anymore.

See example here

    $newClient = new  \GuzzleHttp\Client(['base_uri' => $base]);
    foreach($documents->documents as $doc){

        $params = [
            'language' =>'eng',
            'text' => $doc->summary,
            'apikey' => $key
        ];

        $requestArr[$doc->reference] = $newClient->getAsync( '/1/api/sync/analyze/v1?' . http_build_query( $params) );
    }

    $time_start = microtime(true);
    $responses = \GuzzleHttp\Promise\unwrap($requestArr); //$newClient->send( $requestArr );
    $time_end = microtime(true);
    $this->get('logger')->error(' NewsPerf Dev: took ' . ($time_end - $time_start) );

Update: As suggested in comments and asked by @sankalp-tambe, you can also use a different approach to avoid that a set of concurrent request with a failure will not return all the responses.

While the options suggested with Pool is feasible i still prefer promises.

An example with promises is to use settle and and wait methods instead of unwrap.

The difference from the example above would be

$responses = \GuzzleHttp\Promise\settle($requestArr)->wait(); 

I have created a full example below for reference on how to handle the $responses too.

require __DIR__ . '/vendor/autoload.php';
use GuzzleHttp\Client as GuzzleClient;
use GuzzleHttp\Promise as GuzzlePromise;

$client = new GuzzleClient(['timeout' => 12.0]); // see how i set a timeout
$requestPromises = [];
$sitesArray = SiteEntity->getAll(); // returns an array with objects that contain a domain

foreach ($sitesArray as $site) {
    $requestPromises[$site->getDomain()] = $client->getAsync('http://' . $site->getDomain());
}

$results = GuzzlePromise\settle($requestPromises)->wait();

foreach ($results as $domain => $result) {
    $site = $sitesArray[$domain];
    $this->logger->info('Crawler FetchHomePages: domain check ' . $domain);

    if ($result['state'] === 'fulfilled') {
        $response = $result['value'];
        if ($response->getStatusCode() == 200) {
            $site->setHtml($response->getBody());
        } else {
            $site->setHtml($response->getStatusCode());
        }
    } else if ($result['state'] === 'rejected') { 
        // notice that if call fails guzzle returns is as state rejected with a reason.

        $site->setHtml('ERR: ' . $result['reason']);
    } else {
        $site->setHtml('ERR: unknown exception ');
        $this->logger->err('Crawler FetchHomePages: unknown fetch fail domain: ' . $domain);
    }

    $this->entityManager->persist($site); // this is a call to Doctrines entity manager
}

This example code was originally posted here.

5
  • While this works great for my use case of loading multiple image URLs' data at once, how can one handle a situation where one of the URLs being loaded throws a 404 error? When this happens, Guzzle freaks out and throws a guzzle exception. I cannot guarantee the availability of the URLs so I was hoping to just load multiple requests and use the ones that actually go through.
    – georaldc
    Jan 18, 2016 at 17:23
  • 1
    Nevermind, I just re-did my code to use GuzzleHttp\Pool instead. Seems to work well too and gives me a bit more control.
    – georaldc
    Jan 18, 2016 at 18:29
  • 3
    can you share your code for multiple urls with Pool. Jan 25, 2016 at 7:17
  • @georaldc i didnt like the Pool approach that much. Sankalp you can see a full example alternative that will return all responses instead of an exception in my answer update.
    – Bizmate
    Jun 5, 2016 at 9:31
  • @Bizmate thanks for this it worked for me. I am curious on the level of concurrency and how much more performant it's than the old asynchronous way?
    – awm
    Nov 15, 2017 at 16:28
11

Guzzle 6.0 has made sending multiple async requests very easy.

There are multiple ways to do it.

You can create the async requests and add the resultant promises to a single array, and get the result using the settle() method like this:

$promise1 = $client->getAsync('http://www.example.com/foo1');
$promise2 = $client->getAsync('http://www.example.com/foo2');
$promises = [$promise1, $promise2];

$results = GuzzleHttp\Promise\settle($promises)->wait();

You can now loop through these results and fetch the response using GuzzleHttpPromiseall or GuzzleHttpPromiseeach. Refer to this article for further details.

In case if you have an indeterminate number of requests to be sent(say 5 here), you can use GuzzleHttp/Pool::batch(). Here is an example:

$client = new Client();

// Create the requests
$requests = function ($total) use($client) {
    for ($i = 1; $i <= $total; $i++) {
        yield new Request('GET', 'http://www.example.com/foo' . $i);
    }
};

// Use the Pool::batch()
$pool_batch = Pool::batch($client, $requests(5));
foreach ($pool_batch as $pool => $res) {

    if ($res instanceof RequestException) {
        // Do sth
        continue;
    }

    // Do sth
}
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.