2

Good Day!

cURL is acting very slow when requesting a page. I know its not the page requested, because the page returns in the browser instantly.

2 things I've noticed

  • The starttransfer_time is regularly almost 20
  • The local_port seems to change every time. Is that normal?
  • Occasionally, cURL will respond instantly

I have the following code:

$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url ); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, false);
$output = curl_exec($ch); 
curl_close($ch); 

Echoing curl_getinfo() gives me the following

[url] => http://127.0.0.1:80/wpengine/?json=t
[content_type] => text/html; charset=iso-8859-1
[http_code] => 302
[header_size] => 215
[request_size] => 64
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 17.238
[namelookup_time] => 0
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 221
[speed_download] => 12
[speed_upload] => 0
[download_content_length] => 221
[upload_content_length] => 0
[starttransfer_time] => 17.238
[redirect_time] => 0
[certinfo] => Array
    (
    )

[primary_ip] => 127.0.0.1
[primary_port] => 80
[local_ip] => 127.0.0.1
[local_port] => 51875
[redirect_url] => 

Can anyone give me some pointers on how to figure out what is going on?

Here are a few lines from the Apache access log

127.0.0.1 - - [06/Dec/2013:12:01:22 -0500] "GET /wpengine/?json=t HTTP/1.1" 302 221
127.0.0.1 - - [06/Dec/2013:12:01:12 -0500] "GET /community HTTP/1.1" 200 6266
127.0.0.1 - - [06/Dec/2013:12:01:22 -0500] "GET /public/js/jquery.js?b=10 HTTP/1.1" 304 -
7
  • 1
    If you're running this locally, what server do you have set up? Can you check the server logs? Dec 6, 2013 at 16:19
  • @r3mus I am running Wamp with Apache. I edited my post to include a few lines from the access log
    – Ben
    Dec 6, 2013 at 17:03
  • 2
    Curious if maybe CURL is trying IPv6 first, and failing to resolve? Try adding curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); Dec 6, 2013 at 17:27
  • Hmm, that did not make a difference. I find this very puzzling. Could it have something to do with sessions as it is on the same domain? Just a shot in the dark...
    – Ben
    Dec 6, 2013 at 18:55
  • 1
    Not really sure, but looking at your logs, once CURL actually hits the server, it's super fast. Any way you can try this on a public server somewhere to eliminate the localhost variable? Dec 6, 2013 at 19:06

4 Answers 4

5

This is mostly because of the Expect: 100-continue header CURL send to the server when it handles large POST data, and the server just happened not support this feature. You can confirm by add -vv for curl command line and see the outputs.

To solve this, you can either

  1. fix your server to support Expect: 100-continue header, or
  2. force curl not to send it by add -H "Expect:" option (in php, you can refer to this answer: How can I stop cURL from using 100 Continue?).

Links about Expect: 100-continue:

https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3 https://support.urbanairship.com/entries/59909909--Expect-100-Continue-Issues-and-Risks

2

I've found a similar issue when hitting Google Analytics' MP API ... the starttransfer time is consistently 1.0 seconds plus a tiny amount, from which I infer there is a 1 second delay somewhere, I am guessing in cURL. It's not using CPU at my end, and I somehow doubt Google is putting in the delay... otuput from curl_getinfo() ...

Array
(
    [url] => http://www.google-analytics.com/collect
    [content_type] => image/gif
    [http_code] => 200
    [header_size] => 388
    [request_size] => 201
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 1.021528
    [namelookup_time] => 8.7E-5
    [connect_time] => 8.8E-5
    [pretransfer_time] => 0.000324
    [size_upload] => 1736
    [size_download] => 35
    [speed_download] => 34
    [speed_upload] => 1699
    [download_content_length] => 35
    [upload_content_length] => 1736
    [starttransfer_time] => 1.002496
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [redirect_url] => 
)
3
  • Yeah, a quick command line script using lwp-request (starting a new Perl interpreter for every POST) is miles faster than cURL looped in one PHP process. Jan 30, 2014 at 1:01
  • 5
    I found out a way to fix this, at least for my needs ... I was passing the above argument list to CURLOPT_POSTFIELDS expecting it to generate a urlencoded post body, but in fact it generates multipart. In order to get it to send a plain URLencoded POST you have to encode the body parameters yourself and pass a ready-to-use string. For some reason, the multipart process has a one second delay, the plain POST works at full speed. Jan 31, 2014 at 0:32
  • YES. Even though I don't understand why multipart takes so long to start, it worked, delay has gone! You should really update your answer with this info Sep 18, 2015 at 9:24
2

One of the comments by @brandonscript in this thread helped me solve the problem.

His answer:

Curious if maybe CURL is trying IPv6 first, and failing to resolve?

Try adding curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );

My problem was the opposite, I had to set the following option:

curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V6 );

Thank you, wonderful people.

1
  • Yes, this was the solution for me. The destination server only had IPv6. This immediately removed the delay on starttransfer_time Sep 8 at 8:55
0

I used Fiddler app from Telerik to debug connections, and did not find any problem, I think it may be related to web server child process workers. To debug using Fiddler paste the following line before curl_exec($ch):

curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:8888');

Thank you all for the pointers

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.