12

I'm having an issue with a large curl call.

I get

  • nread <= 0, server closed connection, bailing
  • transfer closed with outstanding read data remaining

and the content is partial delivered

    GET /stats/?stats_breakdown=track__track&campaign=&search_criteria=2&period=0&date_month=11&date_day=03&date_year=2015&start_date_month=11&start_date_day=03&start_date_year=2015&end_date_month=12&end_date_day=31&end_date_year=2014 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Host: domain.com
Accept: */*
Cookie: sessionid=xxg4gglsm7o3b224wihqz8od19wl31h1; csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C
Cache-Control: no-cache
Connection: Keep-Alive
Keep-Alive: 600
Accept-Language: en-us
X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR
Content-Type: application/x-www-form-urlencoded
Origin: https://domain.com
Referer: https://domain.com

* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Server: nginx/1.8.0
< Date: Wed, 04 Nov 2015 12:54:05 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Vary: Cookie, Accept-Language
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< Content-Language: en
* Replaced cookie csrftoken="JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C" for domain domain.com, path /, expire 1478091245
< Set-Cookie: csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C; expires=Wed, 02-Nov-2016 12:54:05 GMT; Max-Age=31449600; Path=/; secure
<
* nread <= 0, server closed connection, bailing
* transfer closed with outstanding read data remaining
* Closing connection #0

This is the php configuration I use

function getHeaders()
{
    $headers = array();
    $headers[] = 'Cache-Control: no-cache';
    $headers[] = 'Connection: Keep-Alive';
    $headers[] = 'Keep-Alive: 600';
    $headers[] = 'Accept-Language: en-us';
    $headers[] = 'X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR';

    $headers[] = 'Content-Type: application/x-www-form-urlencoded';
    $headers[] = 'Origin: https://domain.com';
    $headers[] = 'Referer: https://domain.com';

    return $headers;

}

curl_setopt($connection, CURLOPT_URL, $url);
    curl_setopt($connection, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($connection, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($connection, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
    curl_setopt($connection, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($connection, CURLOPT_SSL_VERIFYHOST, false);
    curl_setopt($connection, CURLOPT_COOKIEFILE, 'cookie.txt');
    curl_setopt($connection, CURLOPT_COOKIEJAR, 'cookie.txt');
    curl_setopt($connection, CURLOPT_CONNECTTIMEOUT ,550000000);
    curl_setopt($connection, CURLOPT_TIMEOUT, 5500000000); //timeout in seconds
    curl_setopt($connection, CURLOPT_HTTPHEADER, getHeaders());
    curl_setopt($connection, CURLOPT_VERBOSE, 1);
2
  • Are you hitting a server behind a load balancer or some DDOS protection like miracloud? they might close long connections for security reasons, even if you have data streaming
    – Bolovsky
    Nov 4, 2015 at 13:07
  • I have no control over the remote server. But it loads perfectly in chrome..
    – user555854
    Nov 4, 2015 at 14:00

3 Answers 3

14

Encountered similar issue, my server is behind nginx. Curl is able to receive a response if it connects to the server directly but if curl connects to the server via nginx, curl is throwing below error

Session
* transfer closed with outstanding read data remaining
* Closing connection 0 curl: (18) transfer closed with outstanding read data remaining

When I connect to same nginx URL using a browser, Response is showing up fine. It is very weird, When I tried to connect to the same nginx URL using curl, it was throwing above error.

After comparing headers sent by the browser and curl. I found that the browser is able to receive the response because of the below header which curl is not sending:

'Accept-Encoding: gzip'

Sending above header using curl is working fine. So, what the above header is doing is compressing response to gzip thus reducing the response size.

After some more digging, found that nginx is not able to send any payload > 80kb. After wasting a lot of time, found that issue is with nginx buffering and nginx worked like a charm after adding below proxy_buffering property in nginx.conf:

 location / {
 proxy_buffering off;
 }

The accepted answer didn't resolve my issue. Writing this answer, so that anyone need not waste their time if they are facing the same issue as I did.

1
  • Had a similar issue. This fixed it, although not sure if there are other setting to use: e.g. ngx_http_proxy_module
    – SScotti
    Sep 3, 2020 at 0:44
12

Okay, after some searching and IRC chat's I found solution, but not 100% sure what the cause is. Looks like the keep-alives weren't send enough to keep the connection going on. Will post the solution here, hopefully I can help someone out.

What helped for me is adding

--keepalive-time 2

An explanation of the curl option

--keepalive-time <seconds>

This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (meaning Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used. (Added in 7.18.0)

If this option is used several times, the last one will be used. If unspecified, the option defaults to 60 seconds.

Looks like the default was too high to keep my connection open.

Here is the full command I used for my call

curl URL -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,et;q=0.6,nl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' --compressed -v --keepalive-time 2

And I'm running this version of curl on osx

curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

If someone would like to have this option used in PHP curl, the --keepalive-time option is available since PHP 5.5. You can use it as following:

curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1);
curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2);

Hope this helps someone struggling with the same issue!

2
  • How to add CURLOPT_TCP_KEEPALIVE & CURLOPT_TCP_KEEPIDLE in php Guzzle/HttpClient ?
    – user6067756
    Apr 29, 2016 at 18:15
  • @user6067756 You just have to pass them as curl-settings: justrocketscience.com/post/…
    – NaN
    Sep 2, 2017 at 11:42
0

libcurl only tells you that the connection was cut by the server in an unclean way where it didn't deliver data it had promised to do. It looks like the chunked-encoding didn't signal the end of the transfer.

Browsers are notorious for being extremely liberal in what they receive so they overlook and are fine with all sorts of protocol violations to a much larger extent than libcurl is.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy