I use WWW::Curl to upload files:

use WWW::Curl::Easy 4.14;
use WWW::Curl::Form;

my $url = 'http://example.com/backups/?sid=12313qwed323';
my $params = {
    name => 'upload',
    action => 'keep',
    backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
};

my $form = WWW::Curl::Form->new();
foreach my $k (keys %{$params}) {
    if (ref $params->{$k}) {
        $form->formaddfile(@{$params->{$k}}[0], $k, 'multipart/form-data');
    } else {
        $form->formadd($k, $params->{$k});
    }
}

my $curl = WWW::Curl::Easy->new() or die $!; 
$curl->setopt(CURLOPT_HTTPPOST, $form);
$curl->setopt(CURLOPT_URL, $url);

my $body;   
$curl->setopt(CURLOPT_WRITEDATA, \$body);
my $retcode = $curl->perform();
my $response_code = $curl->getinfo(CURLINFO_HTTP_CODE); 

nothing special here and this code works well.

I want to upload large files and I don't want to preload everything in the memory. At least that is what I heard that libcurl is doing.

CURLOPT_READFUNCTION accepts callbacks which returns parts of the content. That means that I cannot use WWW::Curl::Form to set POST parameters but that I have to return the whole content through this callback. Is that right?

I think that the code could look like this:

use WWW::Curl::Easy 4.14;

my $url = 'http://example.com/backups/?sid=12313qwed323'
my $params = {
    name => 'upload',
    action => 'keep',
    backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
};

my $fields;
foreach my $k (keys %{$params}) {
    $fields .= "$k=".(ref $params->{$k} ? '@'.@{$params->{$k}}[0] : uri_escape_utf8($params->{$k}))."&";
}
chop($fields);

my $curl = WWW::Curl::Easy->new() or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_POSTFIELDS, $fields); # is it needed with READFUNCTION??
$curl->setopt(CURLOPT_URL, $url);

my @header = ('Content-type: multipart/form-data', 'Transfer-Encoding: chunked');
$curl->setopt(CURLOPT_HTTPHEADER, \@header);

#$curl->setopt(CURLOPT_INFILESIZE, $size);
$curl->setopt(CURLOPT_READFUNCTION, sub {

    # which data to return here?
    # $params (without file) + file content?

    return 0;
});

Which data does CURLOPT_READFUNCTION callback have to return? $params + File(s) content? In which format?

Do I really have to create the data (returned by CURLOPT_READFUNCTION) by myself or is there a simple way to create it in the right format?

Thanks

share|improve this question
    
Are you committed to using WWW::Curl? I think this would be easier with LWP, if you can switch. – wes Feb 28 '12 at 16:25
    
LWP or better use WWW::Mechanize – Gilles Quenot Feb 28 '12 at 16:56
    
I know this answer is not directly related to your code, but I spent a significant amount of time troubleshooting a similar issue using WWW::Mechanize only to discover that the MaxPostSize on the web server had been set by our admin to some arbitrary limit. – AWT Feb 28 '12 at 19:58
    
I have already done it with LWP and it is much slower than libcurl. I will check WWW::Mechanize. thanks – toktok Feb 29 '12 at 10:22
up vote 4 down vote accepted

Test 16formpost.t is relevant. As you can see, it's completely disabled. This fact and my fruitless experiments with various return values for the callback function lets me believe the CURLOPT_READFUNCTION feature is known broken in the Perl binding.

I have to return the whole content through this callback. Is that right?

No, you can feed it the request body piecewise, suitable for chunked encoding. The callback will be necessarily called several times, according to the limit set in CURLOPT_INFILESIZE.

Which data does CURLOPT_READFUNCTION callback have to return?

A HTTP request body. Since you do a file upload, this means Content-Type multipart/form-data. Following is an example using HTTP::Message. CURLOPT_HTTPPOST is another way to construct this format.

use HTTP::Request::Common qw(POST);
use WWW::Curl::Easy 4.14;

my $curl = WWW::Curl::Easy->new or die $!;
$curl->setopt(CURLOPT_POST, 1);
$curl->setopt(CURLOPT_URL, 'http://localhost:5000');
$curl->setopt(CURLOPT_HTTPHEADER, [
    'Content-type: multipart/form-data', 'Transfer-Encoding: chunked'
]);
$curl->setopt(CURLOPT_READFUNCTION, sub {
    return POST(undef, Content_Type => 'multipart/form-data', Content => [
        name    => 'upload',
        action  => 'keep',
        backup1 => [ '/tmp/backup1.zip' ],   # 1st file for upload
    ])->content;
});
my $r = $curl->perform;
share|improve this answer
    
thanks. very useful info. Problem with HTTP::Message is that it loads whole content in the memory. I have limited memory (64MB) and I that would mean that request has to be much smaller. I will try today to create a callback function which returns the content in chunks without buffering whole content. – toktok Feb 29 '12 at 10:26
    
just received word from the WWW::Curl maintainer about the CURLOPT_READFUNCTION feature: "yeah, looks pretty broken. I'm planning to do a WWW::Curl overhaul in the coming weeks, might fix this too." – daxim Mar 1 '12 at 14:58
    
Why do you think it is broken? I have no problems with READFUNCTION, it behaves exactly as explained on curl homepage. "it works for me" ;-) – toktok Mar 8 '12 at 11:30
    
Confirm: when you run it, the program in my answer transmits a non-empty request body. – daxim Mar 8 '12 at 13:47
    
READFUNCTION has to return 0 if there are no more bytes to return. I had/have a problem with "Transfer-Encoding: chunked" because a server responded with "Content-Length missing" although it was chunked transfer. I removed that header line and it works. – toktok Mar 12 '12 at 13:06

The CURLOPT_READFUNCTION callback is only used for chunked tranfer encoding. It may work, but I haven't been able to get it to and found that doing so wasn't necessary anyway.

My use case was for upload of data to AWS, where it's not ok to upload the data as multi-part form data. Instead, it's a straight POST of the data. It does require that you know how much data you're sending the server, though. This seems to work for me:

my $infile = 'file-to-upload.json';
my $size = -s $infile;
open( IN, $infile ) or die("Cannot open file - $infile. $! \n");

my $curl = WWW::Curl::Easy->new;
$curl->setopt(CURLOPT_HEADER,       1);
$curl->setopt(CURLOPT_NOPROGRESS,   1);
$curl->setopt(CURLOPT_POST,         1);
$curl->setopt(CURLOPT_URL,          $myPostUrl);
$curl->setopt(CURLOPT_HTTPHEADER,   
    ['Content-Type: application/json']); #For my use case
$curl->setopt(CURLOPT_POSTFIELDSIZE_LARGE, $size);
$curl->setopt(CURLOPT_READDATA, \*IN);

my $retcode = $curl->perform;

if ($retcode == 0) {
    print("File upload success\n");
} 
else {
    print("An error happened: $retcode ".$curl->strerror($retcode)."\n");
}

The key is providing an open filehandle reference to CURLOPT_READDATA. After that, the core curl library handles the reads from it without any need for callbacks.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.