1681

In HTTP there are two ways to POST data: application/x-www-form-urlencoded and multipart/form-data. I understand that most browsers are only able to upload files if multipart/form-data is used. Is there any additional guidance when to use one of the encoding types in an API context (no browser involved)? This might e.g. be based on:

  • data size
  • existence of non-ASCII characters
  • existence on (unencoded) binary data
  • the need to transfer additional data (like filename)

I basically found no formal guidance on the web regarding the use of the different content-types so far.

2
  • 119
    It should be mentioned that these are the two MIME types that HTML forms use. HTTP itself has no such limitation... one can use whatever MIME type he wants via HTTP.
    – tybro0103
    Commented Mar 21, 2014 at 15:18
  • FYI for anyone using the C curl library, libcurl, here is the option to use: curl_easy_setopt(myCurlEasyhandle, CURLOPT_MIMEPOST, mimeHandle);: curl.se/libcurl/c/CURLOPT_MIMEPOST.html: "This option is the preferred way of posting an HTTP form, replacing and extending the deprecated CURLOPT_HTTPPOST option." Full example: curl.se/libcurl/c/smtp-mime.html Commented May 13, 2022 at 22:57

7 Answers 7

2472
+50

TL;DR

Summary; if you have binary (non-alphanumeric) data (or a significantly sized payload) to transmit, use multipart/form-data. Otherwise, use application/x-www-form-urlencoded.


The MIME types you mention are the two Content-Type headers for HTTP POST requests that user-agents (browsers) must support. The purpose of both of those types of requests is to send a list of name/value pairs to the server. Depending on the type and amount of data being transmitted, one of the methods will be more efficient than the other. To understand why, you have to look at what each is doing under the covers.

For application/x-www-form-urlencoded, the body of the HTTP message sent to the server is essentially one giant query string -- name/value pairs are separated by the ampersand (&), and names are separated from values by the equals symbol (=). An example of this would be: 

MyVariableOne=ValueOne&MyVariableTwo=ValueTwo

According to the specification:

[Reserved and] non-alphanumeric characters are replaced by `%HH', a percent sign and two hexadecimal digits representing the ASCII code of the character

That means that for each non-alphanumeric byte that exists in one of our values, it's going to take three bytes to represent it. For large binary files, tripling the payload is going to be highly inefficient.

That's where multipart/form-data comes in. With this method of transmitting name/value pairs, each pair is represented as a "part" in a MIME message (as described by other answers). Parts are separated by a particular string boundary (chosen specifically so that this boundary string does not occur in any of the "value" payloads). Each part has its own set of MIME headers like Content-Type, and particularly Content-Disposition, which can give each part its "name." The value piece of each name/value pair is the payload of each part of the MIME message. The MIME spec gives us more options when representing the value payload -- we can choose a more efficient encoding of binary data to save bandwidth (e.g. base 64 or even raw binary).

Why not use multipart/form-data all the time? For short alphanumeric values (like most web forms), the overhead of adding all of the MIME headers is going to significantly outweigh any savings from more efficient binary encoding.

16
  • 115
    Does x-www-form-urlencoded have a length limit, or is it unlimited?
    – Pacerier
    Commented Mar 9, 2013 at 16:56
  • 54
    @Pacerier The limit is enforced by the server receiving the POST request. See this thread for more discussion: stackoverflow.com/questions/2364840/… Commented May 28, 2013 at 13:23
  • 7
    @ZiggyTheHamster JSON and BSON are each more efficient for different types of data. Base64 is inferior to gzip, for both serialization methods. Base64 does not bring any advantages at all, HTTP supports binary pyloads.
    – oxygen
    Commented Jun 4, 2013 at 17:28
  • 31
    Also note that if a form contains a named file upload, your only choice is form-data, because urlencoded doesn't have a way to place the filename (in form-data it's the name parameter to content-disposition). Commented Mar 6, 2014 at 18:00
  • 4
    @EML see my parenthetical "(chosen specifically so that this boundary string does not occur in any of the "value" payloads)" Commented Apr 18, 2014 at 12:23
205

READ AT LEAST THE FIRST PARA HERE!

I know this is 3 years too late, but Matt's (accepted) answer is incomplete and will eventually get you into trouble. The key here is that, if you choose to use multipart/form-data, the boundary must not appear in the file data that the server eventually receives.

This is not a problem for application/x-www-form-urlencoded, because there is no boundary. x-www-form-urlencoded can also always handle binary data, by the simple expedient of turning one arbitrary byte into three 7BIT bytes. Inefficient, but it works (and note that the comment about not being able to send filenames as well as binary data is incorrect; you just send it as another key/value pair).

The problem with multipart/form-data is that the boundary separator must not be present in the file data (see RFC 2388; section 5.2 also includes a rather lame excuse for not having a proper aggregate MIME type that avoids this problem).

So, at first sight, multipart/form-data is of no value whatsoever in any file upload, binary or otherwise. If you don't choose your boundary correctly, then you will eventually have a problem, whether you're sending plain text or raw binary - the server will find a boundary in the wrong place, and your file will be truncated, or the POST will fail.

The key is to choose an encoding and a boundary such that your selected boundary characters cannot appear in the encoded output. One simple solution is to use base64 (do not use raw binary). In base64 3 arbitrary bytes are encoded into four 7-bit characters, where the output character set is [A-Za-z0-9+/=] (i.e. alphanumerics, '+', '/' or '='). = is a special case, and may only appear at the end of the encoded output, as a single = or a double ==. Now, choose your boundary as a 7-bit ASCII string which cannot appear in base64 output. Many choices you see on the net fail this test - the MDN forms docs, for example, use "blob" as a boundary when sending binary data - not good. However, something like "!blob!" will never appear in base64 output.

16
  • 70
    While a consideration of multipart/form-data is the ensure the boundary does not appear in the data this is fairly simple to accomplish by choosing a boundary which is sufficiently long. Please do not us base64 encoding to accomplish this. A boundary which is randomly generated and the same length as a UUID should be sufficient: stackoverflow.com/questions/1705008/….
    – Joshcodes
    Commented Apr 30, 2014 at 22:01
  • 29
    @EML, This doesn't make sense at all. Obviously the boundary is chosen automatically by the http client (browser) and the client will be smart enough not to use a boundary that clashes with the contents of your uploaded files. It's as simple a a substring match index === -1.
    – Pacerier
    Commented Dec 11, 2014 at 8:01
  • 16
    @Pacerier: (A) read the question: "no browser involved, API context". (B) browsers don't construct requests for you anyway. You do it yourself, manually. There's no magic in browsers.
    – EML
    Commented Dec 11, 2014 at 9:40
  • 14
    @BeniBela, He's probably going to suggest to use '()+-./:= then. Yet random generation with substring check is still the way to go and it can be done with one line: while(true){r = rand(); if(data.indexOf(r) === -1){doStuff();break;}}. EML's suggestion (convert to base64 just to avoid matching substrings) is just plain odd, not to mention it comes with unneeded performance degradation. And all the trouble for nothing since the one line algorithm is equally straightforward and simple. Base64 is not meant to be (ab)used this way, as HTTP body accept all 8-bit octets.
    – Pacerier
    Commented Jul 27, 2015 at 10:51
  • 53
    This answer not only adds nothing to the discussion, but also gives wrong advice. Firstly, whenever transmitting random data in separated parts, it is always possible that the chosen boundary will be present in the payload. The ONLY way to make sure this doesn't happen is to examine the entire payload for each boundary we come up with. Completely impractical. We just accept the infinitesimal probability of a collision and come up with a reasonable boundary, like "---boundary-<UUID here>-boundary---". Secondly, always using Base64 will waste bandwidth and fill up buffers for no reason at all.
    – vagelis
    Commented May 5, 2016 at 12:22
114

I don't think HTTP is limited to POST in multipart or x-www-form-urlencoded. The Content-Type Header is orthogonal to the HTTP POST method (you can fill MIME type which suits you). This is also the case for typical HTML representation based webapps (e.g. json payload became very popular for transmitting payload for ajax requests).

Regarding Restful API over HTTP the most popular content-types I came in touch with are application/xml and application/json.

application/xml:

  • data-size: XML very verbose, but usually not an issue when using compression and thinking that the write access case (e.g. through POST or PUT) is much more rare as read-access (in many cases it is <3% of all traffic). Rarely there where cases where I had to optimize the write performance
  • existence of non-ascii chars: you can use utf-8 as encoding in XML
  • existence of binary data: would need to use base64 encoding
  • filename data: you can encapsulate this inside field in XML

application/json

  • data-size: more compact less that XML, still text, but you can compress
  • non-ascii chars: json is utf-8
  • binary data: base64 (also see json-binary-question)
  • filename data: encapsulate as own field-section inside json

binary data as own resource

I would try to represent binary data as own asset/resource. It adds another call but decouples stuff better. Example images:

POST /images
Content-type: multipart/mixed; boundary="xxxx" 
... multipart data

201 Created
Location: http://imageserver.org/../foo.jpg  

In later resources you could simply inline the binary resource as link:

<main-resource>
 ...
 <link href="http://imageserver.org/../foo.jpg"/>
</main-resource>
4
  • 1
    Interesting. But when to use application/x-www-form-urlencoded and when multipart/form-data?
    – max
    Commented Oct 25, 2010 at 6:02
  • 6
    application/x-www-form-urlencoded is the default mime-type of your request (see also w3.org/TR/html401/interact/forms.html#h-17.13.4). I use it for "normal" webforms. For API I use application/xml|json. multipart/form-data is a bell in thinking of attachements (inside response body several data-sections are concattenated with a defined boundary string). Commented Oct 25, 2010 at 19:38
  • 9
    I think the OP was probably just asking about the two types that HTML forms use, but I'm glad this was pointed out.
    – tybro0103
    Commented Mar 21, 2014 at 15:08
  • Did you ever try if browsers can submit form-fields e.g. with Json-Mime-type ?
    – Radon8472
    Commented Oct 25, 2020 at 9:17
29

I agree with much that Manuel has said. In fact, his comments refer to this url...

http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

... which states:

The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

However, for me it would come down to tool/framework support.

  • What tools and frameworks do you expect your API users to be building their apps with?
  • Do they have frameworks or components they can use that favour one method over the other?

If you get a clear idea of your users, and how they'll make use of your API, then that will help you decide. If you make the upload of files hard for your API users then they'll move away, of you'll spend a lot of time on supporting them.

Secondary to this would be the tool support YOU have for writing your API and how easy it is for your to accommodate one upload mechanism over the other.

2
  • 1
    Hi, does it mean that every time we post somethings to web server, we have to mention what is the Content-type in order to let web server know should it decode the data? Even we craft the http request ourself, we MUST mention the Content-type right?
    – Sam YC
    Commented Jul 17, 2013 at 6:21
  • 2
    @GMsoF, It's optional. See stackoverflow.com/a/16693884/632951 . You may want to avoid using content-type when crafting a specific request for a specific server to avoid generic overheads.
    – Pacerier
    Commented Dec 11, 2014 at 8:10
3

Just a little hint from my side for uploading HTML5 canvas image data:

I am working on a project for a print-shop and had some problems due to uploading images to the server that came from an HTML5 canvas element. I was struggling for at least an hour and I did not get it to save the image correctly on my server.

Once I set the contentType option of my jQuery ajax call to application/x-www-form-urlencoded everything went the right way and the base64-encoded data was interpreted correctly and successfully saved as an image.


Maybe that helps someone!

1
  • 6
    What content type was it sending it before you changed it? This problem could have been due to the server not supporting the content type you were sending it as.
    – catorda
    Commented Jan 13, 2016 at 17:24
2

If you need to use Content-Type=x-www-urlencoded-form then DO NOT use FormDataCollection as parameter: In asp.net Core 2+ FormDataCollection has no default constructors which is required by Formatters. Use IFormCollection instead:

 public IActionResult Search([FromForm]IFormCollection type)
    {
        return Ok();
    }
0

In my case the issue was that the response contentType was application/x-www-form-urlencoded but actually it contained a JSON as the body of the request. Django when we access request.data in Django it cannot properly converted it so access request.body.

Refer this answer for better understanding: Exception: You cannot access body after reading from request's data stream

Not the answer you're looking for? Browse other questions tagged or ask your own question.