43

I have some web services that I am writing and I am trying to be as RESTful as possible. I am hosting these web services using a HTTPHandler running inside of IIS/ASP.NET/SharePoint.

Most of my services expect a HTTP GET. I have two of these that are simply returning some data (i.e., a query) and will be Idempotent, but the parameters may be somewhat complex. Both of them could include characters in the parameters of the service that are not allowed for at least the PATH portion of the URL.

Using IIS, ASP.NET, and SharePoint I have found that the following characters in the URL path don't even make it to my HttpHandler even if Url encoded (the request blows up and I don't have any easy control over this):

  • % (%25)
  • & (%26)
  • * (%2a, but didn't Url encode)
  • + (%2b)
  • : (%3a)
  • < (%3c)
  • (%3e)

The following characters made it to my HttpHandler, but the UriTemplate could not handle them properly even if Url encoded:

  • (%23)

  • . (%2e, but didn't Url encode; UriTemplate removed the "." if is is the last character before a /)
  • ? (%3f)
  • / (%2f - UriTemplate fails for obvious reasons even if UrlEncoded)
  • \ (%5c)

So, I've been somewhat thorough, but I need to test these url encoded characters in the query string. It appears that this will work for the most part there.

In one of my services, the special characters that are a parameter are semantically part of a query/filter (actually search terms for a search service), but in another they are not really part of a query/filter so ideally they are part of the path and not the query string.

My question is, what option is best? Here are some I know of:

  1. Use HTTP GET and query string. Anything that may use special characters should be on the query string and Url Encoded. This is where I am leaning, but I am concerned about extremely long query strings (IE has a 2083 limit)

  2. Use HTTP GET and base64 encoding within path. Use a Modified Base64 for URL for any parameters that might use special characters and keep them as part of the path if preferred. I have tried this and it works, but it is kind of ugly. Still a concern about extremely long query strings.

  3. Use HTTP POST and message body. Anything that may use special characters should be in the body of the request. Seems like a decent solution, but posts are understood to not be Idempotent and (I thought) are generally meant for changes (whereas no change is occurring here).

  4. Use HTTP GET and message body. Anything that may use special characters should be in the body of the request. This seems like a bad idea according to SO: HTTP GET with request body and Roy Fielding.

  5. Use a combination of #3 and either #1 or #2 above depending on how large the request can be.

  6. Other???

Note that in some cases I may be able to change things around to prevent special characters (and I may do that), but I won't be able to do this in all cases.


Regarding URI length, RFC2616 Sec3.2.1 says the following:

The HTTP protocol does not place any a priori limit on the length of a URI. Servers MUST be able to handle the URI of any resource they serve, and SHOULD be able to handle URIs of unbounded length if they provide GET-based forms that could generate such URIs. A server SHOULD return 414 (Request-URI Too Long) status if a URI is longer than the server can handle (see section 10.4.15).

  Note: Servers ought to be cautious about depending on URI lengths
  above 255 bytes, because some older client or proxy
  implementations might not properly support these lengths.

In addition the Maximum URL length is 2,083 characters in Internet Explorer.

Related: How to pass complex queries in REST?

8
  • 1
    Just so you know, this question has nothing to do with REST, just proper HTTP usage. So please don't refer to this as being "RESTful" thanks.
    – aehlke
    Aug 12, 2009 at 17:05
  • @Wahnfrieden, this wouldn't be the first time I misused terms, but I do want my services to be RESTful. I agree that this is basically about proper HTTP usage, but it seems to me that RESTful may come into play more when you decide between a POST and other options (or maybe not :-S). Aug 12, 2009 at 20:22
  • Nope, that's just HTTP. Many people mistake REST for HTTP for some reason, since it's become a trendy buzzword. Proper HTTP is necessary for REST's constraints, but not sufficient.
    – aehlke
    Aug 12, 2009 at 22:52
  • >>In addition the Maximum URL length is 2,083 characters in Internet Explorer. I'd better won't use that. According to HTTP spec, servers may discard GET requests which length is more than 256 symbols
    – glaz666
    Aug 13, 2009 at 7:51
  • 1
    @glaz666 The spec says "Note: Servers ought to be cautious about depending on URI lengths above 255 bytes, because some older client or proxy implementations might not properly support these lengths" This was written in 1999, I highly doubt that anyone is still running a proxy with this limitation anymore. Aug 13, 2009 at 12:31

11 Answers 11

37

There's no perfect way to do this.

The correct HTTP/REST way would be to use a GET and have all your parameters in the URL as query arguments. You've identified two practical problems with this approach

  1. Your server software is not correctly passing some characters to you, even if URL encoded. That surprises me, actually, and you should look more closely at what's going on that you can't even get a % through the URL. Does your framework give you raw access to PATH_INFO or otherwise unprocessed characters? That may give you a workaround.
  2. Your query strings may be too long. You mention the 2083 byte limit in MSIE. That may or may not be a practical problem for you, depending on whether MSIE is a client of your API. (Ie: via Javascript making calls to a JSON API). But in my experience very long URLs will end up breaking mysteriously in several places; proxy caches along the path, even a stateful firewall. If you have absolute control over the clients and network path you can probably live with the dangers of long URLs. If it's a public API, forget it.

Hopefully you can make the straightforward GET work in your environment. You may even want to consider refactoring your API to make the query data smaller.

But what if you can't make the GET work? You propose several alternatives. I would immediately dismiss two of them. Don't put content in the GET request body; too much software will break if you try that, and anyway it violates the very REST spirit you're trying to capture. And I wouldn't use base64 encoding. It may help you work around problem 1, your server not handling some characters in URLs right. But if applied wrong it will actually make your URLs longer, not shorter, compounding problem 2. Even if you do base64 right and include some compression it won't make URLs significantly shorter, and will make the client much more complicated.

Your most practical solution is probably option 3, an HTTP POST. This isn't RESTful; you should be using GETs for read-only queries. And you'll lose some advantages of the REST approach with caching of GETs and the like. On the other hand it will work correctly, and simply, with a large variety of Internet infrastructure and software libraries. You can then pass as much data you want in the POST body either via multipart/form-data encoding, JSON, or XML. (I've built two major public web services using SOAP, which is just XML on POSTs. It's ugly and not RESTful, but it does work reliably.)

REST is a great design paradigm. It's a guideline. If it doesn't fit your app, don't feel you need to stick with it. HTTP is not good at passing large amounts of data to the server with a GET. If you need have giant query parameters, do something else.

7
  • Thanks for the well formulated response. You provide a lot of good information. For simple gets I like to do testing with the address on the browser. You have just made me realize that this may be giving me some issues with the URL that wouldn't be there through Javascript or server-side code. I will need to test out this new theory that it may only be a problem from the browser address bar. Aug 17, 2009 at 21:32
  • Based on the statement in #1 above, I did some more testing and have found that IIS or ASP.NET is stopping requests with bad characters and the request is not reaching my HTTP Handler (it has nothing to do with a browser address bar). I suppose it could be a HTTPModule added by SharePoint. At any rate, these are variables we'd rather not try to control. Aug 18, 2009 at 16:48
  • Nelson, this was a very helpful response. I wish I could have given you the correct answer as well. Aug 19, 2009 at 13:49
  • "option 3 (HTTP POST) is not RESTful". I am not sure this is right. It depends on you model. If you treat a search as being a resource then it definitely is RESTful.
    – redben
    Nov 24, 2014 at 15:22
  • The use case that brought me here is the need to GET a set of N resources identified by GUIDs, which would easily overrun acceptable URL lengths. Everything I've seen points to POST being the best alternative, and @Nelson summarizes it well with this answer.
    – Seth
    Oct 17, 2016 at 0:08
20

If the query is too big to go in the URI, turn your query into a resource (like a saved search). I worked on a restful API for a hotel's booking system; the search query had too many params (preferences, rooming list...etc) so I turned it into a resource that I POST to the server. The server then replies with a URI uniquely identifing the search which body is the posted query + its results:

POST http://hotels.xyz/searches
body <search><query>...</query></search>

Response

201 Created - Location: http://hotels.xyz/searches/someID
Body <search><query>...</query><results>...</results></search>
8
  • Nice idea! I'll consider that next time. Dec 21, 2010 at 16:59
  • The same idea was presented in serialseb's answer to the question HTTP GET with request body which you reference in your question. Sep 28, 2011 at 20:34
  • @redben, this is really a great idea! it is really RESTful. I have a question on the implementation of this: how do you manage the life cycle for the created searches? is it a good to be permanent or session scope? if permanent, how to cleanup obsolete searches? and how does user look up the created searches for reuse? if session scope, it is not straightforward for client to renew the searches after session time out.
    – Shuping
    Aug 22, 2013 at 2:52
  • @Shuping sessions and Rest theoretically don't mix :). In my case, created searches are simply cached, normal caching as you'd use for anything else. So managing the life cycle depends on your stack/technologies.
    – redben
    Aug 22, 2013 at 16:18
  • @redben So, how does your client deal with the cached searches? how do you define when the client should create (post) a search and when should use an existing search? and when an existing search will be expired?
    – Shuping
    Aug 23, 2013 at 2:07
6
+150

I recommend you to read the HTTP 1.1 specification, especially the sections 3.2 Uniform Resource Identifiers and 9.1.1 Safe Methods. Those will hopefully answer your question.


Here’s some additional information:

6
  • Thanks for the links. I have looked through them and have been also doing reading which references them. It hasn't led me to any conclusions yet, but I found the discussion on URL length helpful. Aug 12, 2009 at 20:51
  • Those two new links really helped. They don't quite address special characters, but I feel pretty comfortable with using a query string. It's interesting how The Definitive Guide to GET and POST uses 2048 characters as a guide and dismisses the 256 number. Aug 19, 2009 at 13:49
  • @Kirk Liemohn: Handling of special characters is specified by RFC 1738.
    – Gumbo
    Aug 19, 2009 at 14:37
  • 1
    I think the "Definitive Guide" has moved. This looks like the same post: blog.teamtreehouse.com/the-definitive-guide-to-get-vs-post
    – Chris
    Dec 19, 2013 at 19:30
  • 4
    Perhaps one of the massive linked documents contains the answer to the question, but this reply certainly doesn't.
    – Neutrino
    Jan 14, 2018 at 12:17
5

Use custom HTTP headers with HTTP GET if nothing else works out. HTTP headers can be set by nearly all clients.

Generally it is best to use URL parameters in the query string. Too many URL parameters indicates that you need to split into more fine-granular services.

1
  • 1
    I agree. If the query string has become too long to manage it suggests to me that the data model needs refining. Aug 17, 2009 at 12:32
3

I'd aim for HTTP POST. It's nicely tokenized when it gets to PHP (or whichever you're using) and it doesn't have the size limits the others have.

2
  • I agree, but it bothers me that the defintion/usage of POST as defined in rfc2616 sec9.5 doesn't seem to include this case and a POST is not cacheable. Aug 12, 2009 at 20:29
  • 1
    @Kirk requests with query strings are generally not cached either.
    – aehlke
    Aug 13, 2009 at 14:04
2

If you are generating these long urls on server, you can make use of compression for path info.

So if you have something like /?param1=bla-bla&param2=bla-bla you just compress that parameters and make url looks like /?query=ASsadcnfAFFASFscnsdlc

When you get such request, you just decompress them and parse parameter string

4
  • Is there a suggested compression algorithm? One that is relatively easy to implement in JavaScript, .NET, and Java? Aug 12, 2009 at 20:49
  • check this out: coderanch.com/t/327797/Java-General/java/compress-String
    – glaz666
    Aug 13, 2009 at 7:49
  • Just convert them into a number, and then decide how many characters are valid to use in the URI, and use that as the base for the number. For example, if only 16 characters were valid, it would look like hex. But you can get away with around 60 (a-zA-Z0-9 and some symbols like - and _) so it will compress fairly well. This has been asked before on SO.
    – aehlke
    Aug 14, 2009 at 14:37
  • 2
    shortening to any variably sized querystring only slows people noticing the problem of overflowing the max URL length, it doesn't "solve" it. shortening to a fixed sized querystring is the only way to solve it, which i try to explain my method for in my answer.
    – ifatree
    Aug 17, 2009 at 20:32
2

You should place the parameters in the query string, using an HTTP GET request. Limits in some older web browsers are not a concern, because the only people browsing through an API in a web browser are likely to be developers (or at least technical).

Remember that client applications should not be manipulating the URLs your API provides them. URLs are opaque identifiers to the clients, used only for directing them to where particular resources may be found.

If this is not possible for whatever reason, I would use a POST request with the parameters form-encoded into the body. It won't be entirely RESTful, but assuming your resources are designed properly the impact on client code should be minimal.

6
  • Thanks for the response. Can you explain why you say the POST option would not be entirely RESTful? I'm trying to get a clear understanding of this. Aug 17, 2009 at 13:21
  • 1
    POST should be used as a last resort, when there is not another HTTP method available which better describes the request. In this case, the semantics of GET perfectly match what you're trying to achieve, and thus GET is preferable to POST. You should use POST for this only if there is some technical reason why GET cannot be used (eg: server doesn't support long query strings). Aug 17, 2009 at 16:59
  • 1
    The character limit can still be a problem. For instance, if the API client is Javascript fetching JSON from inside the browser, it might have the same character limitation as the browser itself.
    – Nelson
    Aug 17, 2009 at 20:36
  • Good comments. We do the fetching from both servers and Javascript within browsers so I'll want to work with the lowest common denominator. Aug 17, 2009 at 21:23
  • 1
    Kirk: I encourage you to perform testing, before basing your API on possible browser limits. You could also allow that resource to be accessed using either GET or POST, which will be perfectly RESTful while still allowing legacy browsers to use it. Aug 18, 2009 at 5:19
2

I'd definitely have started where you started: URL shortening. I'd try to shorten the parameter names (?a=XXX;b=YYY;c=zzz); Reencode the entire query to Base64; GZip the Base64; Huffman encode the GZip; ... whatever it takes. Once I got the inkling that shortening won't work for all cases (you've got some dynamic filter-creating system that can be added onto indefinitely, or w/e), then you've got to admit maybe trying to do everything within a single request might not work...

I'm NOT going to suggest you throw multiple GETs with split parameters and try to keep track across requests that way...

The only 'robust' method I CAN suggest is to store/set the requested querystring in one request (POST) and have it return a fixed-sized ID (or guid) that identifies the request parameter location in your data store (filterID), then make the actual GET request using the filterID token instead of the full filter query string value. This will allow all kinds of neat things like cacheing responses based on filterID so you could (in theory) reuse the same filters later (instead of re-entering them by hand, just save a "label" along with the filter body and select from the last 5 filters by label), or at least keep them stored with your data so that each time you refresh the page it's not re-sending the entire filter request.

1
  • Wow. If resorting to a POST, why do a separate call with an ID/GUID when the POST can return the entire data set? I guess it allows for multiple gets on that ID, but seems like overkill. Just curious if I'm missing something. Aug 17, 2009 at 21:25
2

Roy Fielding would likely approve of using POST in this situation, but you'd have to ask him.

In general, most applications that involve user-supplied data being supplied to the server are not safe. The only exception is when the information is in the form of generalized query parameters, for which there is a trade-off between GET and POST that usually involves the size of the parameter content. GET is only desirable for those cases where the parameters can be expressed as a meaningful URI.

0

base64 should do it. other wise use the %-sign which is standard.

0

Consider supporting :
- GET requests with short query string
- POST requests with long query string into the body and X-HTTP-Method-Override: GET (https://en.wikipedia.org/wiki/List_of_HTTP_header_fields)

Beware not mixing "POST /orders" that is a bulk creation of new orders and "POST /orders" with a "X-HTTP-Method-Override: GET" that is a search of order.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.