102

I have a site that treats / and %2F in the path portion (not the query string) of a URL differently. Is this a bad thing to do according to either the RFC or the real world?

I ask because I keep running into little surprises with the web framework I'm using (Ruby on Rails) as well as the layers below that (Passenger, Apache, e.g., I had to enable ALLOW_ENCODED_SLASHES for Apache). I am now leaning toward getting rid of the encoded slashes completely, but I wonder if I should be filing bug reports where I see weird behavior involving the encoded slashes.

As to why I have the encoded slashes in the first place, basically I have routes such as this:

:controller/:foo/:bar

where :foo is something like a path that can contain slashes. I thought the most straightforward thing to do would be to just URL escape foo so the slashes are ignored by the routing mechanism. Now I am having doubts, and it's pretty clear that the frameworks don't really support this, but according to the RFC is it wrong to do it this way?

Here is some information I have gathered:

RFC 1738 (URLs):

Usually a URL has the same interpretation when an octet is represented by a character and when it encoded. However, this is not true for reserved characters: encoding a character reserved for a particular scheme may change the semantics of a URL.

RFC 2396 (URIs):

These characters are called "reserved", since their usage within the URI component is limited to their reserved purpose. If the data for a URI component would conflict with the reserved purpose, then the conflicting data must be escaped before forming the URI.

(does escaping here mean something other than encoding the reserved character?)

RFC 2616 (HTTP/1.1):

Characters other than those in the "reserved" and "unsafe" sets (see RFC 2396 [42]) are equivalent to their "% HEX HEX" encoding.

There is also this bug report for Rails, where they seem to expect the encoded slash to behave differently:

Right, I'd expect different results because they're pointing at different resources.

It's looking for the literal file foo/bar in the root directory. The non escaped version is looking for the file bar within directory foo.

It's clear from the RFCs that raw vs. encoded is the equivalent for unreserved characters, but what is the story for reserved characters?

3

7 Answers 7

45

From the data you gathered, I would tend to say that encoded / in an URI are meant to be seen as / again at the application or CGI level.

That's to say, that if you're using Apache with mod_rewrite for instance, it will not match pattern expecting slashes against URI with encoded slashes in it. However, once the appropriate module/cgi/... is called to handle the request, it's up to it to do the decoding and, for instance, retrieve a parameter including slashes as the first component of the URI.

If your application is then using this data to retrieve a file (whose filename contains a slash), that's probably a bad thing.

To sum up, I find it perfectly normal to see a difference of behaviour in / or %2F as their interpretation will be done at different levels.

2
  • 2
    This is pretty much what I've been thinking too. Unfortunately it looks like there is not much support for doing it this way in the real world. I'll keep working around for now but if I were to start over I would try a different escaping mechanism.
    – user85509
    Dec 25, 2009 at 1:01
  • @user85509 I wonder if I should be filing bug reports where I see weird behavior involving the encoded slashes. – Please do each time you observe behaviour not in line with the RFC which clearly states that / as a reserved character is different to its percent encoded form %2F. Dec 8, 2020 at 11:26
34

The story of %2F vs / was that, according to the initial W3C recommendations, slashes «must imply a hierarchical structure»:

The slash ("/", ASCII 2F hex) character is reserved for the delimiting of substrings whose relationship is hierarchical. This enables partial forms of the URI.

Example 2

The URIs

http://www.w3.org/albert/bertram/marie-claude

and

http://www.w3.org/albert/bertram%2Fmarie-claude

are NOT identical, as in the second case the encoded slash does not have hierarchical significance.

1
  • 1
    What about queries? Like, a query doesn't have hierarchical structure. They're just variables. It's a petty that it's not explained by the W3C recommendation.
    – Eksapsy
    Oct 18, 2020 at 23:07
11

I also have a site that has numerous URLs with URL encoded characters. I am finding that many web APIs (including Google webmaster tools and several Drupal modules) trip over URL encoded characters. Many APIs automatically decode URLs at some point in their process and then use the result as a URL or HTML. When I find one of these problems, I usually double encode the results (which turns %2f into %252f) for that API. However, this will break other APIs which are not expecting double encoding, so this is not a universal solution.

Personally I am getting rid of as many special characters in my URLs as possible.

Also, I am using id numbers in my URLs which do not depend on URL decoding:

example.com/blog/my-amazing-blog%2fstory/yesterday

becomes:

example.com/blog/12354/my-amazing-blog%2fstory/yesterday

in this case, my code only uses 12354 to look for the article, and the rest of the URL gets ignored by my system (but is still used for SEO.) Also, this number should appear BEFORE the unused URL components. that way, the URL will still work, even if the %2f gets decoded incorrectly.

Also, be sure to use canonical tags to ensure that URL mistakes don't translate into duplicate content.

1
  • This method seems to work pretty well for reddit.com.
    – StockB
    Dec 13, 2016 at 14:46
4

If you use Tomcat, add

-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true

in VM properties.

https://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security

3

What to do if :foo in its natural form contains slashes? You wouldn't want it to Isn't that the distinction the recommendation is attempting to preserve? It specifically notes,

The similarity to unix and other disk operating system filename conventions should be taken as purely coincidental, and should not be taken to indicate that URIs should be interpreted as file names.

If one was building an online interface to a backup program, and wished to express the path as a part of the URL path, it would make sense to encode the slashes in the file path, as that is not really part of the hierarchy of the resource - and more importantly, the route. /backups/2016-07-28content//home/dan/ loses the root of the filesystem in the double slash. Escaping the slashes is the appropriate way to distinguish, as I read it.

-1

try below obj=here your any object...

Gson gson = new GsonBuilder().disableHtmlEscaping().setPrettyPrinting().create();
       return Base64.getUrlEncoder().withoutPadding().encodeToString(gson.toJson(obj).getBytes());
2
  • It seems unlikely that this question is about Java. They say they are using CGI. Dec 28, 2022 at 9:17
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Dec 28, 2022 at 20:42
-3

encodeURI()/decodeURI and encodeURIComponent()/decodeURIComponent are utility functions to handle this. Read more here https://stackabuse.com/javascripts-encodeuri-function/

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.