9

& is a reserved character in html therefore everywhere I have url's pointing to some path with querystring I put & instead of & so that I get valid HTML.

There are a many different crawlers that goes over the website and access this url's but they don't use html decode methods to get the correct url values so they make requests to my website with:

mywebsite.com/?p1=v1&p2=v2

instead of

mywebsite.com/?p1=v1&p2=v2

Right now I am responding with the error page as the robots that makes this requests are of no interest to me.

But my question is, what are the best practice to handle this kind of requests?

Do you know if there is of any use to support handling this kind of requests? ( for example are there any popular crawlers or browsers that doesn't properly converts this url's ?)

3
  • 1
    Is this in attributes? Don't do that.
    – Ry-
    Jun 18, 2012 at 14:07
  • 9
    You could configure your web server to fix it but it seems like a crawler that isn't figuring that out on it's own probably isn't worth the trouble
    – Cfreak
    Jun 18, 2012 at 14:10
  • In fact, W3C recommends using semicolon (;) instead of &. But supporting ; usually requires a configuration change on the server. You would probably still have to support & by redirecting to ;, since when submitting an HTML form with type=GET, the browser sends you to a & URL. [link]w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.2.2 Jun 20, 2012 at 13:25

2 Answers 2

1

I think you can expect that any major crawler is able to handle valid escaped URLs. So I won't worry about the rest.

If you really like to then you may want to add rewrite rules to your Apache or whatever you use. But this may lead to other problems when an URL really contains the charsequence & and got replaced with & by your rewrite rule for error.

In my opinion it is better to leave this untouched. It is not your fault and when you do not really care about these crawler - so what? :)

-3

Yes & is a reserved character but your not gonna put it in website links.

Correct

<a href="mywebsite.com/?p1=v1&p2=v2">mywebsite.com/?p1=v1&amp;p2=v2</a>

Incorrect

<a href="mywebsite.com/?p1=v1&amp;p2=v2">mywebsite.com/?p1=v1&amp;p2=v2</a>
3
  • Not sure of what your point is, but &p2=v2 is an invalid HTML entity. Don't confuse escaping with actual contents. Jul 3, 2012 at 9:27
  • Directly from validator.w3.org on your correct version: Error: & did not start a character reference. (& probably should have been escaped as &amp;.) <a href="mywebsite.com/?p1=v1&p2=v2">mywebsite.com/?p1=v1&amp;p2=v2</a>
    – Dorin
    Jul 3, 2012 at 14:11
  • In fact, your correct version is incorrect and your incorrect version is correct. Sounds contradictory, but it's true! :) Jul 3, 2012 at 16:10

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.