& is a reserved character in html therefore everywhere I have url's pointing to some path with querystring I put & instead of & so that I get valid HTML.
There are a many different crawlers that goes over the website and access this url's but they don't use html decode methods to get the correct url values so they make requests to my website with:
mywebsite.com/?p1=v1&p2=v2
instead of
mywebsite.com/?p1=v1&p2=v2
Right now I am responding with the error page as the robots that makes this requests are of no interest to me.
But my question is, what are the best practice to handle this kind of requests?
Do you know if there is of any use to support handling this kind of requests? ( for example are there any popular crawlers or browsers that doesn't properly converts this url's ?)
;
) instead of&
. But supporting;
usually requires a configuration change on the server. You would probably still have to support&
by redirecting to;
, since when submitting an HTML form withtype=GET
, the browser sends you to a&
URL. [link]w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.2.2