19

I have created a sample asp.net application and try to scrape data from my server using httpwebrequest. But some times I got this above error. I have done some searches on google but all are saying that you should add the property "<httpWebRequest useUnsafeHeaderParsing="true" />" in web.config.

This property have 'UNSAFE" word, so I am too worried about this. I can't add this is in my site configuration. Is there any other option to read the response of my scrape URL. Please let me know how can it be possible without "<httpWebRequest useUnsafeHeaderParsing="true" />"

Thanks in advance, Laxmilal Menaria

1
  • I've seen a different case with the same error - Imperva's Incapsula caching mechanism tried to cache a specific response. so, instead of redirecting to the real server, Incapsula replied with the same response, but somehow, the headers were a little different and was not complying with Windows CR-LF (\r\n).
    – itsho
    Jul 9, 2017 at 10:20

4 Answers 4

26

This is certainly a server problem-- the server is not following the HTTP specification and the .NET client is flagging this as a potential problem. "Unsafe" is I think somewhat of a misnomer. There's not really a big security issue here, only non-compliance with the RFC which is bad but unfortunately not rare.

So, as you found in Google, the way to get around the problem is to apply the following configuration change:

<configuration> 
 <system.net> 
  <settings> 
   <httpWebRequest useUnsafeHeaderParsing="true" /> 
  </settings> 
 </system.net> 
</configuration>
1
  • 1
    If you're wondering where to add this, look in the solution explorer for App.config. Jun 16, 2015 at 11:22
2

One way to debug this (and to make sure it is the protocol violation that is causing the problem), is to use Fiddler (Http Web Proxy) and see if the same error occurs. If it doesn't (i.e. Fiddler handled the issue for you) then you should be able to fix it using the UseUnsafeHeaderParsing flag.

If you are looking for a way to set this value programatically see the examples here: http://o2platform.wordpress.com/2010/10/20/dealing-with-the-server-committed-a-protocol-violation-sectionresponsestatusline/

0
2

Another possibility: when doing a POST, the server responds with a 100 continue in an incorrect way.

This solved the problem for me:

request.ServicePoint.Expect100Continue = false;

0

You may want to check the content of your headers. Try adding the following, as suggested by this link:

Accept: text/html, application/xhtml+xml, */*

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.