26

I'm trying to upload very large files (>2GB) to my WebAPI application (Running on .NET 4.5.2, Windows 2012R2).

Setting the httpRuntime maxRequestLength property is of no use because it's only working with files smaller than 2GB.

I'm currently using a custom MultipartFormDataStreamProvider to read the entire stream on the Server and I've already turned off buffering using a custom WebHostBufferPolicySelector.

What I discovered is that ASP.NET (or WebAPI for that matter) uses a HttpBufferlessInputStream under the Hood which has a field called _disableMaxRequestLength. If I set this value to to true (via reflection), I can stream files of any size.

However, fiddling around these with these internas is clearly not a good way.

The HttpRequest class used for the request has a method called GetBufferlessInputStream which has an overload that allows to disable the maxRequestLength.

My question is: How can I get the WebAPI to use this overload instead of the Standard one?

Is there any way to replace the Default HttpRequest or HttpContext class? Or do I really Need to use reflection for the whole stuff?

This is the code I'm currently using to disable the maxRequestLength:

    private void DisableRequestLengthOnStream(HttpContent parent)
    {
        var streamContentProperty = parent.GetType().GetProperty("StreamContent", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
        if (streamContentProperty == null) return;
        var streamContent = streamContentProperty.GetValue(parent, null);
        if (streamContent == null) return;
        var contentProperty = typeof(StreamContent).GetField("content", BindingFlags.Instance | BindingFlags.NonPublic);
        if (contentProperty == null) return;
        var content = contentProperty.GetValue(streamContent);
        if (content == null) return;
        var requestLengthField = content.GetType().GetField("_disableMaxRequestLength", BindingFlags.Instance | BindingFlags.NonPublic);
        if (requestLengthField == null) return;
        requestLengthField.SetValue(content, true);
    }
5
  • 1
    +1 Very interesting question. Out of curiosity, have you considered just using FTP to do the transfer? Sep 9, 2014 at 20:43
  • That is not possible in my Scenario... Sep 9, 2014 at 20:44
  • By 'the standard one', do you mean using an InputString?
    – Justin R.
    Sep 9, 2014 at 20:50
  • @JustinR.No. There are two GetBufferlessInputStream methods on the HttpRequest class. One takes a Parameter wich Switches the maxRequestCheck on/off, and the other one just Returns the HttpBufferlessInputStream with the Check turned on. Sep 9, 2014 at 20:55
  • @Henning Krause Give this a read. strathweb.com/2012/09/…
    – Brian
    Sep 9, 2014 at 21:20

3 Answers 3

7

Ok, I found a pretty simple solution. The answer from @JustinR. would work, of course. But I wanted to continue to use a MultipartFormDataStreamProvider because that handles all the MIME stuff.

The solution is to simply create a new StreamContent instance with the correct bufferless Input stream and populate it with the headers from the original Content:

var provider = new MultipartFormDataStreamProvider(path);
var content = new StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(true));
foreach (var header in Request.Content.Headers)
{
    content.Headers.TryAddWithoutValidation(header.Key, header.Value);
}

await content.ReadAsMultipartAsync(provider);
9
  • Doesn't break the 2GB barrier...soooo
    – Nicholi
    May 30, 2015 at 0:10
  • Thanks, works for me, I have uploaded a 21 megabyte xml file (in dev anyway).
    – S Meaden
    Aug 6, 2015 at 18:36
  • 2
    The IIS has a limit at 2GB... no way around. I changed my implementation to perform chuncked uploads (jquery/blueimp) Aug 19, 2015 at 18:08
  • You're better off that way anyway just for easier resume capability, imo.
    – Bon
    Aug 28, 2015 at 16:40
  • 1
    @HenningKrause Thx for sharing the code. Did you try Owin Selfhost instead of IIS?
    – smedasn
    Oct 9, 2017 at 23:38
2

According to MSDN, the way to read an unlimited stream length is HttpRequest.GetBufferlessInputStream. You could do something like:

public void ReadStream(HttpContext context, string filePath)
{
    using (var reader = new StreamReader(context.Request.GetBufferlessInputStream(true)))
    using (var filestream = new FileStream(filePath, FileMode.Create, FileAccess.Write, FileShare.Read, 4096, true))
    using (var writer = new StreamWriter(filestream))
    {
        var readBuffer = reader.ReadToEnd();
        writer.WriteAsync(readBuffer);
    }
}
3
  • 5
    I know its only an example, but that var readBuffer = reader.ReadToEnd(); doesn't strike me as facilitating reading "unlimited stream length"; I've had problems with (single) large allocations, even with the memory available. :( Sep 9, 2014 at 21:10
  • @MeirionHughes I absolutely agree. I have similar code that streams this progressively into the file. It also does so asynchronously. However, all of the additional code to do so distracts from the usage of the function, and I wanted to make the example clear.
    – Justin R.
    Sep 10, 2014 at 3:30
  • With the above code I am getting below error "The stream is currently in use by a previous operation on the stream."
    – Thavudu
    Jun 28, 2018 at 4:12
1

IMHO there is no easy way to do that.

The call to GetBufferlessInputStream is buried deep inside HttpControllerHandler, which is the lowest possible layer of ASP.NET Web API (it's an HTTP Handler on top of which the entire Web API stack is built.

You can see the code here.

As you see it's full of statics, long methods with nested logical conditions, internals and privates, so it's not really customisable at all.
While the entire HttpControllerHandler in Web API can theoretically be replaced with a custom implementation (this is done inside HttpControllerRouteHandler - by overriding the GetHttpHandler method), it's de facto impossible (you could try to internalize this code in your application, but you'll end up dragging lots of extra internal classes too).

The best thing (and I dread to say that) that comes to my mind is to modify the source HttpControllerHandler class to use the overload of GetBufferlessInputStream that disables the request length limit and recompile the System.Web.Http.WebHost assembly, and deploy that modded version with your app.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.