When I'm uploading large files to my web api in ASP.NET Core, the runtime will load the file into memory before my function for processing and storing the upload is fired. With large uploads this becomes an issue as it is both slow and requires more memory. For previous versions of ASP.NET there are some articles on how to disable the buffering of requests, but I'm not able to find any information on how to do this with ASP.NET Core. Is it possible to disable the buffering of requests so I don't run out of memory on my server all the time?
-
I write my file upload backend to support uploading files in small chunks according to the api for flowjs– jltremApr 5, 2016 at 23:57
-
Hello @jltrem, could you please share asp.net core controller and angular code that handles files uploaded using flowjs?– Santosh Prasad SahSep 17, 2016 at 8:42
3 Answers
Use the Microsoft.AspNetCore.WebUtilities.MultipartReader
because it...
can parse any stream [with] minimal buffering. It gives you the headers and body of each section one at a time and then you do what you want with the body of that section (buffer, discard, write to disk, etc.).
Here is a middleware example.
app.Use(async (context, next) =>
{
if (!IsMultipartContentType(context.Request.ContentType))
{
await next();
return;
}
var boundary = GetBoundary(context.Request.ContentType);
var reader = new MultipartReader(boundary, context.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
// process each image
const int chunkSize = 1024;
var buffer = new byte[chunkSize];
var bytesRead = 0;
var fileName = GetFileName(section.ContentDisposition);
using (var stream = new FileStream(fileName, FileMode.Append))
{
do
{
bytesRead = await section.Body.ReadAsync(buffer, 0, buffer.Length);
stream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
section = await reader.ReadNextSectionAsync();
}
context.Response.WriteAsync("Done.");
});
Here are the helpers.
private static bool IsMultipartContentType(string contentType)
{
return
!string.IsNullOrEmpty(contentType) &&
contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
}
private static string GetBoundary(string contentType)
{
var elements = contentType.Split(' ');
var element = elements.Where(entry => entry.StartsWith("boundary=")).First();
var boundary = element.Substring("boundary=".Length);
// Remove quotes
if (boundary.Length >= 2 && boundary[0] == '"' &&
boundary[boundary.Length - 1] == '"')
{
boundary = boundary.Substring(1, boundary.Length - 2);
}
return boundary;
}
private string GetFileName(string contentDisposition)
{
return contentDisposition
.Split(';')
.SingleOrDefault(part => part.Contains("filename"))
.Split('=')
.Last()
.Trim('"');
}
External References
-
1It appears that this code works perfectly on dnx451 but it has a memory leak on dnxcore50. Might be something that is being fixed for RC2 though.– MartinApr 9, 2016 at 12:54
-
1GetFileName() from the snippet above causes server to buffer the whole file. I replaced it with randomized file name, but after writing all chunks to file I see that memory usage increased by the ammount equal to file size. It may be actually ok, if further GC will get rid of it. But seems like not ok– EvAlexMay 13, 2016 at 7:17
-
@EvAlex
GetFileName
just parses a string. Is there something else that's happening before or after that could be causing the server to buffer the whole file? May 13, 2016 at 12:37 -
@ShaunLuttin Thx for sharing the code. Does this also work for files larger than 4 gb?– smedasnOct 9, 2017 at 23:30
-
1
Shaun Luttin's answer is great, and now much of the work he's demonstrated is provided by ASP.Net Core 2.2.
Get the boundary:
// Microsoft.AspNetCore.Http.Extensions.HttpRequestMultipartExtensions
var boundary = Request.GetMultipartBoundary();
if (string.IsNullOrWhiteSpace(boundary))
return BadRequest();
You still get a section as follows:
var reader = new MultipartReader(boundary, Request.Body);
var section = await reader.ReadNextSectionAsync();
Check the disposition and convert to FileMultipartSection:
if (section.GetContentDispositionHeader())
{
var fileSection = section.AsFileSection();
var fileName = fileSection.FileName;
using (var stream = new FileStream(fileName, FileMode.Append))
await fileSection.FileStream.CopyToAsync(stream);
}
-
2The editor marked error on this line
if (section.GetContentDispositionHeader())
Cannot convert ContentDepositionHeaderValue to bool Mar 3, 2019 at 22:35 -
In your Controller
you can simply use Request.Form.Files
to access the files:
[HttpPost("upload")]
public async Task<IActionResult> UploadAsync(CancellationToken cancellationToken)
{
if (!Request.HasFormContentType)
return BadRequest();
var form = Request.Form;
foreach(var formFile in form.Files)
{
using(var readStream = formFile.OpenReadStream())
{
// Do something with the uploaded file
}
}
return Ok();
}
-
Using Request.Form.Files will cause the uploaded files to be saved to disk if they are bigger than a certain size. ISTR seeing 64MB mentioned somewhere. Apr 19, 2018 at 10:46
-
2