Dealing with large files in ASP.NET Web API

Strath

September 24th, 2012

Dealing with large files in ASP.NET Web API

Because if you are not careful, it's easy to OOM

There are a couple of things to consider when dealing with large file uploads and ASP.NET Web API. First of all, there are typical issues related to hosting within IIS context, and the limitations imposed on the service by default by the ASP.NET framework.

Secondly, there is the usual problem of having to deal with out of memory issues when allowing external parties or users to upload buffered large files. As a consequence, streamed uploads or downloads can greatly improve the scalability of the solutions by eliminating the need for large memory overheads on buffers.

Let’s have a look at how to tackle these issues.

1. Configuring IIS

If you are using Web Hosting within IIS context, you have to deal with maxAllowedContentLength and/or maxRequestLength – depending on the IIS version you are using.

Unless explicitly modified in web.config, the default maximum IIS 7 upload filesize is 30 000 000 bytes (28.6MB). If you try to upload a larger file, the response will be 404.13 error.

If you use IIS 7+, you can modify the maxAllowedContentLength setting, in the example below to 2GB:

2. ASP.NET

Additionally, ASP.NET runtime has its own file size limit (which was also used by IIS 6, so if you use that version, that’s the only place you need to modify) – located under the httpRuntime element of the web.config.

By default it’s set to 4MB (4096kB). You need to change that as well (it’s set in kB not in bytes as the previous one!):

It’s worth remembering, that out of the two configuration elements, whichever request length limit is smaller, will take precedence.

3. Out of memory through buffering

Web API, by default buffers the entire request input stream in memory. Since we just increased the file upload limit to 2GB, you can easily imagine how this could cause OOM really quickly – i.e. through a couple of simultaneous uploads going on.

Fortunately there is an easy way to force Web API into streamed mode of dealing with the uploaded files, rather than buffering the entire request input stream in memory.

We were given control over this through this commit (yes, it’s part of Web API RTM) and Kiran Challa from MSFT had a very nice short post about how to use that. Basically, we need to replace the default IHostBufferPolicySelector with our custom implementation.

That’s the service that makes a decision whether a given request will be buffered or not – on a per request basis. We could either implement the interface directly, or modify the only existing implementation – System.Web.Http.WebHost.WebHostBufferPolicySelector. Here’s an example:

In the case above, we check if the request is routed to the UploadingController, and if so, we will not buffer the request. In all other cases, the requests remain buffered.

You then need to register the new service in GlobalConfiguration (note – this type of service needs to be global, and the setting will not work when set in the per-controller configuration!):

I will not go into details about how to upload files to Web API – this was already covered in one of my previous posts. Instead, here is a simple example of how resource management on your server improves when we switch off input stream buffering. Below is memory consumption when uploading a 400MB file on my local machine, with default Web API settings:

and here, after replacing the default IHostBufferPolicySelector:

So what happens under the hood? Well, in the Web API pipeline HttpControllerHandler makes the decision whether to use buffering of the request input stream or not by inspecting the lazily loaded instance of policy selector (a.k.a. our IHostBufferPolicySelector). If the request should not be buffered (like ours), it uses the good old HttpRequest.GetBufferlessInputStream from the System.Web namespace. As a result the underlying Stream object is returned immediately and we can begin processing the request body before its complete contents have been received.

Unfortunately, while it seems very straightforward, it’s not all rainbows and unicrons. As you have probably already noticed, all this, is very heavily relying on System.Web making it unusable outside of the hosting context of ASP.NET. As a result, the beautiful symmetry of Web API in self- and web-host scenarios is broken here.

5. Self host

If you are trying to achieve any of the above in self hosting, you are forced to use a workaround – and that is setting your self host’s configuration to:

TransferMode is a concept from WCF, and comes in four flavors as shown in the code snippet above – Buffered (default), streamed, streamed request and streamed response.

What’s worth noting, is that while for HTTP transports like Web API, there are not bigger repercussions in switching to Streamed mode, for the TCP services Streamed mode also changes the the native channel shape from IDuplexSessionChannel (used in buffered mode) to IRequestChannel and IReplyChannel (but that’s beyond the scope of our Web API discussion here).

Anyway, the downside of this solution is that TransferMode is a global setting for all requests to your self hosted service and you can’t decide on a per request basis whether to buffer or not (like we did in the example before). A workaround would be to set up two parallel self-host instances, one used for general purposes, the other for upload only, but again, that’s far from perfect and in many cases not even acceptable at all.

6. Using HttpClient

Finally, a short note on buffering on the client side – i.e. reading a file asynchronously. By default, HttpClient in version 4 of the .NET framework will buffer the body so if you want to have control over whether the response is streamed or buffered (and prevent your client from OOM-ing when dealing with large files), you’d be better off using HttWebRequest directly instead, as there you have granular control through HttpWebRequest.AllowReadStreamBuffering and HttpWebRequest.AllowWriteStreamBuffering properties. As a matter of fact, HttWebRequest lies under HttpClient, but these properties are not exposed to the consumer of HttpClient.

In .NET 4.5 HttpClient does not buffer the response by default, so there is no issue.

Summary

Dealing with large files in Web API and – for that matter – in any HTTP scenario, can be a bit troublesome and sometimes even lead to disastrous service outages, but if you know what you are doing there is no reason to be stressed about anything.

On the other hand, request/response stream buffering is one of those areas where Web API still drags a baggage of System.Web with it in the web host scenarios, and therefore you need to be careful – as not everything you can do in web host, can be achieved in self host.

Be Sociable, Share!

  • JV

    Max Int32 is 2147483647 ;-)

  • Tom

    Terrific post. Thank you.

  • http://ilovedevelopment.blogspot.com Luke Baughan

    Great article – possibly worth pointing out that maxAllowedContentLength is in Bytes http://msdn.microsoft.com/en-us/library/ms689462(v=vs.90).aspx and MaxRequestLength is in KB http://msdn.microsoft.com/en-us/library/system.web.configuration.httpruntimesection.maxrequestlength.aspx – posting the same value into both (2147483648 in the example) doesn’t represent the same length! The equivalent MaxRequestLength would be 2097152.

    • Filip W

      ah shoot! good catch!
      updated the post – thx

  • Palak Chokshi

    Great article because I have been struggling with this for a while. If you could bear with me for the long post I would really appreciate some help. Here goes
    let’s start from the system configuration

    IIS 7.5
    ASP.NET MVC 3
    Visual Studio 2010 SP1
    Windows Server 2008 R2

    I am trying to upload a large file using Ajax FormData and get a 413 request entity too large error. Any file over 1MB gets this error.

    First the Ajax Call
    self.createMediaItem = function (parentCollectionID, title, file) {

    var formData = new FormData();
    formData.append(‘appToken’, ‘MyWebApp’); //internal use
    formData.append(‘userToken’, MyApp.SessionToken); //Internal use
    formData.append(‘parentCollectionID’, parentCollectionID);
    formData.append(‘Title’, title);
    formData.append(‘file’, file);
    $.ajax({
    url: ‘/api/Item/Media/Create/’,
    contentType: false,
    cache: false,
    processData: false,
    async: false,
    data: formData,
    type: ‘POST’,
    success: function (result) {
    if (!result.IsSuccess)
    MyApp.showErrorDialog(result.Message, true);
    },
    error: function (jqxhr, textStatus, errorThrown) {
    }
    });
    };

    This call works for files less than 1MB

    Now let’s see my web.config settings related to large file uploads

    I’ve even set the uploadReadAheadSize to a large number.

    With these configurations if I try to upload a large file using Visual Studio 2010 SP1′s web server (Debug mode) I can upload the file without problems but if I deploy the app on Windows Server 2008 R2 IIS 7.5 I get the 413 Request Entity too large error in the console. I am using Google Chrome for testing.

    Thanks

  • Palak Chokshi

    Seems like my web.config settings got stripped by the commenting system in my previous comment. Anyways I’ve set the maxRequestLength to 100MB and maxAllowedContentLength to 100MB
    And the request still fails with a 413 Request Entity too large error.

  • Domlia

    Nice overview, thanks.

  • Nilesh Joshi

    Exhaustive details… nice article, thanks

  • Walter Johnston

    ASP.NET is a great platform with lots of advantages, it provides Full support to XML, CSS and many other new and established Web standards.

  • http://timothyferrell.com Tim Ferrell

    When runnning the code you supplied, and placing it into my Global.asax.cs file, I get the following error: The service type WebHostBufferPolicySelector is not supported.

    I’m not sure where to turn on this one. Any ideas?

    • http://twitter.com/danielcrenna Daniel Crenna

      It’s because you copy-pasted ‘IHostBufferPolicySelector’ which is part of the framework already, and you’re effectively trying to register an unknown type with configuration.

  • Gary Metzker

    So this was working good for me, however it broke once I added the WIF modules: WSFederationAuthenticationModule and SessionAuthenticationModule. If I disable these modules in the web.config the BufferlessInputStream works again.

    Any idea why WIF modules breaks it? It appears maybe these modules are trying to invoke one of the stream methods.

    When I try to upload I get an exception from:

    HttpRequest.GetInputStream:

    This method or property is not supported after HttpRequest.Form, Files, InputStream, or BinaryRead has been invoked.

    at System.Web.HttpRequest.GetInputStream(Boolean persistEntityBody, Boolean disableMaxRequestLength)
    at System.Web.HttpRequest.GetBufferlessInputStream()
    at System.Web.Http.WebHost.HttpControllerHandler.ConvertRequest(HttpContextBase httpContextBase)
    at System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(HttpContextBase httpContextBase, AsyncCallback callback, Object state)
    at System.Web.Http.WebHost.HttpControllerHandler.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext httpContext, AsyncCallback callback, Object state)
    at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

    • Gary Metzker

      Specifically when I try and use the System.IdentityModel.Services.WSFederationAuthenticationModule in my Web.Config, the Httprequest.GetBufferlessInputStream blows up.

    • Gary Metzker

      I believe the WSFederationAuthenticationModule is attempting to read the request input stream before the IHostBufferPolicySelector can invoke the Bufferless input stream.

      If I create an HttpModule and make sure it is added before the WSFAM module then I can call Request.GetBufferlessInputStream() to force it. I just need some better conditional logic. This seems like a hack.

      Is there no way to get IHostBufferPolicySelector to run before the HTTP module (in this case it gets invalidated by something WSFam is doing).

    • Gary Metzker

      I found a solution to my issue by creating an HttpModule that on HttpApplication.BeginRequest it forces a bufferless input stream on any requests that need uploads for. My installing the module before WSFederationAuthenticationModule I can assure that Request has already got a chance to cache the appropriate BufferlessInputStream before WSFam has a chance to screw it up.

      If any one wants more detail see my post here: http://forums.asp.net/t/1859232.aspx/1?HttpRequest+GetBufferlessInputStream+or+does+not+work+when+using+WSFederationAuthenticationModule

      Or message and I can post a solution on git.

  • http://www.facebook.com/jim.wright.965 Jim Wright

    Great post! Fixed it for me.

  • http://twitter.com/mcintyre321 Harry McIntyre

    Do you know if streaming mode would work when hosted in IIS, or if request would be buffered?

    • http://www.strathweb.com/ Filip W

      if you provide a custom WebHostBufferPolicySelector then, yeah, it would. In fact, that’s only for web hosting mode.

      • http://twitter.com/mcintyre321 Harry McIntyre

        Cool, thanks! I have written a reverse proxy in WebApi which I want to use for logging and filtering requests. I was worrying that I would lose the streaming capabilities if I hosted in IIS (which should take care of SSL). Cheers. PS did you ever get my email about ApiController Formatter parameter binding?

  • Pingback: Microsoft Exam 70-487 – Developing Windows Azure and Web Services Study Guide « The Pragmatic Developer

  • Pingback: Upload files to Azure Blob storage through WebApi without accessing local filesystem | BlogoSfera

  • Pingback: Upload files to Azure Blob storage through WebApi without accessing local filesystem - Windows Azure Blog

  • Pingback: How to get uploaded file stream in web api - Windows Azure Blog

  • Rohan

    How do you add System.web.http and System.net.http to VS 2013. My application is using .Net 3.5 I can’t find above dlls in Add reference. I am using webforms.

  • Aravinth Santosh

    thank you it helps me

  • InternetSavage

    Seems simple enough just wish I could get it to work. For some reason in my case the UseBufferedInputStream is never even called when I try to upload and when it is called for other API calls the value for context.Request.RequestContext.RouteData.Values["controller"] is always null and breaks the request.