HttpHandler or HttpModule for file upload, large files, progress indicator?

Question

_{Sunday, September 22, 2002 6:36 PM}

I would like to use the built-in file upload handling features of ASP.NET, however I have found that even if you increase the maxRequestLength and executionTimeout values, after 180 seconds, the aspnet_wp is recycled because it thinks it is deadlocked. In addition, there is the issue that there is no feedback to the client of whether the upload is proceeding or encountered an error (the globe just sits and spins). What would be ideal would be to asynchronously intercept the incoming request in an HttpModule or HttpHandler, and since the content-length is first, be able to set some current bytes/total bytes value in the application. Then with the help of a little client-side javascript that opens a self-refreshing "progress indicator" modelessdialogbox that could read these values every few seconds, not only would the deadlock issue be solved, but the client would also get some visual indication that the upload is proceeding. After scouring everything I could find in the way of documentation and newsgroups, I have not been able to locate any information on how to just "count the bytes" of a request as they come in, but it seems like the async StartBeginRequest event may be the key. This sounds like a real common scenario, can anyone help me? Thanks in advance!

All replies (184)

_{Friday, September 27, 2002 10:18 AM}

Somebody in this forum most have dealt with this issue, and I could really use some advice. Perhaps my last post was not worded correctly, here's a simplification: The built-in file upload handling features of ASP.NET load the entire request into server memory before you can save the file. If the file is large, this can cause execution timeouts, or at the very least, consume *huge* amounts of server memory. What I am looking for is a way to intercept an incoming request (asynchronously, I suppose) so that I can process the incoming bytes in smaller chucks, saving each chunk to the file as it is received, rather than waiting for the entire request to get loaded into server memory before processing. On the client side, I am using the standard tag, so the uploaded file(s) are sent in a multipart/form-encoded POST to the server. Has anybody dealt with processing incoming request bodies as they arrive, rather than after the entire request is received? Is there a way to use a custom HttpHandler to do this? Perhaps with the asynchronous BeginProcessRequest() function? Has anyone here done asynchronous processing? Any help would be *greatly* appreciated!

_{Monday, September 30, 2002 4:27 PM}

See: http://www.asp.net/ControlGallery/ControlDetail.aspx?Control=222&tabindex=2 They've figured it out. This control is nice and should do all that you need to. D

_{Monday, November 11, 2002 11:59 AM}

Did you ever resolve this problem? We have been the same problem - we use modem access and occasionally upload files of around 2MB into a SQL2000 DB. Often this fails (not due to the internet connection), but due to some timeout or other. We thought it may be the session timeout, but it happens before the session expires. I have also increased the scripttimeout and execution timeout properties without success.... If you have found a solution I would be very interested to hear about it. James

_{Monday, November 11, 2002 10:08 PM}

To micklemj: Increasing the script timeout and execution timeout won't help, it's the aspnet_wp.exe process that thinks it's deadlocked after 180 seconds and "recycles" itself for you (how nice of it!). I have found a partial solution though. As long as ASP.NET is serving some requests, aspnet_wp won't timeout. I have used this solution for uploads that take up to an hour with success. The way to make it work is to spawn a new window from the page before starting the upload. The code in the spawned page just refreshes itself every few seconds until the upload is finished, then it closes itself. However, this does not fix my main problem, which is that the *entire* uploaded file is kept in server memory during the upload. My uploads can be hundreds of megabytes, so that is not feasible. That's why I was asking about anyone who had worked with asynchronous handlers... I thought maybe I could grab every 4000 bytes or so and dump it to a temp file, rather than wait for the entire request to get there. I know that there are 3rd party utilities for this, but that's not an option for me, I have to write the code and no client-side ActiveX controls are allowed (other than what comes with Internet Explorer) Hope this helps...

_{Tuesday, November 12, 2002 2:31 AM}

I briefly researched that problem and I found two possible solutions (other than using ActiveX, Java or .NET DLLs on the client): - open a second upload window (or use an IFRAME) that does the file POST. write a httphandler that handles the upload process on the server. in the while() loop for reading in the file from the stream, write the current number of received bytes into a static variable. use XMLHTTP or a simple page refresh every 5s or so in the first window to call back to the server. have the server read that static variable from the httphandler and return it to the client after the post-back. - use ADO on the client to read in the file and upload it via an IFRAME. I forgot how that exactly worked, but you should be able to find a link on google if you look for 'file upload ADO object' or something similar. hope this help!! Marc

_{Tuesday, November 12, 2002 2:46 PM}

To Marc Hoeppner: Yes, as I mentioned in my previous post, I am using the "second window" method to keep the upload alive (actually I use a modeless dialog box that contains aframeset that contains an aspx page that contains client script that refreshes every few seconds, but that's not my issue) My issue is that the post from the main page uses the control, so the *entire* file (or files) are sent in one big http request. While this works, the aspx page receiving this huge post (which can be hundreds of megabytes for my application) stores the *entire* request in server memory before I get a chance to begin handling it. Obviously, this will only marginally work with a single user, and if several users try to upload simultaneously, the server will run out of memory. What I need to do is intercept the request *AS IT COMES IN*, not after it is complete. That's why I was asking about asynchronous httprequest handlers. Do you (or anyone else reading this) have any experience with these? Can they be used to read an incoming request in chunks, so that the server doesn't have to buffer the *entire* posted file in memory? Any pointers greatly appreciated! J B. Podolak

_{Tuesday, November 12, 2002 4:00 PM}

>Do you (or anyone else reading this) have any experience with these? >Can they be used to read an incoming request in chunks, so that the >server doesn't have to buffer the *entire* posted file in memory? Without having done exactly that as of yet, from my experience with HttpHandlers the answer is yes. You should be able to get the BeginRequest event and take over from there. The standard ASPX handler certainly does something similar for the native ASP.NET file upload. Anyway, it certainly is possible with an ISAPI filter/module, so that should work with either HttpModule or HttpHandler. I took a quick look for you at my link list that I luckily still have from that research, so you may want to take a look at these links for an alternative ADO/XMLHTTP way of uploading files: http://www.15seconds.com/issue/010522.htm http://www.pstruh.cz/tips/detpg_uploadvbsie.htm If you control the environment totally, you may want to think about a .NET DLL that is loaded via the object tag right inside the browser. You then can do anything like you would in an ordinary Windows Form application. The client needs .NET framework 1.1 installed in his machine, so this may be a show-stopper at the current time. One last thought is to try something completly different: for a CMS-like application that we are currently writting, we have a similar problem with large files. Obviously the main problem with large files is the browser timeout that is hard to control. We are probably going to change the process totally, so that the user first gets fills out a form with all the meta data (name, description, author, keywords, categories, etc.) and then gets a ticket number from the system. He then mails the file to an email service on the same server. A windows service on the machine constantly checks for incoming email with a valid ticket number and then finishes the 'upload' process by adding both the meta data and the file to a database. The really nice thing about this is that the upload works asynchronously without the user having to wait in front of his browser or forcing him to deal with timeouts :) Hope this helps! Best regards, Marc

_{Sunday, November 17, 2002 9:43 PM}

sounds interesting, anyone willing to post some sample code?

_{Wednesday, December 4, 2002 2:03 PM}

Hi I've been trying to crack this nut for the last 4 months. This is what I've got so far: 1. It can easily be done, if you set up a TcpListener, and direct the upload to a port other than 80. Via the TcpListener, you get the raw request stream, and you call all the shots yourself. No reading into server memory. I got this solution up and running. Tradeoff is, that not all networks are open to other ports than 80. Not all clients will be able to upload. So I took it down again... :-( 2. Studying the disassembly from abcupload.net, they only tap into the BeginRequest event via a HttpModule when implementing the progress bar. In other words: It is possible to split the http request somehow, without using either a HttpModule or HttpHandler. This must be first priority. Turn this point around, and you get this: They DO need to use a httpmodule to tap into the upload process when they implement the progress bar. That must mean, that their upload class does not fully split the upload, even though they do enable saving of the request in chunks of 8 kb. Otherwise, they would not have bothered with the HttpModule. I only make this point in the hope that it will point us in the right direction. I think it means, that a bufferedreader must be introduces somewhere... Right now I am looking at the following two hopes: 1. The HttpContext.Request.Filter allows you to set a stream through which the request must pass. Maybe we can block the request in the filter...? Hmm...not counting on it though! 2. You can assign a new HttpApplication to a HttpContext, and set it to a class of your own. Remember, HttpApplication is the class "serving" HttpContext. I think this is the class that does the request reading. So if we could make our own implementation of this class, we could override the default reading method....somehow...hmm...sounds difficult! 3. Yes, async Http request handlers might also work. But don't they need to be registered in web.config? Based on the fact that abcupload.net does not need registration of handlers, I find it unlikely (Although they might create the async handler via a HttpHandlerFactory, thus not needing alterations in web.config.....hmm...is that possible?) This problem MUST be solved, and abcupload.net shows that it can! We need a site dedicated to solving this problem. Or maybe a newsgroup. A place where we can meet and crack this nut. Anyone got any suggestions? If anyone makes any progress concearning this problem, please email me at [email protected] Kind regards, Rasmus Denmark

_{Thursday, December 5, 2002 10:26 AM}

I can provide a website for development and testing. I could even be talked into hosting a source control system of some type if there was enough interest. Regards, Doug Wilson Fridge Productions

_{Thursday, December 5, 2002 11:24 AM}

To Rasmus, I looked into asynch httphandlers, but it would seem that these are for when your custom processing needs to do asynch work. ASP.NET just calls you once, then you call ASP.NET back when you are done with your processing. From what I have found in many, many web searches for docs, the first thing that happens when a request (posted file) comes in is that IIS mapping determines that it should go to the ASP.NET dll, and then IIS "forwards" the request to whatever BeginRequest handler is currently associated (built-in or your custom handler). But, if I understand this process correctly, by the time BeginRequest has been called, IIS has *already* received (and buffered) the *entire* request in memory, so even though a custom BeginRequest handler could read the request in, say, 8000 byte chunks and save it to a temp file, the stream that it is reading is *already* completely loaded into IIS memory, so it wouldn't help. However, as you point out, AspUpload.Net claims that their GigUpload technology keeps server memory load really low (from their docs, it looks like the "chunksize" is all that is kept in memory). So either they are only being *partially* truthful in their marketing (ASP.NET only uses, say, 8k, but IIS uses the full file size), or I am missing something. I know that the interception of an incoming request "as it comes in" can be done in an ISAPI extension or ISAPI filter, but everything that I have read states that by the time *any* ASP.NET code sees the request, IIS has already received the *entire* request. Any Microsoft folks want to clarify this for me? Given all that, I really don't know how AspUpload.Net does it without using a client side control or a real (unmanaged) custom ISAPI filter. There is one other way to make this work. If you can count on the browser being Internet Explorer, you can use an ADO objects (the XMLHttp and Stream objects) along with client-side javascript to post the upload in small chunks. The problem with this is that the security settings for Internet Explorer have to be relaxed to allow reading of local files, and I'm pretty sure that most people won't want to do this... J B. Podolak

_{Thursday, December 5, 2002 3:09 PM}

I too am working on this problem for an application and here's my thoughts so far: In the Application_BeginRequest event handler you do have control before the entire stream is buffered into memory. Just set a breakpoint in that method somewhere and you'll see that it fires immediately upon clicking your submit button. So that means that there is the ability to do some pre-processing on the request before the upload begins a-la ABC Upload. What the pre-processing is that we need to do is a mystery.... Here's what I have done: I've trapped the request for the upload in Application_BeginRequest and cached it in the Cache object. I then launch a popup window which reads the cached Request object to try and access the Request.TotalBytes property to get the length of the input stream. The problem is that the input stream is so busy doing the upload that it never responds to the line of code trying to get the TotalBytes property. I know this because if I hit the Stop button on the main browser window (the one doing the upload) my popup immediately displays the amount of bytes uploaded to that point. I feel so close, yet I think my approach may be too simplistic.... Jay

_{Thursday, December 5, 2002 4:27 PM}

To J B. Podolak and ITJ Sounds like async handlers are out of the question. Too bad - they looked promising :-). Regarding IIS buffering the request: My experience is, that it is possible to intercept the event before the buffering occurs - as ITJ points out. However, as soon as a reference to the request stream is made, the *entire* request is read into memory, as J.B. points out. I think we need to override the method supplying the request to HttpContext. I am not sure how to do this. However, one approach would be to implement your own hosting scenario. Look for System.Web.Hosting in the documentation, and it states, that you are able to host your own virtual address, thus CIRCUMVENTING IIS. Basically it requires you to implement the SimpleWorkerRequest interface, which apparently will be responsible for serving the HttpContext object. To me, this sounds like good news. However, I haven't been down that road yet, and are reluctant to go there, because it doesn't seem to be what abcupload.net is doing. However, after my exams (18/12), I'll probably try this. I can only support your request for a microsoft developer to look at this. In my opinion, we need an optimal solution to this problem, as the current upload method clearly has shortcomings or even a memory leak, as indicated by the 4. post in this thread: http://www.asp.net/Forums/ShowPost.aspx?tabindex=1&PostID=95255

_{Thursday, December 5, 2002 7:24 PM}

I've been doing some more digging in System.Web.Hosting assisted by the ms opensource Cassini web server documentation I really believe we could get somewhere by giving this a shot. It is the backdoor into the top of the http request pipeline that we need. When IIS receives a request for an aspx resource, it instantiates a HttpRuntime object, and calls ProcessRequest(HttpWorkerRequest hwr) The HttpRuntime object is TOP of the http pipeline, as shown here: http://msdn.microsoft.com/msdnmag/issues/02/09/HTTPPipelines/default.aspx The hwr argument passed to the HttpRuntime object is the toolbox that we want to alter. Inside HttpWorkerRequest are all the methods that HttpRuntime uses when it wants the request processed. HttpWorkerRequest is an abstract class, so we can implement our own version, and override the methods responsible for reading the request. There is even a class called SimpleWorkerRequest, which is already implementing HttpWorkerRequest. So in fact we only need to implement our own SimpleWorkerRequest. By creating our own hosting environment, we do the job of IIS: We pass an instance of our SimpleWorkerRequest-derived class to HttpRuntime.ProcessRequest, to make it the new toolbox that HttpRuntime must use, when it wants the request processed. How to setup a separate hosting environment? This articles explains how - and apparently, it seams quite possible: http://www.iunknown.com/Weblog/HostingASP.NETinaWinForms.html However, I find it slightly annoying that we have to go so far to theoretically have a chance of achieving the goal. I've found indications in MSDN about a possibility of changing the HttpWorkerRequest object, even without creating your own hosting environment, and I'll be looking at that next. Hmm...once again my exam preparation was cancelled in favour of .net. Terrible. Only 4 days left. I'll be seeing you guys on the other side of the exams.

_{Thursday, December 12, 2002 4:48 PM}

Please, could some asp.net team people comment on these ideas? This nut is not nearly cracked, and I really could use a pointer in the right direction. Kind regards, Rasmus

_{Friday, December 13, 2002 9:37 AM}

UPS, I'm working on this problem as well. Like you, I have too many demands on my time, but rest assured, I will post here as I make progress.

_{Friday, December 13, 2002 5:30 PM}

We'll get this together! :-) I am now more certain that I am onto the right track: First of all, don't bother setting up your own hosting environment just to get to HttpWorkerRequest. As I indicated in my last post, it really is quite possible to get the HttpWorkerRequest directly from the current request, though I'm still working on how :-). This is the plan: 1. Intercept request with a HttpModule, setup the BeginRequest event handler 2. In the BeginRequest event, get access to the HttpWorkerRequest object - somehow :-( 3. Call the appropriate methods of hwr object, such as ReadEntityBody(byte[] buffer, int size) in a while loop with an appropriate sized buffer, and store the request chunks somewhere (not in memory) between loops, untill full request has been read. 4. Extract just the file data from the saved request. 5. Somehow indicate to the rest of the processing pipeline, that the request has already been read, so no more calls to ReadEntityBody and methods alike are made. (yes, very abstract :-)). The reason why I am more sure now, is after a conversation with abcupload.net support, and some more testing. It seems that abcupload.net is NOT able to do its division of the upload without a httpmodule! I know I've indicated something to the contrary earlier - however, I've also speculated that something was wrong about them only using the httpmodule for progress indication. Generally, I believe I had misunderstood the functionality of abcupload.net...sorry about that :-) However, with help from the support team, it has now become clear, that abcupload.net DOES rely on a httpmodule for BOTH gigupload technology and progress indication! So that opens up for the intuitively approach stated above, and gives some assurance of this being a possible solution. Still quite a few puzzles are left unsolved. So once again: asp.net team people, please tell how I can get the HttpWorkerRequest from HttpApplication? - I know it must be possible somehow. This is really the most essential need. Regards, Rasmus

_{Monday, December 16, 2002 2:34 PM}

Rasmus (ups101): Here's how to get at the HttpWorkerRequest from within a BeginRequest handler (I found it in another thread): private void OnBeginRequest(object src, EventArgs e) { HttpApplication app = (HttpApplication)src; HttpWorkerRequest workerRequest = (HttpWorkerRequest)app.Context.GetType() .GetProperty("WorkerRequest", (BindingFlags)36) .GetValue(app.Context, null); } Also of interest, here's what I found on "ending a request" in an HttpModule: The HttpApplication object exposes the CompleteRequest method. If an HTTP module's event handler calls HttpApplication.CompleteRequest, normal pipeline processing is interrupted after the current event completes (including the processing of any other registered event handlers). A module that terminates the normal processing of a message is expected to generate an appropriate HTTP response message. Given the above, along with ReadEntityBody() maybe we can make this work correctly... Please post your results! J B. Podolak

_{Monday, December 16, 2002 5:17 PM}

J.B. Podolak Beautiful! I think you're right; this should be possible now. I'll post results in a few days...just one more exam to go wednesday. BTW: I believe (BindingFlags)36 can be replaced by BindingFlags.NonPublic | BindingFlags.Instance IL looks the same... Kind regards, Rasmus

_{Wednesday, December 18, 2002 11:55 PM}

Ok, I have some very preliminary results from experimenting with the ideas presented above. I build an HttpModule to process the upload in chunks successfully. I used a 64K buffer. It made a huge difference vesus using the .NET classes to pull in the file. No more loading the entire file's content in memory!!! But, that leaves more work for us.... If you have an incoming http request and you capture it using the BeginRequest event in the ASP.NET pipeline and you access the HttpRequest object (from the Context object) in any way, ASP.NET will load the entire request's content, regardless if it is a multipart request or not, into memory. I experimented with various ways before attempting to use the HttpWorkerRequest class. Bottom line, if you access even one property of the HttpRequest object, you're getting everything all at once. Ok, so using the methods presented above, I was able to easily snag the httpWorkerRequest interface and exploit it. Indeed you have to use the ReadEntityBody method, but not exactly in the way you might think. When a multipart HTTP request comes in, the first pice of the request is "preloaded." The ReadEntutyBody method is only good for the portion of the request that is NOT preloaded. Therefore every HTTP request that comes in has some or all of its content preloaded. So you use the GetPreloadedEntityBody method to snag the preloaded portion of the request. The you use the IsEntireEntityBodyIsPreloaded method to check if you have the entire request or not. If not, then you can use the ReadEntityBody method to finish reading the request. FYI - the preloaded portion of the request is always 49152 bytes on my system. So you chunk-in the rest of the request writing the chunk to disk and get th desired solution to the original problem presented in this thread. But.....there's always a BUT! We're now talking about a raw HTTP request. (As if you didn't see that coming...) So now we need a reliable method of parsing the request as it comes in such that we are able to detect that an HTTP frame contains a file and we are only writing that file's contents to disk. All in all, it shouldn't be too bad. I don't have time tonight to write a parser! I have real work to do, but this thread was very interesting! I wrote my code in VB.NET (sorry, C# dudes). Luckily, I code in the framework, not like a typical VB programmer. So I'll place the code in the next reply. CodeRage!

_{Thursday, December 19, 2002 12:07 AM}

Ok, here's the html file tha has the form:

UploadForm

File to Upload: 

Ok, now add the following to your Web.config file inside the element: This configures the timeout to 30 minutes, accepts up to 500MB files, and registers the HttpModule. My assembly name is HttpUploadApp as the namespace is the same thing. Next is the HttpModule class. This is a rough example that writes every HTTP request that has a content body to a file whose name is a GUID with a ".txt" extension saved in a directory named "C:\Upload\. This was just for test purposes!!! HttpModule class (VB.NET): Imports System Imports System.IO Imports System.Web Public Class UploadModule Implements IHttpModule Public Sub New() MyBase.New() End Sub Public Sub Dispose() Implements System.Web.IHttpModule.Dispose End Sub Public Sub Init(ByVal context As System.Web.HttpApplication) Implements System.Web.IHttpModule.Init AddHandler context.BeginRequest, AddressOf BeginRequest End Sub Private Sub BeginRequest(ByVal source As Object, ByVal e As EventArgs) Dim app As HttpApplication = CType(source, HttpApplication) Dim context As HttpContext = app.Context ' Get the HttpWorkerRequest Object!!! Dim hwr As HttpWorkerRequest = CType(context.GetType.GetProperty("WorkerRequest", Reflection.BindingFlags.Instance Or Reflection.BindingFlags.NonPublic).GetValue(context, Nothing), HttpWorkerRequest) If hwr.HasEntityBody Then Dim g As Guid = Guid.NewGuid() Dim fn As String = "C:\Upload\ + g.ToString + ".txt" Dim fs As New FileStream(fn, FileMode.CreateNew, FileAccess.Write, FileShare.Read) Try Const BUFFSIZE As Integer = 65536 Dim Buffer As Byte() Dim Received As Integer = 0 Dim TotalReceived As Integer = 0 Dim ContentLength As Integer = CType(hwr.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength), Integer) Buffer = hwr.GetPreloadedEntityBody() fs.Write(Buffer, 0, Buffer.Length) Received = Buffer.Length TotalReceived += Received If Not hwr.IsEntireEntityBodyIsPreloaded Then While (ContentLength - TotalReceived) >= Received Buffer = New Byte(BUFFSIZE) {} Received = hwr.ReadEntityBody(Buffer, BUFFSIZE) fs.Write(Buffer, 0, Received) TotalReceived += Received End While Received = hwr.ReadEntityBody(Buffer, (ContentLength - TotalReceived)) fs.Write(Buffer, 0, Received) TotalReceived += Received End If fs.Flush() Catch ex As Exception Finally fs.Close() End Try context.Response.Redirect("UploadForm.aspx") End If End Sub End Class Ok, that's it. What I didn't address here was keeping th aspnet_wp.exe process from recyling itself by opening another client-side browser winfow that loads an .aspx page with a client-side (JavaScript) timer refreshing itself every 10 seconds or so. Anyone can write that code it's so simple!!! CodeRage!

_{Thursday, December 19, 2002 4:56 AM}

YAHOO! This is just wonderful CodeRage! Your solution to redirect after upload is perfect! That solves the problem of signalling the rest of the pipeline. Yes, the http must be parsed. But as you point out, this is no big deal compared to what we've got not. WOW! Was that nice or what? We're now able to do REAL uploading, and besides from the abcupload.net team, this must be valuable information to a lot of people! For me, exams are now over, THIS NUT HAS BEEN CRACKED, and the weather is not too cold...this has really made my day, and many to come :-) Kind Regards, Rasmus

_{Friday, December 20, 2002 12:49 AM}

I did everything you said but every time I post I get the following error: Maximum request length exceeded. Exception Details: System.Web.HttpException: Maximum request length exceeded. Stack Trace: [HttpException (0x80004005): Maximum request length exceeded.] System.Web.HttpRequest.GetEntireRawContent() +891 System.Web.HttpRequest.FillInFormCollection() +129 System.Web.HttpRequest.get_Form() +50 System.Web.TraceContext.InitRequest() +703 System.Web.TraceContext.VerifyStart() +160 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData) +216 System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +479 I even modified the httpRuntime key in machine.config to have the value 524288000 and still get the same result.

_{Friday, December 20, 2002 4:27 AM}

Ok, boys I found your posts last week and started working on your problem. I hereby post you my C# code, since the code above is written in some sort of VB.net code :) I'm working on the rest of the story now to pipe everything from the request to the iis but after I cut out the file from the request. In order not to do any redirection anymore. I hope you find this usefull : using System; using System.Web; using System.IO; using System.Reflection; namespace My.PlayGround { ///

/// Summary description for UploadHandler. ///

public class UploadHandler : IHttpModule { public UploadHandler() { } public void Init(HttpApplication application) { // Register our event handler with with the application object application.BeginRequest += new EventHandler(this.BeginRequest); } public void Dispose() {} public void BeginRequest(object sender, EventArgs args) { try { // Create an instance of th application object HttpApplication application = (HttpApplication) sender; // Create an instance of the HTTP worker request HttpWorkerRequest request = (HttpWorkerRequest) application.Context.GetType().GetProperty("WorkerRequest", (BindingFlags)36).GetValue(application.Context, null); // Only trigger if the request is of type 'multipart/form-data' if(application.Context.Request.ContentType.IndexOf("multipart/form-data") > -1) { // Create a new unique identifier to identify each request string guid = Guid.NewGuid().ToString(); // Please alter pathname because the root of the C drive is not really the place to put your tempfiles string filename = "c:\requests\request_" + guid + ".txt"; // Check if a request body is sent by the browser if(request.HasEntityBody()) { // Get the content length of the request int content_length = Convert.ToInt32(request.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)); int content_received = 0; // This is a nice feature to redirect users if they upload a file // larger then 100Mb BEFORE it is being uploaded if(content_length > 102400000) { application.Context.Response.Redirect("http://www.av.com"); } // Create a file to store the stream FileStream newFile = new FileStream(filename, FileMode.Create); // Get the preloaded buffered data byte[] body = request.GetPreloadedEntityBody(); content_received += body.Length; // Write the preloaded data to new file newFile.Write(body, 0, body.Length); // Get all the other data to be written to file if available if(!request.IsEntireEntityBodyIsPreloaded()) { // Create an input buffer to store the incomming data byte[] a_buffer = new byte[16384]; int bytes_read = 16384; while((content_length - content_received) >= bytes_read) { bytes_read = request.ReadEntityBody(a_buffer, a_buffer.Length); content_received += bytes_read; newFile.Write(a_buffer, 0, bytes_read); } // Read the last part of the stream bytes_read = request.ReadEntityBody(a_buffer, (content_length - content_received)); newFile.Write(a_buffer, 0, bytes_read); content_received += bytes_read; } // Flush all data to the file newFile.Flush(); // Close the file newFile.Close(); // Redirect to the page to avoid the browser hanging string current_page = application.Context.Request.CurrentExecutionFilePath; current_page = current_page.Substring(current_page.LastIndexOf("/")+1) + "?" + application.Context.Request.QueryString; application.Context.Response.Redirect(current_page); } } } catch(System.Threading.ThreadAbortException) {} catch(Exception exception) { object e = exception; } } } }

_{Friday, January 3, 2003 12:16 PM}

Bjron/CodeRage, Wow... Its a great code... Thanks for your code. Here is the thing, once i upload a file(hello.doc) to web server. Its corrupted with .txt extension. Even after giving rite extension. Its getting corrupted with some extra junk something like this... 7d315142b02c8 Content-Dis-data; name="__VIEWSTATE" dDwtMTI3OTMzNDM4NDs7Ppg0hIHAIt5o+GJ8PugQQtOjF3TU 7d315142b02c8 Content-Dis-data; name="file"; filename="C:\download\ProgressBar_src.zip" Content-Type: application/x-zip-compressed HOW DO I CORRECT THIS PROBLEM? Please let me know. Thx, Shan.

_{Friday, January 3, 2003 12:30 PM}

Hello. I have implemented the C# code available in this thread and in fact i get the file uploaded. My problem is that i loose at least some of the events that should fire. The upload process is trigered when i click an image button present in a form... Did anyone has this problem. How can i solve this? Should the ....response.redirect be there in the end? Thank you. Telmo Sá ps: Could you also reply to [email protected] please?

_{Sunday, January 5, 2003 1:30 PM}

Shan: What you are looking at is viewstate, added by asp.net. You can minimize viewstate, but you cannot get completely rid of it, without removing the runat=server attribute from the

tag. This should solve your problem, if your form only contains the file control. If your form contains other controls than just the file control, you have to rewrite all controls to old-style html controls, and add the runat=server to each control. Example: becomes: These types of controls will not complain about not being inside a runat=server form. Kind regards, Rasmus

_{Sunday, January 5, 2003 1:40 PM}

Telmo: The reason for response.redirect in the end, is exactly to prevent you from running into problems like the one you descripe. I think the problems is this: Once the file is uploaded, you have altered the request. Somewhere asp.net keeps track of the expected length of the request. After upload, the request no longer has the expected length, as you have removed the file. This will lead asp.net to "hang", waiting for the missing bytes to be supplied by the client. I have not found a way to indicate that the file has been removed; that is, I have not found the place where the expected request length is kept. So currently you do need to redirect after upload. Note: Abcupload.net does somehow solve this problem. kind regards, Rasmus

_{Monday, January 6, 2003 9:18 AM}

ups101, Well. Thanks for your reply. By the way, whats the point if the file got corrupted after Uploading to web server; Say like Once i upload a Doc file(hello.doc), i'm not able Open that file... There should be some way to work around to get rid of those JUNK values(_ViewState .. blah blah). If anyone know how to solve this problem, let me know. Thx, Shan.

_{Monday, January 6, 2003 9:40 AM}

Shan: You have the raw http request. All you need to do is filter away the headers, then you're left with the uploaded file. Look some more at the code, and I'm sure you'll agree, that this is no problem at all. regards, Rasmus

_{Monday, January 6, 2003 3:41 PM}

Hi, Is anyone out there with the solution for 'filtering Http Headers'. Like following junk here 7d327122b0224 Content-Dis-data; name="__VIEWSTATE" dDwtMTI3OTMzNDM4NDs7Ppg0hIHAIt5o+GJ8PugQQtOjF3TU 7d327122b0224 Content-Dis-data; name="file"; filename="C:\Test.txt" Content-Type: text/plain Test here.... 7d327122b0224 Content-Dis-data; name="Submit1" Submit 7d327122b0224-- Waiting for your reply ya. Thanks, MaduraiKaran

_{Monday, January 6, 2003 3:50 PM}

I've only briefly looked over this article, but it looks like it could be applied to your application: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnservice/html/service11052002.asp

_{Monday, January 6, 2003 4:31 PM}

Cyberflote, Thats not gonna help. All i want is filterin out the header from uploaded file. U know that header corrupt the uploaded files. Thanks, MaduraiKaran.

_{Tuesday, January 7, 2003 10:43 AM}

Hello, How to filter following junk values using HTTP Filter?. 7d334818100336 Content-Dis-data; name="__VIEWSTATE" dDwtMTI3OTMzNDM4NDs7Ppg0hIHAIt5o+GJ8PugQQtOjF3TU 7d334818100336 Content-Dis-data; name="file"; filename="C:\Test\hello.txt" Content-Type: text/plain Hello there.... 7d334818100336 Content-Dis-data; name="Submit1" Submit 7d334818100336 Content-Dis-data; name="hdSession" aaa 7d334818100336-- Let me know. Thanks, MaduraiKaran.

_{Tuesday, January 7, 2003 6:09 PM}

check out http://www.ietf.org/rfc/rfc2388.txt and http://www.ietf.org/rfc/rfc1867.txt

_{Tuesday, January 7, 2003 6:44 PM}

What the code I posted does is retrieve the raw HTTP request. I didn't provide a 100% perfectly clean solution because of 2 reasons: I don't have time, and you have to do something! If you feel that you can just copy others' code and be done with it, you should purchase a 3rd party component instead. Ok, pay attention. The raw HTTP request must be parsed. It is extremely simple. Just location the request header that represents the field describing the file being uploaded. The content portion of the header contains the raw file data. Only that raw file data should actually be written to disk. It is a tivial task to parse HTTP headers. CodeRage!

_{Wednesday, January 8, 2003 5:10 AM}

I have to agree with CodeRage This culprit of this problem has been solved, and CodeRage has demonstrated the ideas from our discussions. There is a reason why this is categorised as an advanced discussion - what you're left with is trivial, and you should be able to find lots of examples on parsing http requests - if it isn't already obvious. regards, Rasmus

_{Thursday, January 9, 2003 11:55 AM}

CodeRage/Rasmus, I too agree with you guys'. I tried soo many ways to get just binary data from 'Raw data'; No luck. Is anyone outthere with any solution for this. That will be great... Let me know. With Rgds, MaduraiKaran...

_{Thursday, January 9, 2003 12:35 PM}

CodeRage/Rasmus, I too with you guys. I tried soo many ways to get just binary data from 'Raw data'; No luck. Is anyone outthere with any solution for this. That will be great... Let me know. With Rgds, MaduraiKaran...

_{Sunday, January 12, 2003 8:24 PM}

MaduriKaran, raw data is probably base64 encoded? You will need to decode the data before writing to disk. If this is incorrect, please correct me.

_{Monday, January 13, 2003 7:29 AM}

Here's the code I use to solve the need to redirect. This code populates the _contentAvailLength, _contentTotalLength, _preloadedContent, and _preloadedContentRead properties that .NET uses to track the expected length of the request. private byte[] PopulateRequestData(HttpWorkerRequest hwr, byte[] data) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = hwr.GetType().BaseType; int i = (int)data.Length; type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR, true); }

_{Monday, January 13, 2003 9:57 AM}

dreadman> Wonderful! I've been looking for stuff like that for ages! How did you find these properties? Kind regards, Rasmus

_{Tuesday, January 14, 2003 10:04 AM}

Is anyway to stop the Upload Process in between... ;)

_{Wednesday, January 15, 2003 10:00 AM}

Hello CodeRage/ups101/All, I'm tryin to stop the Upload process in between I tried with context.Response.buffer=true; context.Response.clear(); context.Response.end(), its not working. Can anyone help me on that. With Regards, Mdu

_{Monday, January 20, 2003 10:38 PM}

an interesting post that explains a little about the content boundries.. http://www.c-sharpcorner.com/Code/2002/Dec/DotNetBugs.asp :-)

_{Tuesday, January 21, 2003 3:40 AM}

Ok, I'm back again. I've been working on the upload handler the last few weeks and got it all figured out. To give you an idea of the complexity of the subject, I'll list the classes I used to get it all working. 1. The uploadHandler -> handles all incomming requests on IIS and triggers if it encounters Multipart-formdata stuff 2. The UploadDataProvider -> A class I wrote to get all the data posted during the request, it contains a method that passes the whole request byte per byte. Very usefull for parsing the request. 3. The UploadProgress -> A class to keep track of the status of the upload. An instance of this object is created at the beginning of the request and is used to store all the info about the upload in progress. 4. UploadedFiles -> Contains a ArrayList of all the files uploaded (ArrayList of UploadedFile) 5. UpoadedFile -> Contains all the information about the files that were uploaded. As you can see, it's not as easy as we think it is. If I get my code working in a stable way I'll let you guys know.

_{Thursday, January 23, 2003 5:12 PM}

Bjorn.B, Wow... We r all waitin for your great code here... Just 1 question: How do I stop execution in the middle of current HTTP module(upload)? Let me know your email id too. Thanks a lot. MDU

_{Tuesday, February 11, 2003 6:15 PM}

I have tried to implemented two custom versions of the algorthm stated in this thread for stripping the form content of forms into a text file. I have even ran a copy of the code that was posted on this thread. I am getting a problem where the response hangs on the server. GetPreloadedEntityBody() is fine - but ReadEntityBody(...) calls consistantly return 0 and if I manually press stop on my browser while testing the transfer goes through without a hitch. I have including several versions of flushing and redirection etc. and nothing seems to work. I am obviously missing something simple. I have the parsing and even progress indicator - but I still have to give it a kick after I click submit for things to start rolling. It boggles my mind how a product like .net would make the implementation of this so cryptic that it takes until late 2002 for the community to get a publicly workable solution. Oh BTW - thanks for the hard work! -Goyaman

_{Thursday, February 13, 2003 3:45 PM}

Goyaman> Are you sure you're not reading past the length of the request? If you call ReadEntityBody with a value larger than what is available, the server will hang. This is because the process is waiting for the rest of the bytes to read - or some solid confirmation that the bytes are not coming, like the client pressing stop. So the solution is to only read as much as you have available - and not less than that either, as this will also cause problems (my experience at least). I believe the proposed code from coderage retrieves the length of the request by parsing the Request-Length header. Regards, Rasmus

_{Friday, February 14, 2003 12:01 AM}

Funny thing when you forget your nose in spite of your face: 1. I was reading a bit too far on the request, so -- I turned on tracing and it all went pear shaped. ANYONE trying to do this note that you CAN'T turn TRACING ON! Well - I should go back to the noob threads... :) -Brett

_{Tuesday, March 11, 2003 9:57 PM}

You are near to figuring everything out. But not quite. What you must try to do is to intercept byte by byte the upcoming stream or blocks by blocks. And then you must look for the boundaries with the parameter "filename". If that's the case then you are receiving the body of a file. You don't just go saving the file on the disk because it will contain the header used by the multipart/form-data protocol. Commercial upload controls implement the RFC 1867 protocol. There are VBScript code that parse form data. Look for them and port them to .NET.

_{Wednesday, March 26, 2003 11:26 PM}

I tried the code but got the following error : Can you please help?? Configuration Error Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately. Parser Error Message: File or assembly name HttpUploadApp, or one of its dependencies, was not found. Source Error: Line 21: Line 22: Line 23: Line 24: Line 25: Source File: c:\inetpub\wwwroot\adb\web.config Line: 23 Assembly Load Trace: The following information can be helpful to determine why the assembly 'HttpUploadApp' could not be loaded. === Pre-bind state information === LOG: DisplayName = HttpUploadApp (Partial) LOG: Appbase = file:///c:/inetpub/wwwroot/adb LOG: Initial PrivatePath = bin Calling assembly : (Unknown). === LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind). LOG: Post-policy reference: HttpUploadApp LOG: Attempting download of new URL file:///C:/WINNT/Microsoft.NET/Framework/v1.0.3705/Temporary ASP.NET Files/adb/820ab95c/40870a47/HttpUploadApp.DLL. LOG: Attempting download of new URL file:///C:/WINNT/Microsoft.NET/Framework/v1.0.3705/Temporary ASP.NET Files/adb/820ab95c/40870a47/HttpUploadApp/HttpUploadApp.DLL. LOG: Attempting download of new URL file:///c:/inetpub/wwwroot/adb/bin/HttpUploadApp.DLL. LOG: Attempting download of new URL file:///c:/inetpub/wwwroot/adb/bin/HttpUploadApp/HttpUploadApp.DLL. LOG: Attempting download of new URL file:///C:/WINNT/Microsoft.NET/Framework/v1.0.3705/Temporary ASP.NET Files/adb/820ab95c/40870a47/HttpUploadApp.EXE. LOG: Attempting download of new URL file:///C:/WINNT/Microsoft.NET/Framework/v1.0.3705/Temporary ASP.NET Files/adb/820ab95c/40870a47/HttpUploadApp/HttpUploadApp.EXE. LOG: Attempting download of new URL file:///c:/inetpub/wwwroot/adb/bin/HttpUploadApp.EXE. LOG: Attempting download of new URL file:///c:/inetpub/wwwroot/adb/bin/HttpUploadApp/HttpUploadApp.EXE.

_{Thursday, March 27, 2003 4:15 AM}

I manage to make the code work. i no more get an error . BUT , i cannot see a progress indicator when i upload a larger file. Can anyone please help? thanks

_{Thursday, April 24, 2003 9:25 PM}

I'm stuck on this too. I started with the code originally posted by coderage but have heavily modified it to try and solve the problem of ReadEntityBody() hanging or returning 0 bytes. First I started (as demonstrated in coderages' post) with small buffer and tried to call ReadEntityBody() but it only ever returned 0 and no bytes into the buffer. When that didn't work, I tried to allocate a buffer to handle all the remaining bytes in one go, however doing that causes an exception from deep inside the framework. Even if this had worked, it seems kind of dumb to do it that way since the whole point was to not have to load the entire stream into memory for processing and thus consuming substantial server resources. Finally I decided to try reading one byte at a time. This also does not work with the call to ReadEntityBody() hanging or returning with 0 bytes read. Can someone at least point me in the right direction? Is there at least a way to know how many bytes are available to be returned by ReadEntityBody() allowing me to read them and then sleep until more arrive?

_{Monday, April 28, 2003 5:49 PM}

Hi all: Man, I've followed this post from the earlier messages, and let me say, you all have done one hell of a job! I was just curious is anyone has made any progress that they'd be willing to share... Thanks!!!

_{Tuesday, April 29, 2003 8:13 AM}

Hi Rob I have a hope that your compliment is partly aimed at my post, so thanks :-) I think a few of us have complete solutions by now - and I also regret, that I cannot share mine. My code has been sold to a company, which is not to pleased with the code-sharing principle :-). However, I really do believe that the earlier posts, completing with coderage's example code, is really all that is needed. The rest - parsing a request and making a javascript progress bar - is trivial, and the web has many examples of such practices. I think your post is a pleasant return to the level that this discussion should have: Not a tutorial, but a deep discussion of how to go about this problem. The essential code (really just how to get the HttpWorkerRequest and work the preloaded data) is in place. Now we can discuss how to structure the solution. The following is my suggestion: I can recommend dividing the solution into a low-level scanner and a high-level parser. The scanner is responsible for filling a byte array, thereby encapsulating the problems related to reading the contents of the request, using the HttpWorkerRequest. That is, the role of the scanner is primarily to solve the following 2 problems: 1. Some data is located in the preloaded buffer 2. Reading past the end of the request resolves in a deadlock-like situation. -And to supply the parser with a function that fills a byte array, returning also the number of bytes read. This is exactly what you would expect from any stream reader. This greatly reduces the problems for the parser, now only responsible for dividing the request into appropriate pieces: -Text fields, posted along with the files -Files Going about this requires the parser to be able to identify the boundary that divides the request into these pieces. This boundary is also the FIRST LINE in the request, so the first step in dividing the request should be to read this line. So each piece of the request starts and ends with this boundary. Now the nature of these pieces (either text field or file) can be identified by parsing the line immediately following the boundary. If this line contains something like "filename=XXX", the piece following is a file - otherwise, if it contains "name=XXX", the piece following is a text field. Now that we know the nature of the pieces, we are able to parse them appropriately. For example, I store the posted text fields in a hash table, and save the files - in chunks - to a predetermined place on the hard drive (supplied by a parameter). This step is pretty simple - we are just reading until we meet the boundary again. So the parser algorithm has the following form: 1. Identify boundary line 2. Identify type of piece (text field or file) 3. Read piece until next boundary is met 4. Go to 2 That was the the basic building blocks of the solution - a scanner and a parser. What is left, is adding a class having a lot of "statistics" variables - like bytes read, length of request, and so on. This class should be accessible not only to the scanner, but also to some other process, responsible for displaying upload progress information to the user, likely in form of a progress bar. This part is really trivial - just remember that you cannot use session variables when dealing with the httpmodule. Instead I've experiemented with a static hashtable, containing instantiations of these "statistics"-classes, each related to an upload-id (just a really long random number). This upload id should be supplied to both the scanner and the process responsible for displaying the statistics. So they can both look up and write to / read from the same object in the static hash table. Wiring it all together should be a httpmodule, which implements the scanner (and passes it the httpworkerrequest), the parser, and the so-called settings class, also accessible to the scanner and from the static hash table. So that was my solution, in steps. No code - but I hope others will comment on the proposed solution. Kind regards, Rasmus

_{Tuesday, April 29, 2003 10:24 AM}

Thanks for that Rasmus. I too have often been in the situation where the code I've written (good or bad) cannot be shared and I appreciate the overview you have provided. In fact, I think you've outlined a very effective solution. I would dearly love to see your implementation, but understand completely your reasons for not posting it. However, I'm still stuck at the very beginning. That is, reading the full request from the incoming stream, which is the problem that had to initially be solved for this approach to work. Everything I have tried, mostly based on posts in this thread, has not orked. Reading the bytes that are preloaded is easy and I have that working successfully. Its when I try to read the remainder of the incoming bytes I encounter either the deadlock that has been described in previous posts or the call simply returns with my buffer empty and 0 bytes read. I just don't see how I can be reading past the end of the buffer if I'm reading one byte at a time. I would really appreciate some help or at least a push in the right direction. I know that once I have the ability to read all the bytes in the stream, the rest is more or less trivial. Thanks for everything that has been contributed so far! Cheers, Doug

_{Tuesday, April 29, 2003 11:30 AM}

Hi Doug Sounds mighty frustrating. Could you post your code please (or maybe just fragments)? I suppose you redirect to another page after attempting to read, right? Also, what does the ContentLength variable give you? (request.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)); (And a stupid question, but I have to ask : You do use a httpmodule, right? :-)) Regards, Rasmus

_{Tuesday, April 29, 2003 8:33 PM}

Thanks Rasmus and everyone else, Here is my code. I've stripped out all my logging to save some space here. public class UploadModule : IHttpModule { private const Int32 BUFF_SIZE = 65536; private const Int32 MAX_LENGTH = 1024 /* bytes */ * 1024 /* kilobytes */ * 5 /* megabytes */; void IHttpModule.Dispose() { /* Empty */ } void IHttpModule.Init(System.Web.HttpApplication context) { context.BeginRequest += new EventHandler(BeginRequest); } private void BeginRequest(object source, EventArgs e) { String ticks = DateTime.Now.Ticks.ToString(); String logfilename = @"D:\FPI\_" + ticks + ".log"; String datafilename = @"D:\FPI\_" + ticks + ".txt"; HttpApplication app = (HttpApplication)source; HttpContext context = app.Context; // Get the HttpWorkerRequest Object!!! HttpWorkerRequest request = context.GetType().GetProperty("WorkerRequest", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(context, null) as HttpWorkerRequest; if (request.HasEntityBody()) { Int32 contentLength = Convert.ToInt32(request.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)); Int32 bytesReceived = 0; Int32 bytesRemaining = contentLength; Int32 bytesRead = 0; Int32 loopcount = 0; // Create a file to store the stream using (FileStream dataFile = new FileStream(datafilename, FileMode.Create)) { // Get the preloaded buffered data byte[] body = request.GetPreloadedEntityBody(); dataFile.Write(body, 0, body.Length); bytesRead = body.Length; bytesReceived += bytesRead; bytesRemaining = contentLength - bytesReceived; if ( request.IsEntireEntityBodyIsPreloaded() == false) { Byte[] buffer = new Byte[BUFF_SIZE]; while (contentLength - bytesReceived >= bytesRead && loopcount++ < 100) { bytesRead = request.ReadEntityBody(buffer, BUFF_SIZE); dataFile.Write(buffer, 0, bytesRead); bytesReceived += bytesRead; bytesRemaining = contentLength - bytesReceived; } bytesRemaining = contentLength - bytesReceived; bytesRead = request.ReadEntityBody(buffer, bytesRemaining); dataFile.Write(buffer, 0, bytesRead); bytesReceived += bytesRead; } dataFile.Flush(); dataFile.Close(); } // using (FileStream dataFile } // if (request.HasEntityBody()) } }

_{Tuesday, April 29, 2003 11:16 PM}

Amen! Finally, someone willing to step up to the plate and not be concerned with putting cash in their pocket! Thanks CyberFloatie!!! Now, for those of us who are of lesser .NET talent, do we put this in it's own application? The reason I ask, is that I used the previous C# code provdied before, and used the web.config modifications in the project that I wanted to use the upload module in. Then, each form that was submitted gave me an error about the request length being exceeded (in fact, I believe CyberFloatie actually posted the debug log for the exact error.) How would we go about implementing this awesome solution? THANKS!!!!

_{Wednesday, April 30, 2003 10:09 AM}

robrichard, My code doesn't work! The reason I posted it here was because I needed help with it, however, I do think I figured it out myself last night. The relevant changes are these: Byte[] buffer = new Byte[BUFF_SIZE]; while (bytesRemaining > BUFF_SIZE) { bytesRead = request.ReadEntityBody(buffer, BUFF_SIZE); dataFile.Write(buffer, 0, bytesRead); bytesReceived += bytesRead; bytesRemaining = contentLength - bytesReceived; } buffer = new Byte[bytesRemaining]; bytesRead = request.ReadEntityBody(buffer, bytesRemaining); dataFile.Write(buffer, 0, bytesRead); bytesReceived += bytesRead; You'll also not that this is nowhere near a complete solution. As mentioned in previous posts, you still need to parse the incoming stream to remove the uploaded files (and potentially decode them), however, you must also pass the unused portion of the stream back out so that the rest of the process has something to do. In other words, you can't just filter out the file and let the request end or your browser will sit there forever with that dumb look on it's face. Some people have solved this by doing a redirect when their done and ignoring the rest of the stream. Others have managed to replace the incoming stream with their own so that it can still be processed by the .ASPX that it was originally posted to. Me, well, I'm just happy I got this far. I'll figure out what I'm doing the next time I get motivated. This was supposed to be a fun project but it turned out to be a lot harder than I initially anticipated. I'll keep slogging through it and posting here.

_{Thursday, May 1, 2003 6:06 PM}

Hi cyberfloatie Your latest changes look really nice. I would stick to just the one counter, bytesRead, so the call would be something like: while(contentLength - bytesRead > BUFF_SIZE) { bytesRead += request.ReadEntityBody(buffer, BUFF_SIZE); ... ... } buffer = new Byte[contentLength - bytesRead]; ... ... ... (nothing I have tested...just an idea). Either way should work. So, it sounds like you've got it working now? That sounds great!. Gotta get to bed, early up in the morning. Regards, Rasmus >rob "Amen! Finally, someone willing to step up to the plate and not be concerned with putting cash in their pocket!". Yes, I need the money. Not exactly a rich life being a student. Also, if I've worked damn hard for this, I don't see any problem with trying to earn a buck on it. Do you?

_{Thursday, May 8, 2003 12:50 AM}

Thanks for the code cyberfloatie, your code works great. Does any one know to get the data from the uploadModule to say an asp.net page? So far the only thing i can think of, is to just write out an html file. Also, does anyone know how to parse the mime type and put the right extention on the file? All im doing right now is using the exention from the file that was posted.

_{Tuesday, May 13, 2003 8:45 AM}

Hi Code Rage, I have a problem when trying to make your code work. The GUID.txt file in the Upload folder contains the name of the file (as a file=filename) and not the the file itself. Is there any additional configuration required? TIA, Marek

_{Tuesday, May 13, 2003 11:26 AM}

XOnic, The post in this thread by dredman on 13 Jan 2003 might help you. Without having tried to do this myself (I will eventually) it looks like you can write back to the stream anything you want allowing any remaining modules and eventually the .aspx to process it. When I do finally get around to it, the approach I'm thinking about will be to write the files to the disk into a temp location and then rewrite the posted data to include only the non-file fields and some addidtional information that another component can parse to gain access to the uploaded files in the temp location. Has anyone else tried this approach?

_{Tuesday, May 13, 2003 11:29 AM}

MarekK, Please read this whole thread. You will find that while coderage's code is great, it is not a complete solution. However, there have been enough tidbits posted in this thread that you should be able to develop your own solution from them. As for a "compile and go" solution, well, you won't find it here.

_{Friday, May 16, 2003 4:51 AM}

Cyberfloatie, No, I've stuck to the simple response.redirect. Instead I used a delegate that was called after the upload completed, allowing for post-processing of the non-file fields. The way I read dredmans post, it seems possible to correct the length of the request, so the worker process wont stall, waiting for bytes we've taken out. However, it doesn't seem he's saying that you can write back to the stream itself? Or did you read that somewhere else? However, your proposed solution seems "cleaner" than mine...even if you have to introduce a new structure for both the fields and the files. Let us know what you find out :-) kind regards, Rasmus

_{Friday, May 16, 2003 11:05 AM}

Rasmus, I'm basing this entirely on dredman's post repeated here for convenience: private byte[] PopulateRequestData(HttpWorkerRequest hwr, byte[] data) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = hwr.GetType().BaseType; int i = (int)data.Length; type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR, true); } It stands to reason that _preloadedContent is the byte array that GetPreloadedEntityBody() is the accessor for. You can probably figure it out for sure by checking the contents of the variable before the call GetPreloadedEntityBody(). If it turns out that it does contain the preloaded content, there shouldn't be any reason that it can't contain content after the module is done parsing the full stream. Now in dredmans PopulateRequestData() he passes in the worker request and a byte array. I'm assuming that the variable inData is a typo and should actually be the byte array parameter. If I'm correct, then the contents of the byte array can be the original incoming stream minus the posted file(s). You can then do as I suggested in a previous post and add a couple of your own fields to the stream, adjust the _contentAvailLength, _contentTotalLength and _preloadedContentRead. Then when the code in the target of the form post is executed, you can instantiate another object that knows how to read those variables and access the uploaded files. So, why am I wasting time posting all of this speculation rather than trying it myself? I've just bought a house and have to paint it, clean it and then move within the next two weeks and won't be touching any projects until long after thats done. I'm hoping someone can try what I've suggested here and post their findings (and maybe even some code) so that when I do get back on this I won't have to pick up from where I left off. dw

_{Friday, May 16, 2003 5:14 PM}

Cyberflotie> Ahh....nice! I was a bit too fast in judging dredmans code, I can see that now. Thanks for the elaboration. I think you are right about the typo...it's gotta be the array parameter. Congrats on the house! Whereat in the world, may I ask? I am starting to post again, because it is exam period again...we've actually been posting in this same thread for 2 exam periods now :-). As always, everything else is alot more interesting than studying for the exams. Anyway, I'll also give your solution a try sometime next month. BTW, any thoughts as to how we can "most cleanly" access the statistics variables (bytes read, file length, etc.)? The problem is, that we need access to these things two places....the place where we write to them, and the place where we read them, and show them to the user during upload. Another solution that could really use a clean-up, is the progress bar. The standard html progress bar updates itself every 3 second or so, and works just like any asp page. So you've got one frame doing an upload, and another continously updating...ugly. We've gotta be able to make a cleaner version of this. The culprit of the problem is how to stream the statistics nicely to the user. Here are some starting ideas: 1. Load via XML, as described here : http://webfx.eae.net/dhtml/xmlextras/xmlextras.html - But does that work in netscape...in older browsers? 2. Load via XML through flash...which is possible, but requires an open port...and everything should really run on port 80. And not everyone has gotten a flash player new enough to support xml loading. We should keep it in html, if at all possible. 3. Trick the browser into thinking it is downloading an image. You can determine if an image is downloaded or not in javascript...many preload progress bars exist already. Now the server just needs to dish out these empty images when say 5% is complete, 10% is complete, and so on. The height of the image could be the kilobytes read, the width could be total length...or something like that. Problem is, that we have to settle for pretty low resolution (eg. 5% intervals) - or have a lot of pictures. 4. Java or ActiveX solutions could easily do it...but too much hassle with Java code signing, too much IE-specific stuff with ActiveX. 5. Somehow send chunks of a page to the user at a time...possible downloading a javascript in pieces...but how? This one should be possible...but I cannot get it working. I could really use a smoothly increasing progress bar, without the continous refreshing in the background....hope you guys have some thoughts about this? kind regards, Rasmus

_{Friday, May 16, 2003 8:45 PM}

Rasmus said: Congrats on the house! Whereat in the world, may I ask? Kelowna, BC. Great spring, summer and fall. Lousy winters unless you're a skier, which I'm not. any thoughts as to how we can "most cleanly" access the statistics variables (bytes read, file length, etc.)? My first thought is that there should be some way to store them in the user's session. Then it would simply be a matter of sending them back to the thermometer, whatever form that takes. Actually, it might be better to store them at the application level, then you could build a global upload montoring page! could really use a smoothly increasing progress bar, without the continous refreshing in the background. Well, I can't think of any way to get away from the continuous refreshing, but you could certainly reduce the appearance of it by using your #2 suggestion. A couple years ago I coded the server side for a Penguin Chat that was more or less real time. We worked only with Flash 4 and basically form POSTs, no XML Sockets. It worked very smoothly and with a half decent Flash artist who can handle the action script, it should be no problem. As far as I know, all major browsers ship with at least Flash 4. Doug

_{Saturday, May 17, 2003 6:31 AM}

Cyberfloatie wrote: My first thought is that there should be some way to store them in the user's session My first choice too. But remember, at the time we are processing the files, the session hasn't been setup yet. We're working too high up in the http pipeline to use session data. Application level is - as you say - possible. However, not very nice in my opinion. Here is an ambitious plan on how we could get to the session data: 1. In the httpModule, we only read the "dredman vars" and then set them to 0. Then we let the page load as normal...it should ignore the posted data(?). 2. In the normal Page_OnLoad event, we reset the dredman vars to their original values, grab the HttpWorkerRequest, and do the upload processing. That way we get the session data, and move the whole processing to a "standard" level in the pipeline - where all code is normally executed. How impossible is that plan? We worked only with Flash 4 and basically form POSTs, no XML Sockets Ah, posting in Flash...thats another possibility. Also, one could make a function that approximates the bytes uploaded, given only a few time-value pairs. That greatly reduces the need for postbacks...and makes solution 3 a real possibility. Sounds like new skies will be a hit for christmas :-)

_{Sunday, May 25, 2003 4:31 PM}

Can anyone give us clueless a few tips on how to parse the stream? I know i can take the byte[] and create a memory stream from that, and pass that stream to a streamreader. Right now i just read every line and check it for the content boundary, im sure there is a faster and more efficent way to do this. Thanks for your help.

_{Friday, May 30, 2003 8:09 AM}

Cyberfloatie/ups101, Have you (or anyone else) been able to prevent uploading of files larger than a certain size? Before I buffer the incoming entity body, I do a check on the "content_length" similar to the code that Bjorn.B provided on the 20th of December. If the content_length is greater than a certain size, the idea is to stop to remainder of the incoming request and redirect the user off to a page that informs them that the file they are uploading is too big. Whenever I attempt to do the redirection to another page or URL (using HttpApplication.Context.Response.Redirect(...), HttpApplication.Response.Redirect(...) or even something like ....Response.End()) I receive a DNS error in the browser. Sure enough, the file upload is ceased, however I would like to be able to alert the user to the fact they are attempting to upload a file that is too big. I have thought about HttpHandlers, however I would tend to think that by the time the Handler processes the request, the request has already been fully uploaded. I have a feeling that because the HTTPModule is attempting to redirect the request while the request is still being sent by the user, that it gets confused and "spits the dummy". Has anybody had a similar problem to this, or have any suggestions as to what direction I should be heading? The reason I need to do this is to prevent unnecessary data being transferred to our server (and hence any related download charges). Thanks, Paul.

_{Wednesday, June 4, 2003 4:39 PM}

Hi Paul, regarding redirect I have a feeling that because the HTTPModule is attempting to redirect the request while the request is still being sent by the user, that it gets confused and "spits the dummy". I think you're right. Do you correct "the dredman vars" before leaving the request? (see earlier posts for explanation). I have thought about HttpHandlers, however I would tend to think that by the time the Handler processes the request, the request has already been fully uploaded I remember experiementing with this, a long time ago. I am certain that the upload first will take place when you touch on any part of the request object, including checking the content length. So I sugest you try something like this: 1. Post your upload to a "standard" http handler (ie. one not using session state - this is default, as I recall). 2. Get the content length in a httpmodule. If size > max allowed, either leave immediately, or first reset the dredman vars (not sure if necessary). Let the request proceed to the http handler, maybe after appending the request size as a request string to the request url (see rewriting request urls elsewhere in this forum). 3. In the http handler, check the request url for the inserted request string. If existing, alert the user appropriately. Otherwise - well, whatever else. The idea being, that you can avoid standard upload processing by posting to a http handler - and you can grab the request size in a http module, without triggering the upload process. X0nic, regarding byte search Right now i just read every line and check it for the content boundary, im sure there is a faster and more efficent way to do this I am using a stupid way myself - converting to a string, then searching that string, in the hope that asp.net has a native string pattern matching algorithm that outperforms whatever I implement anyway. Clearly this is not an advisable solution - but I'm lazy :-). However, it is actually a classical problem, since it is so hard to do right - so you should be able to pick up an algorithm somewhere on the internet. Search for the following (top 2 are most recommended: Boyer-Moore Matcher (linear running time, recommended for larger searches - complex) Knuth-Morris-Pratt pattern matching (linear running time, fast on smaller searches - simple) Rabin-Karp string matching (normally linear running time, performs well on average - simple) Pseudo-code for each algorithm can be found in "Introduction to Algorithms" by Cormen, Leiserson and Rivest. Kind regards, Rasmus

_{Tuesday, June 10, 2003 4:23 PM}

It has been a while since I have checked this thread. However, it appears some of you are still wrestling with the code I originally posted. As some of you noticed, there were some mistyped variables in that code. Here is another snippet of code for those of you still working on this. I'm sorry but I cannot post all my code however, this code snippet should be a huge help for most of you. using System; using System.Web; using System.IO; using System.Reflection; using System.Text; using System.Threading; using System.Collections; namespace NETXUpload { public class ProgressModule : IHttpModule { public string ModuleName { get { return "ProgressModule"; } } public void Dispose() { } public void Init(HttpApplication context) { context.BeginRequest += new EventHandler(this.OnBeginRequest); } private void OnBeginRequest(object src, EventArgs e) { HttpApplication httpApplication = (HttpApplication)src; GetEntireRawContent(httpApplication); } private void GetEntireRawContent(HttpApplication httpApplication) { ArrayList alData; int nCount; int nBytesReadTotal; byte[] arrTemp; int nTempSize; int nBytesRead; int nBufferSize; //Get ContentType string sContentType = httpApplication.Request.ContentType.ToLower(); //Check to see if this is a multipart upload if ( sContentType.IndexOf("multipart/form-data") == -1 ) return; //Get the multipart boundary byte[] arrBoundary = GetMultipartBoundary(sContentType); //Get the HttpWorkerRequest HttpWorkerRequest _wr = GetHttpWorkerRequest(httpApplication); byte[] _rawContent = null; //Create the progress monitor ProgressData progress; //Get the upload id string UploadID = httpApplication.Request.QueryString["UploadID"]; //Create and load the progress object for this upload progress = ProgressModule.GetProgressObject(UploadID); //Getting Size int ContentLength = httpApplication.Request.ContentLength; //Checking Size int nMaxRequestLength = GetMaxRequestLength(httpApplication); if (ContentLength > nMaxRequestLength) { return; } //Preloading progress.UpdateProgressData(0,ContentLength); byte[] arrBuffer = _wr.GetPreloadedEntityBody(); if (arrBuffer == null) arrBuffer = new Byte[0]; DataReader byteReader = new DataReader(arrBoundary ,httpApplication.Request.ContentEncoding, ContentLength, progress); byteReader.ReadBuffer(arrBuffer, arrBuffer.Length); nBufferSize = (int) arrBuffer.Length; progress.UpdateProgressData(nBufferSize,ContentLength); if (!(_wr.IsEntireEntityBodyIsPreloaded())) { if (ContentLength <= 0) return; int nBytesRemaining = ContentLength - (int) arrBuffer.Length; alData = new ArrayList(); nCount = 0; nBytesReadTotal = arrBuffer.Length; //Transferring while (nBytesRemaining > 0) { arrTemp = new Byte[nBufferSize]; nTempSize = (int) arrTemp.Length; if (nTempSize > nBytesRemaining) nTempSize = nBytesRemaining; nBytesRead =_wr.ReadEntityBody(arrTemp, nTempSize); if ( nBytesRead < nTempSize ) { //Upload stopped unexpectantly return; } if (nBytesRead > 0) { nBytesRemaining = nBytesRemaining - nBytesRead; alData.Add(nBytesRead); alData.Add(arrTemp); nCount++; nBytesReadTotal += nBytesRead; progress.UpdateProgressData(nBytesReadTotal, ContentLength); byteReader.ReadBuffer(arrTemp, nBytesRead); } } progress.Note = Configuration.GetConfig.GetNotesConfig.Finishing; } _rawContent = byteReader.InputStream; //_rawContent = arrBuffer; PopulateRequestData(_wr, _rawContent); //PopulateRequestData(_wr, byteReader.InputStream); } private static HttpWorkerRequest GetHttpWorkerRequest(HttpApplication inApp) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; return (HttpWorkerRequest)inApp.Context.GetType().GetProperty("WorkerRequest", bindingFlags).GetValue(inApp.Context, null); } private static int GetMaxRequestLength(HttpApplication inApp) { object local = inApp.Context.GetConfig("system.web/httpRuntime"); Type type = local.GetType(); BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; return (int)type.GetProperty("MaxRequestLength", bindingFlags).GetValue(local, null); } private static byte[] PopulateRequestData(HttpWorkerRequest inWR, byte[] inData) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = inWR.GetType().BaseType; int i = (int)inData.Length; type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR, true); return inData; } public static ProgressData GetProgressObject(string inID) { int i = 0; try { i = Convert.ToInt32(inID); } catch { i = ProgressMonitor.GetNewID(); } return ProgressMonitor.AddProgress(i); } private byte[] GetMultipartBoundary(string ContentType) { int nPos = ContentType.IndexOf("boundary="); string sTemp = ContentType.Substring(nPos+9); string sBoundary = sTemp = "--" + sTemp; return Encoding.ASCII.GetBytes(sTemp.ToCharArray()); } } }

_{Tuesday, June 10, 2003 4:40 PM}

I will be getting just a tad off the track of the original post, but it is an issue that need to address in a current project that the original post brings to mind: I would presume that downloads are pretty much the same, in that they load the file into memory as it is downloaded from the website. Is there any way to chunk and stream a download in a similar manner as to minimize memory usage of the webserver?

_{Tuesday, June 10, 2003 10:37 PM}

Hi Richard, I also find it reasonable to address this issue here. The solution is to use Response.IsClientConnected. This call will return true as long as the client is connected - and also stall the process until the current chunk has been tranferred to the client, thus minimizing mem usage to a buffer size array. However, I've found that this call in fact cannot detect if a user cancels a download in progress, before the users CLOSES the browser (not just navigates away). Therefore, one should be careful not to leave anything open between calls of IsClientConnected - as the answer might be a long time underway. This is part of what I'm using for downloads (implement in HttpHandler or HttpModule): string FullFileName = folder.FullPath + "\\" + file.Name; if (!System.IO.File.Exists(FullFileName)) {Entry ("Alert : " + FullFileName + " does not exist!");return;} ctx.Response.ContentType = "application/octet-stream"; ctx.Response.AddHeader("Content-Disposition", "attachment;filename=" + file.Name); ctx.Response.AddHeader("Content-Length", new FileInfo(FullFileName).Length.ToString()); long CurPos = 0; byte[] Buffer = new byte[524288]; int BytesRead = 0; FileStream FS = System.IO.File.Open(FullFileName,FileMode.Open,FileAccess.Read,FileShare.Read); FS.Position = CurPos; BytesRead = FS.Read(Buffer,0,Buffer.Length); CurPos = FS.Position; FS.Close(); //updating algorithm 18-05-2003 while (BytesRead > 0 && ctx.Response.IsClientConnected) { ctx.Response.OutputStream.Write(Buffer,0,BytesRead); ctx.Response.Flush(); FS = System.IO.File.OpenRead(FullFileName); FS.Position = CurPos; BytesRead = FS.Read(Buffer,0,Buffer.Length); CurPos = FS.Position; FS.Close(); }BR, Rasmus

_{Wednesday, June 25, 2003 7:59 AM}

Hi All, I want to thank you all for this great thread, a really hard work that help me to build my upload module, many of us got a headache when try to parse the file I will post mi parser, you will need to modify the code a little bit, This is my file header 7d240305e4 Content-Dis-data; name="__VIEWSTATE" dDwxNzc3NzMyODIxOzs+zvmqaKYoxiiqDx5fTsr37EOyJXE= 7d240305e4 Content-Dis-data; name="UploadPath" D:\temp\Uploads\ 7d240305e4 Content-Dis-data; name="UserID" Mariano 7d240305e4 Content-Dis-data; name="file"; filename="C:\archivo.asx" Content-Type: text/html < (first body bytes) you can see 4 fields, file, UploadPath, UserID (i need it), and one generated by the asp.net web form, the UploadPath and UserID are generated by hidden fields on the form, so the header maybe will change on your upload code. The parser work seeking line breaks (byte value of '13') a total of 16 in my header, the same structure always if I dont modify the upload form, the first block of bytes you receive got all you need to parse all the stream, the code is a class with a kind properties for that, thats not all my friend, you need to write the code to parse the file, remember that if you parse after the file is uploaded you will need to rewrite all the file again, imagine that with 40Mb file, so find a way to do that on stream fly. Here you got, Mariano Public Class HeadBodyParser Private _HeadBytes As Byte() Private _BodyBytes As Byte() Private _BoundaryBytes As Byte() Private _BoundaryTotalBytes As Integer Private _UploadPath As String Private _NombreArchivo As String Private _UserID As String Public ReadOnly Property UploadPath() As String Get Return _UploadPath End Get End Property Public ReadOnly Property UserID() As String Get Return _UserID End Get End Property Public ReadOnly Property NombreArchivo() As String Get Return _NombreArchivo End Get End Property Public ReadOnly Property HeadBytes() As Byte() Get Return _HeadBytes End Get End Property Public ReadOnly Property BodyBytes() As Byte() Get Return _BodyBytes End Get End Property Public ReadOnly Property BoundaryTotalBytes() As Integer Get Return _BoundaryTotalBytes End Get End Property Public ReadOnly Property BoundaryBytes() As Byte() Get Return _BoundaryBytes End Get End Property Public Sub New(ByVal Bytes As Byte()) Dim UploadPathInicio As Integer Dim UploadPathFin As Integer Dim UserIDInicio As Integer Dim UserIDFin As Integer Dim ArchivoInicio As Integer Dim ArchivoFin As Integer Dim x As Integer Dim Acumulador As Byte = 0 For x = 0 To Bytes.Length - 1 If Bytes.GetValue(x) = 13 Then Acumulador = Acumulador + 1 If Acumulador = 1 Then _BoundaryTotalBytes = x + 6 Dim BoundaryBytes As Byte() = New Byte(_BoundaryTotalBytes) {} Dim z As Integer For z = 0 To _BoundaryTotalBytes - 1 BoundaryBytes(z) = Bytes(z) Next End If If Acumulador = 7 Then UploadPathInicio = x + 2 End If If Acumulador = 8 Then UploadPathFin = x - 1 Dim UploadPath As String Dim u As Integer For u = 0 To UploadPathFin - UploadPathInicio UploadPath = UploadPath & Chr(Bytes(UploadPathInicio + u)) Next _UploadPath = Trim(UploadPath) End If If Acumulador = 11 Then UserIDInicio = x + 2 End If If Acumulador = 12 Then UserIDFin = x - 1 Dim UserID As String Dim u As Integer For u = 0 To UserIDFin - UserIDInicio UserID = UserID & Chr(Bytes(UserIDInicio + u)) Next _UserID = Trim(UserID) End If If Acumulador = 13 Then ArchivoInicio = x + 2 End If If Acumulador = 14 Then ArchivoFin = x - 1 Dim Archivo As String Dim u As Integer For u = 0 To ArchivoFin - ArchivoInicio Archivo = Archivo & Chr(Bytes(ArchivoInicio + u)) Next Dim arrayArchivo As String() arrayArchivo = Split(Archivo, ";") Archivo = Trim(arrayArchivo(UBound(arrayArchivo))) Dim arrayArchivoFinal As String() arrayArchivoFinal = Split(Archivo, Chr(34)) Archivo = Trim(arrayArchivoFinal(UBound(arrayArchivoFinal) - 1)) Archivo = Path.GetFileName(Archivo) _NombreArchivo = Trim(Archivo) End If End If If Acumulador >= 16 Then Dim y As Integer Dim LHead As Integer = x + 3 Dim LBody As Integer = Bytes.Length - LHead Dim HeadBytes As Byte() = New Byte(LHead) {} Dim BodyBytes As Byte() = New Byte(LBody) {} For y = 0 To HeadBytes.Length - 1 - 2 HeadBytes(y) = Bytes(y) Next _HeadBytes = HeadBytes For y = 0 To BodyBytes.Length - 1 BodyBytes(y) = Bytes(LHead + y - 1) Next _BodyBytes = BodyBytes Exit Sub End If Next End Sub End Class

_{Tuesday, July 15, 2003 10:21 AM}

I must say that this has been one outstanding thread, and I appreciate everyone's posts thus far. I have one question about parsing the raw request. What happens if the end of one buffer contains only part of a boundary? For example, you are reading the request in 5k chunks and one buffer ends with "7d32" and the next one begins with "b51510414". When searching the buffer for the boundary, it will not be found in either of these chunks. Parsing the raw http request is not difficult when you have the entire request, but i'm a little confused about how to do it when it comes in chunks. I know that this is not a likely scenerio, but none the less, it could happen. Any direction would be greatly appreciated. Thanks.

_{Tuesday, July 15, 2003 5:29 PM}

Hi ..dude? :-) Good point about the chunks problem! I've been through a couple of ways to solve the problem - most ended in ugly code. However, this solution might be of interest : I assume you've got a buffer holding a chunk of request data 1. Search buffer for boundary 2. If boundary is not found: write buffer.length - boundary.length to file! This is the trick! Fill buffer. Goto step 1. 3. If boundary is found, only write to boundary...of course :-) It is short and simple to implement - only filling the buffer is a bit cumbersome, since some data will still be left in the boundary between iterations. However, you can isolate this problem in a helper function. kind regards, Rasmus

_{Tuesday, July 15, 2003 5:43 PM}

HTTP Module -> Http Handler: Just want to tell you that the upload component can be written as a HTTP Handler instead of a HTTP Module! There is really no trick to it. Just copy/paste the code into a http handler instead. I've moved to a http handler to get rid of the performance penalty that a http module gives. It is irritating to know that every page request has to go around a http module, even if 99 out of 100 request aren't file uploads. With http handlers we are back to more standard ways of doing things. The upload can still be fully interceptet and uploaded in small chunks, even though we are now at a much lower level at the http pipeline. However, I still think we have to stay away from using session state. I think this will trigger the standard request parsing...though someone might be able to work around this? Kind regards, Rasmus

_{Tuesday, July 15, 2003 6:47 PM}

Hi all. I'm back to my favourite thread. Well everything is working perfectly for me except for one item. The dredman variables. What am I missing in this thinking? 1. Someone submits a 500 meg file from a form page with one form field input. 2. I do some stuff that tells me this person can't upload to the site. 3. Now what do I do? I don't want to sit through the 500mb - I just wanna get this guy out of my hair right now and redirect him to another page or the same page - I don't care. What I don't wanna do is give him a nondescript error page like is happening now. So what exactly do I pass in as the inData variable to the dredman variables? Do I pass in the byte stream of a request I can hard code to a certain page etc. Any help would be appreciated. What would be interesting would be the code for the smallest byte array you can pass in for a valid request. And why don't they showcase a solution on the home page of www.asp.net and help us instead of showcasing that webmatrix joke. -Goyaman

_{Tuesday, July 15, 2003 7:03 PM}

I have been wondering same thing...this thread has been going on for well over a year (I think), and although in other threads you occasionally see a post from someone from microsoft, are there any such posts here? It doesn't seem to be a completely off-the wall propriatary thing, being able to handle file uploads in a user-friendly way... I have not worked recenlty on this problem, but believe I am close to solution now. I am still some distance from where you appear to be goyaman, but my C# is limited so am working with VB :(

_{Tuesday, July 15, 2003 8:17 PM}

Hi Goyaman Nice to have you back :-) So what exactly do I pass in as the inData variable to the dredman variables? Do an upload of an empty file from a form without a viewstate (ie without a runat=server). Read the dredman vars (replace the SetValue with a GetValue - the parameters are almost the same). Once you've got them, insert them to dismiss the upload. If you run into trouble with the dredman code, look for some info on asp.net Reflection on the net. I played around with it some time ago. I think InData was just supposed to be "" - but I cannot quite remember. Anyway, using the above mentioned method you'll find out. kind regards, Rasmus

_{Tuesday, July 15, 2003 8:29 PM}

Goyaman - more on the dredman vars. I found some old code in a backup file. Can't remember how well it worked - I ended up not needing it, so I abandoned it. But it may be of interest. (hwr = the current httpworkerrequest) private void ClearRequestData() { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = hwr.GetType().BaseType; type.GetField("_contentAvailLength", bindingFlags).SetValue(hwr, 0); type.GetField("_contentTotalLength", bindingFlags).SetValue(hwr, 0); type.GetField("_preloadedContent", bindingFlags).SetValue(hwr, null); type.GetField("_preloadedContentRead", bindingFlags).SetValue(hwr, true); //type.GetField("_contentType", bindingFlags).SetValue(hwr, 0); type.GetField("_contentLengthSent", bindingFlags).SetValue(hwr, true); } private void SetRequestData(byte[] Data) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = hwr.GetType().BaseType; type.GetField("_contentAvailLength", bindingFlags).SetValue(hwr, Data.Length); type.GetField("_contentTotalLength", bindingFlags).SetValue(hwr, Data.Length); type.GetField("_preloadedContent", bindingFlags).SetValue(hwr, Data); type.GetField("_preloadedContentRead", bindingFlags).SetValue(hwr, true); } private void GetRequestData() { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; Type type = hwr.GetType().BaseType; ctx.Response.Write("BaseType : " + type.ToString() + "
"); ctx.Response.Write("ContentAvailLength : " + type.GetField("_contentAvailLength", bindingFlags).GetValue(hwr).ToString() + "
"); ctx.Response.Write("ContentTotalLength : " + type.GetField("_contentTotalLength", bindingFlags).GetValue(hwr).ToString() + "
"); ctx.Response.Write("PreloadedContentRead : " + type.GetField("_preloadedContentRead", bindingFlags).GetValue(hwr).ToString() + "
"); ctx.Response.Write("ContentType : " + type.GetField("_contentType", bindingFlags).GetValue(hwr).ToString() + "
"); ctx.Response.Write("ContentLengthSent : " + type.GetField("_contentLengthSent", bindingFlags).GetValue(hwr).ToString() + "
"); ctx.Response.Write("PreloadedContent :
" + DecodeToString((byte[])type.GetField("_preloadedContent", bindingFlags).GetValue(hwr))); }

_{Wednesday, July 16, 2003 11:41 AM}

Gotta Ask! Has anyone done a version of this implemented as an HttpHandler that they're willing to shared a downloadable, working version of (or even post to GotDotNet). This is an issue which many ASP.NET developers don't know about, having a free version of an upload / download component would be great.

_{Wednesday, July 16, 2003 12:20 PM}

Oh man I'm am so way ahead of you. Did this...no luckeroo. > Do an upload of an empty file from a form I get a blank form. Then I placed the exact output to the _preloadedContent. Then when a ridgy didge file comes through I set the dredman vars with the saved byte array of the empty form - and update the size and stuff. No lucky luck. So what else do I have to do? Like all I wanna do is stop a request that's too big depending on the target url!!! and when I say stop I mean redirect it to another page cleanly. Obviously anyone who got this working is not reading this thread anymore. -Goyaman

_{Wednesday, July 16, 2003 2:43 PM}

You are absolutely right. I haven't solved the problem the way you propose - I've tried - but: I don't think you can dismiss the upload simply by fixing the dredman vars, since I believe it is partially related to the client browser expecting the server to accept its request! Try to do an upload to a standard html page and you'll get the same error. Obviously the client browser is involved. However, it really isn't a problem to work around this. Do your upload in a hidden frame (just set a target for the form). Keep another frame for informing the user of progress. When an oversize upload is detected (easy - we are in control the SECOND the upload begins using the solutions discussed above) , set some text to be displayed in the progress frame - and let the hidden page fail! Why not? The users are looking at your progress frame anyway, the server is not wasting resources on it, so who should this irritate? Kind regards, Rasmus

_{Wednesday, July 16, 2003 3:27 PM}

>I don't think you can dismiss the upload simply by fixing the dredman vars, since I believe Okay so in addition or instead of the dredman vars (which are quickly becoming useless) how do I dismiss the upload? So for that matter in what application are the dredman variables useful in any way? And how have people used them. ( the question is not directed at you rasmus, just to somebody who has solved this - if indeed anybody has who is not working for ABCUpload or whatever ) The workaround is fine as a workaround but not a solution. SO HAS ANYBODY AT ALL EVER DISMISSED A DOWNLOAD NICELY WITHOUT A SERVER ERROR? If nobody has I shall stop my work right now and move on to something else. Just all this dredman variable talk was interesting but entirely useless to any application in regard to the component. -Goyaman

_{Wednesday, July 16, 2003 4:29 PM}

The dredman vars still serve a very big purpose. They allow you to do the following trick during upload: 1. Intercept the upload in a http module at a very early stage. Do the file uploads. Correct the dredman vars to fool the rest of the http pipeline so that you may proceed to : 2. Handle the request on a normal aspx page, just as you would any other control. Now besides finishing the handling of the uploaded files, you may use any other posted data - at best the control could work so transparently that it would be completely like every other aspx control (linkbuttons, datagrids,...). So you see the dredman vars are interesting when you need to stop the normal upload from taking place. If you just do the upload and let the processing continue, things will fail as the dredman vars indicate that there are data to be read - when in fact you have already read it. I hope it makes sence - written a bit sloppy :-) kind regards, Rasmus

_{Thursday, August 14, 2003 3:20 PM}

Hi all Ì have search a lot on internet for uploading large file in .Net and find that thread. I have tried to parse the raw HTTP file without any success. Here is my approach: 1. I open the raw file with File.OpenText 2. Get the first line Some ID?!? 3. I read the file file.ReadLine() until i get that same line "Some ID" 4. Get the current line that is the line with the filename="uploadfilepath" 5. Read the next line that is the Content-type of the upload file 6. Read the next 2 lines For each of those step i have a byte conter that increment after each file.ReadLine(), and + 2 for the new line and line feed. That counter should be the start for reading the upload data. The way i get the end byte i simmilar i read while i get the "Some ID" here is some code: StreamReader sr = File.OpenText(requestfile); rid = (string)sr.ReadLine(); startread = rid.Length + 2; tmp = (string)sr.ReadLine(); startread += tmp.Length + 2; while (tmp != rid) { tmp = (string)sr.ReadLine(); startread += tmp.Length + 2; } tmp = (string)sr.ReadLine(); startread += tmp.Length + 2; idx = tmp.IndexOf("filename=\"); upload_filename = tmp.Substring(idx, tmp.Length - idx - 1); upload_content_type = (string)sr.ReadLine(); startread += upload_content_type.Length + 2; tmp = (string)sr.ReadLine(); startread += tmp.Length + 2; endread = startread; tmp = (string)sr.ReadLine(); while (tmp != rid) { endread += tmp.Length + 2; tmp = (string)sr.ReadLine(); } endread -= 3; sr.Close(); Now that i have my start byte and end byte of the upload data, i read with a FileStream and write to another file that should contain just the upload data. For text file i work great, but if i use anything linke image or exe, that does not work. Here is the second part: FileStream f = new FileStream(requestfile, FileMode.Open); filepos = 0; byte[] a_buffer = new byte[1]; int i = 0; while (i < startread) { filepos = f.Read(a_buffer, 0, a_buffer.Length); i += filepos; } filepos = i; //FileStream w = new FileStream("c:\request\" + Path.GetFileName(upload_filename), FileMode.Create); FileStream w = new FileStream("c:\request\test.txt", FileMode.Create); upload_file_read = 0; while (filepos <= endread) { upload_file_read = f.Read(a_buffer, 0, a_buffer.Length); filepos += upload_file_read; w.Write(a_buffer, 0, upload_file_read); } f.Close(); w.Flush(); w.Close(); If anyone can help me with that, i have read your post here and wow i would like to be as good in .Net, but i'am starting and i have lot to learn and experience. Sorry for my english :) thanks

_{Saturday, September 6, 2003 8:51 AM}

Hi Guys I'm posting this in the hope that someone is still tracking this thread. I'm trying to write a an upload function for a website I'm doing. The files that will be uploaded are nothing on the scale of what you guys have been discussing uploading, probably a max of 1 to 1.5 mb,. But they have to be uploaded over a dial connection and I don't trust the users to wait whilst it uploads so I need to popup a progress indicator. The code for the function is as follows Public Function uploadLetter(ByVal postedFile As HttpPostedFile, ByVal fileName As String) As Boolean Const buffsize As Int32 = 8196 Dim fStream As New FileStream(HttpContext.Current.Server.MapPath("/") & "/../data/" & fileName, FileMode.Create, FileAccess.Write, FileShare.Read) Dim buffer(buffsize) As Byte Dim totalLen, recieved As Int32 Dim context As HttpContext = HttpContext.Current Try context.Application("total") = postedFile.ContentLength While totalLen <= postedFile.ContentLength buffer = New Byte(buffsize) {} recieved = postedFile.InputStream.Read(buffer, 0, buffsize) fStream.Write(buffer, 0, recieved) totalLen += recieved context.Application("recieved") = totalLen context.Application("start") = True If totalLen = postedFile.ContentLength Then Exit While End If End While fStream.Flush() fStream.Close() Return True Catch ex As Exception If Not fStream Is Nothing Then fStream.Flush() fStream.Close() End If Dim fs As File If fs.Exists(HttpContext.Current.Server.MapPath("/") & "/../data/" & fileName) Then fs.Delete(HttpContext.Current.Server.MapPath("/") & "/../data/" & fileName) context.Application("error") = ex.Message Return False End Try End Function This code does work to a degree, if I upload a small file then theres no probs, a file of around 1.5mb seems to fail without reason. My progress page is launched when the submit button is clicked and runs under a seperate httphandler which checks the application variables from this function, I'm using application variables because after reading a tutorial on devshed the author said that application variables are updated before the end of a loop. The main problem I appear to be having with this is that entire file seems to be posted before this function writes it to file, I would like to be able to read a certain amount of the file in write that to disk and then loop round. I know this is wrong and I need to get this implementation finished this weekend so any help would be much appreciated. Thanks in advance Dan

_{Saturday, September 6, 2003 9:35 AM}

From my failed attempts at solving this ASP.Net upload issue, I believe once you do "postedFile.ContentLength" it gets the whole file from the client. How to fix, I am sorry I do not know. I hope to have some time to work on this again soon here. I'm almost to the point of looking for commercial option for the handler. Although, ABCupload .Net is extremely expensive for re-distributable version :( Someone has sent me a Java-solution, but I have .Net, I thought I didn't need Java anymore? :~

_{Thursday, September 11, 2003 1:29 PM}

I'm not sure anyone is still tracking this thead but I'll respond anyway... goyaman said: >I get a blank form. Then I placed the exact output to the _preloadedContent. Then when a >ridgy didge file comes through I set the dredman vars with the saved byte array of the >empty form - and update the size and stuff. No lucky luck. You need to make sure that the boundry in your faked form matches the one in the HTTP_CONTENT_TYPE header of the response. Here is my code that calls dreamans method with a fake blank form minus the file upload: string contentType = request.GetServerVariable("HTTP_CONTENT_TYPE"); string boundary = contentType.Substring(contentType.IndexOf("boundary=") + 9); string fakeResponse = boundary + "\r\n" + "Content-Dis-data; name=\__VIEWSTATE\\r\n\r\n" + "dDwxMzEzMzQxNDI3Ozs+kMaQAfIExskEIrXzi20oOWmfvW4=\r\n" + boundary + "\r\n" + "Content-Dis-data; name=\Button1\\r\n\r\n" + "Upload\r\n" + boundary + "--\r\n\r\n"; byte[] fakeBytes = System.Text.Encoding.ASCII.GetBytes(fakeResponse); PopulateRequestData(request, fakeBytes); HTH, Uranys

_{Thursday, September 11, 2003 4:25 PM}

Whoops, wrong code... I meant this: string contentType = request.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentType); string boundary = "--" + contentType.Substring(contentType.IndexOf("boundary=") + 9); string fakeResponse = boundary + "\r\n" + "Content-Dis-data; name=\Button1\\r\n\r\n" + "Upload\r\n" + boundary + "--\r\n\r\n"; byte[] fakeBytes = System.Text.Encoding.ASCII.GetBytes(fakeResponse); PopulateRequestData(request, fakeBytes);

_{Monday, October 6, 2003 11:11 AM}

It works for me... but, I've got some questions...! - I see there's only a text-file placed in the directory I want the file to be placed in... where is the actual file I selected? - How can I read the http buffer from another aspx page to see how many bytes are uploaded? I'm talking about the refresh-page here that you mentioned! Can anyone give me some tips please?! If this is going to work then I'd be very happy!!! Greetz, Gerard

_{Thursday, October 16, 2003 5:50 PM}

The Good: Got the code working and save everything into a text file. Thanks. The Bad: Working on parsing and decoding off the stream as a novice beginner The Ugly: Lost another round against folks pushing Cold Fusion. They seems to have solved the problem aeons ago. They can upload files with a simple tag, without bugging down the server and pass it onto a FTP server or many more if in the same domain. :-(

_{Wednesday, October 22, 2003 9:27 AM}

Would any one care to explain the above code a little more..? I’ve been trying to figure this out but I’m having a little trouble. What would be great if some one could give a quick over view of what happening and what pages are needed, with perhaps some basic code examples of how to use this code from an input field on a aspx page.. Many thanks..

_{Wednesday, October 22, 2003 7:39 PM}

ricc, Not trying to sound "unhelpful", but if you read carefully through all of the posts in this thread, you will find there is more than enough intuition on the topic. A lot of very well skilled people spent a lot of analysis and coding time working with each other to find a solution to the original problem. My suggestions: 1 - understand the concept of this thread 2 - analyse the code generously provided to this thread 3 - do the rest yourself through your own skills. Clearly you have not read through this thread looking at your question - your answers have already been posted. Regards, Paul.

_{Thursday, October 23, 2003 5:43 AM}

I have read through the above posts... All I'm asking is a little bit of help to get started, almost a mile high over view of where to start... taking it back to basics. I appreciate that a lot of talented people who clearly know their stuff have put time & effort into this post, but correct me if I’m wrong but I thought that one of the points of forums like this is to help and spread knowledge, which enables us with “lesser” ability to learn etc. Trust me I’m not taking this forum or any information given for granted. Thanks

_{Thursday, October 23, 2003 4:20 PM}

Someone else from this forum asked me a similar question the other day...I'll post my reply to his question...he was unsure of how to start, but talked about using http module: Hi Dont bother with httpmodules. Setup a httphandler - see the .net framework sdk for tutorials. From handlers ProcessRequest method, grab the HttpWorkerRequest with this function: private HttpWorkerRequest GetCurrentHttpWorkerRequest() { BindingFlags Flags = BindingFlags.NonPublic | BindingFlags.Instance; Type ContextType = HttpContext.Current.GetType(); PropertyInfo ContextTypeInfo = ContextType.GetProperty("WorkerRequest",Flags); object CurrentHttpWorkerRequest = ContextTypeInfo.GetValue(ctx,null); return (HttpWorkerRequest)CurrentHttpWorkerRequest; } eg. HttpWorkerRequest hwr = GetCurrentHttpWorkerRequest(); The request will be partially in a preloaded buffer. get the first part like this: byte[] PreloadBuffer = hwr.GetPreloadedEntityBody(); grab the rest like this byte[] Buffer = new byte[GetRequestLength()]; hwr.ReadEntityBody(Buffer,Buffer.length); using the following function as helper: internal int GetRequestLength() { return Convert.ToInt32(hwr.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength)); } Now you've got the whole request, divided into two byte arrays PreloadBuffer and Buffer. Your task is then to convert the byte arrays to strings and filter away the html formular, leaving only the uploaded file, which can the be saved. Ofcourse you should not read the whole request in one go (eg. not call ReadEntityBody with RequestLength, but in small chunks). Use the following function to convert a byte array to a string: private string DecodeToString(byte[] Bytes) { Decoder Decode = HttpContext.Current.Request.ContentEncoding.GetDecoder(); char[] Chars = new char[Decode.GetCharCount(Bytes,0,Bytes.Length)]; Decode.GetChars(Bytes,0,Bytes.Length,Chars,0); return new String(Chars); } (might require "using System.Text" and other stuff at top of class) Happy coding. Kind regards, Rasmus

_{Friday, October 24, 2003 5:10 AM}

Hello Rasmus Still banging my head against a very hard and cold wall!!. I started of using both the code that codeRage(vb) and Bjorn.B (c#) posted on the first page of the post (in two different solutions). This obviously worked and a text file is written to disk. Now I’ve got the same problem marekK had on page 3 of post, in that The GUID.txt file in the Upload folder contains the name of the file (as a file=filename), and not the raw http request. Why is this? I understand that once the raw request is written to the file I then have to filter away the headers etc… I’m sure that the answer as to why I’m only writing the name of the upload path to the text file lies somewhere in this post but I can’t seem to find it… I’m on a steep learning curve at the moment! One I have this sorted I’ll move on to setting up a httphandler like you suggest. (ie I’ll follow the rest of the code bits and bobs through out the post! Now I’ve started this I’m determined to sort it out :)

_{Friday, October 24, 2003 10:46 AM}

Hi It sounds like you are missing the encType="multipart/form-data" on the

tag on your upload page. Kind regards, Rasmus

_{Sunday, October 26, 2003 9:01 PM}

Count me among those who have been studying this thread. My progress thus far has been to create the initial upload form page, the module and the redirect page. Running my code places a text file in the upload directory that contains the contents of the uploaded file and the header info. What I cannot discern from the thread is how to strip out the unnecessary stuff from the uploaded file and then save/rename to the desired file name.

_{Tuesday, October 28, 2003 2:23 AM}

Hi guys, I have been following the discussions in this thread for the last couple of days. Have you tried using HttpPostedFile class for your problem. It will relieve you of that arduous task of parsing the request stream. Instead you will get a raw file data as HttpPostedFile which you can read / save. The code would be something like this: In the page where your file submission is to be handled: HttpPostedFile myFile = this.Request.Files.Get(0); if (myFile != null) { string strFileName; // Get the file name strFileName = myFile.FileName; strFileName = strFileName.Substring(strFileName.LastIndexOf("\") + 1); // Save myFile.SaveAs("C:\" + strFileName); } Regards, parthoroy

_{Tuesday, October 28, 2003 3:47 AM}

Once you make this call: HttpPostedFile myFile = this.Request.Files.Get(0); It downloads the entire file. No streaming, no progress trapping, and all of the lovely associated-IIS upload problems appear. Someone correct me if I am wrong. I will be attempting this again soon, my last 2 attempts ended in un-acceptable workarounds to the problem. I really can't grasp the concept of streaming through the data byte-by-byte, and having done in an efficient enough way. As close as I got, the streaming part seemed to be a very bad idea. It seemed very poor-performing inefficient code. Hopefully I can find a solution soon. I still don't understand why Microsoft, with everything in .Net and IIS, has not provided a decent integrated solution for uploading.

_{Tuesday, October 28, 2003 3:53 AM}

Thanks parthoroy. File uploading is not the problem. Uploading large files in chunks and having a progress indicator is the problem. I have windows media files that editors upload and they are in the range of 300 meg each.

_{Tuesday, October 28, 2003 9:48 AM}

As they have said, parsing the result is not the subject of this topic. Most of them have already sold their code for commercial use, so no one will post a perfect solution here, you have to figure it out yourself. And if you are lucky as me, have some Mac users, they may post Mac binary, encode their uploads and have multi parts for a single files ... good luck ^_^

_{Tuesday, October 28, 2003 12:19 PM}

Thanks ups101 I had encoding="multipart/form-data"> and not encType="multipart/form-data" on the

tag on my upload page Anyway This has to be one of the most difficult threads to try and work through and learn from! (and unfortunately it’s the only place on the web that’s discussing this topic) I appreciate that no one is willing to post a fully working “compile and go” solution, fair enough, but it does seem that there is a lost of interest in this subject. – Great opportunity for some one to write a tutorial :) Until that point I’ll keep pressing on. I think I’ve got the javaScript progress bar sorted….found an article http://www.devhood.com/tutorials/tutorial_details.aspx?tutorial_id=673 little bit of tweaking but looks promising. However I’ve got a couple of things regarding the more complex area: 1. Would someone please go over the process of how the request is buffered (and the incoming data is “stored” before it is written to disk. – Where are the “Chunks” of the uploaded file stored before the its written to disk? 2. On the same note what is the importance of redirecting the process after upload, I know this has something to do with stopping the browser from hanging ie once there are no more bytes to write. 3. While working through the code snippets I’m trying to figure out the process of Dredman’s code on Page 4, I’m stuck on the following line: DataReader byteReader = new DataReader(arrBoundary,httpApplication.Request.ContentEncoding, ContentLength, progress); byteReader.ReadBuffer(arrBuffer, arrBuffer.Length); When I try to build it I get an build error stating that the type or namespace “DataReader” could not be found. What is this DataReader? I know I’ve asked a lot Thanks

_{Tuesday, October 28, 2003 12:52 PM}

Parsing the files between content-boundry is the largest piece of this topic. To get the un-parsed stream almost didn't require a thread.

_{Tuesday, October 28, 2003 6:12 PM}

To anyone who has a complete solution for the topic under discussion - I would be willing to pay for the original code, preferably in vb. Contact me at [email protected]

_{Wednesday, October 29, 2003 8:54 AM}

Great work guys, but has anyone tried it on IIS6? It works fine on Windows 2003 server in IIS5 compatibility mode but not on IIS6.

_{Wednesday, October 29, 2003 9:10 AM}

I have the proposed code running on a Windows 2003 server. I have never specifically adjusted for IIS5 compatibility - doesn't that mean that it is running "normal" IIS6? And if yes, then it appears to work. If no, how do you adjust the compatibility mode? Kind regards, Rasmus

_{Wednesday, October 29, 2003 9:50 AM}

It's running IIS6 by default. You can switch in IIS manager > Web Sites > Service tab. It goes wrong here: type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); <-- ok type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); <-- ok type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); <-- fails type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR, true); <--fails So I started to play a little with the debugger and found out that when I change the line Type type = inWR.GetType().BaseType; into Type type = inWR.GetType().BaseType.BaseType; all goes well... The difference that the second line returns System.Web.Hosting.ISAPIWorkerRequest and the first System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6. Unfortunately there's no documentation on these classes. I can't even use them for casting, comparing (IS operator) or whatever.

_{Wednesday, October 29, 2003 10:18 AM}

ah, the dreadman vars. That is why I hadn't seen these errors - I am not using them. Nice debugging though :-)

_{Thursday, October 30, 2003 4:33 AM}

Rasmus, if you're not using the dreadman vars, does that mean that you are not filtering the file(s) form the request and only watch the progress? Or do you use a redirect or did you figure out something else?

_{Thursday, October 30, 2003 6:21 AM}

I parse the entire request, dividing it into files and text fields, from a httphandler. Files are saved, fields are placed in a hashtable for later processing when upload completes. I think the dredman vars will give you a cleaner solution, but I haven't been down that road.

_{Thursday, October 30, 2003 3:02 PM}

I came across the same problem when we went from IIS on win2000 to IIS on win2003 and used the follwing private static byte[] PopulateRequestData(HttpWorkerRequest inWR, ref byte[] inData) { BindingFlags bindingFlags = BindingFlags.Instance | BindingFlags.NonPublic; int i = (int)inData.Length; Type type = inWR.GetType(); while (((type != null) && (type.FullName != "System.Web.Hosting.ISAPIWorkerRequest"))) { type = type.BaseType; if (type != null) { type.GetField("_contentAvailLength", bindingFlags).SetValue(inWR, i); type.GetField("_contentTotalLength", bindingFlags).SetValue(inWR, i); type.GetField("_preloadedContent", bindingFlags).SetValue(inWR, inData); type.GetField("_preloadedContentRead", bindingFlags).SetValue(inWR, true); } return inData; }

_{Thursday, October 30, 2003 4:51 PM}

First, I'd like to say thanks to everyone who contributed to this thread. The information here was very helpful. I am almost done with the Upload aspect. Everything is working except for the need to redirect when the upload is completed. I'm looking at the PopulateRequestData(...) method and not sure what does the inData array represents. Does it represent the buffer that you were reading request chunks into? or does it represent the whole request data? Any pointers would be helpful... Thanks again del

_{Friday, October 31, 2003 10:51 AM}

For the record, I did read all the posts in this thread and I know that someone did mention that the byte[] inData should be the actual request data without the uploaded file. However, when I try to do that I still get the "Max Length" error. What I did was basically copy the HTTP header from the GetPreloadedEntityBody() method (leaving out whatever was preloaded from the actual file of course). I then added the end boundry bytes to it. I also checked the contents of the array to ensure that I'm not missing a "\r\n" somewhere and everything was ok... Lastly, I call the PopulateRequestData() with the array that I have constructed and still get the "Max Length" error. Any suggestions? Thanks del

_{Thursday, November 6, 2003 8:42 AM}

I am once again struggling with parsing the headers. My problem must be something very simple, I search each byte-buffer as a string but I don't think I am converting the string back properly to bytes. Here is an examples on how I convert back, before I write to the file: byte[] curBytes=Encoding.ASCII.GetBytes(writebuffer.ToCharArray()); fsCurFile.Write(curBytes,0,curBytes.Length); Any help is greatly appreciated.

_{Thursday, November 13, 2003 3:53 AM}

Sorry, could anyone help me to solve the problem? Could you send me a zip file cointaning the code to solve this problem? I'm becoming mad.. Please, help me. my email address is [email protected]

_{Sunday, December 21, 2003 4:17 PM}

I've implemented the file upload progress based on code posted here, and it has been working a charm. My problem is we've implemented the web garden feature in IIS. The uploads all work, however the actual progress feedback is erratic, I'm assuming that the post is being handled by one process of the web garden and the progress feedback is being passed around the other members of the web garden with each update request. I use a static Hashtable and a static int for the progress data and progress id. Is there a way to share these across the web garden?

_{Tuesday, January 6, 2004 2:51 PM}

This may be a dumb question but I'll ask anyway. I have this set up and working for my large file uploads within a HTTP Module. What if I want to use the regular ASP.NET Upload functionality on some pages, and use this on others. Also, what if I have a number of different pages accessing this module and I want to save to different locations. I have been able to do this parsing out the request stream, but it gets to be a pain. I guess this may be the only way of succeeding here. Any suggestions on how I can accomplish these things (allowing some pages to use regular ASP.Net uploading rather than this module)?

_{Tuesday, January 6, 2004 4:34 PM}

Being able to handle different requests differently is a job for HttpHandlers. However, since you have an HttpModule installed, all requests will go through it and then to the various HttpHandlers. What you could do is check for "something(s)" in the HttpModule's BeginRequest() method. For example, in my case, I check whether the posted form is actually a "multipart/form-data" one. If it is, I call my HandleUpload() and do nothing otherwise. As for having different pages accessing the module and the ability to save to different locations.. that should not be a problem provided that you keep track of the "active" uploads going on inside your HttpModule.

_{Monday, January 19, 2004 1:00 PM}

Is the full source code to this solution available somewhere in one place (rather than spread among 120+ posts)? Thanks, Shawn

_{Monday, January 19, 2004 4:41 PM}

It would be nice to have that. I'm certain that many people have complete solutions by now, but unfortunately, no one is willing to publish his/her solution.

_{Monday, January 26, 2004 12:17 PM}

I must be missing something here... Part of the body data exists in a preloaded byte array (48 k buffer provided by IIS), which is exposed by the HttpWorkerRequest.GetPreloadedEntityBody method. The rest (if there is more that didn't fit into IIS' buffer) is a byte stream that is read from the client using the HttpWorkerRequest.ReadEntityBody method. The basic task here seems to be to read all of this data -- whether from the IIS internal buffer, or from the client directly; parse it; save off what appear to be files (based on the MIME multipart boundary and information immediately after that); and then somehow allow the rest of the request data to be passed on from our 'filter' (HttpModule or HttpHandler) to whatever is waiting for it down-the-line -- eventually, to an .aspx page. It's the last part I don't get. We've read all the data by the time we're done. Both IIS and the client have nothing left to give. How is whatever going to get at the rest of the data from the request body that we didn't care about? E.g., form fields, viewstate, etc.? And what form will that take? E.g., do we instantiate a new HttpRequest object and somehow make it seem that the our new object is the 'real' request? I see dredman's post about the variables that need to be changed to make the expected request size smaller. And breaking changes in IIS 6. Okay. But what about passing along the request itself, along with our -reconstructed- request body? How is that done? BTW, THANKS to everyone involved here. I personally worked at this, off and on, for over a year before coming across this thread. Someone else said you folks were the only ones on the Internet discussing this topic -- must say I agree, and you've done a fantastic job. If you can just fill in this last blank for me, I think I'll be able to build my own solution based on these concepts. Thanks again, - Don

_{Monday, January 26, 2004 3:06 PM}

One thing to keep in mind is that once you use the Request object (or instanciate a new HttpRequest object), the entire request data is loaded into IIS memory! So, if you have 100 MB file being uploaded, you would be loading that 100 MB into memory before you have access to it. You'll have to work with WorkerRequest object (the GetPreloadedEntityBody() and ReadEntityBody() methods). There is a server-side control called ABCUpload.NET. It's really helpful to look at its "ldasm"ed version. They keep track of the upload in buffer (similar to the PostedFiles in HttpRequest). This way your application will keep the "100MB file" in memory and not IIS. Of course there are other solutions ;)

_{Monday, January 26, 2004 4:17 PM}

Yes, I understand the basic problem. Perhaps I wasn't very clear. Following examples provided in this thread, I now have an HttpModule that: - Detects whether the content body is multipart/form-data encoded - Detects whether the request is larger than X bytes (otherwise, exit, let the normal Request.Files thing work) - Buffers the request and spools it onto disk I am working on parsing the request from the GetPreloadedEntityBody and ReadEntityBody methods separately. Specifically, I'm trying to avoid turning these into strings -- trying to parse them within byte arrays without doing any type conversion. What I need to understand is: Assuming that I can get to a point where I have only the files saved off to disk, now what? How do I let the rest of the request -- with files stripped out -- go through the rest of the execution pipeline? I had to read all of the request body bytes, MIME-encoded, in order to prevent the exact thing you're talking about (having the whole request in memory). Parts of the body were files, and they are now in my disk-based spool (presumably awaiting further processing by the page that was requested when my user uploaded the file). Other parts of the request body were form fields or ViewState data that are now needed so that the final aspx page can do its job. I followed the examples given here, but no one really specifically talked about how to 'complete' the request. I understand dredman is setting his famous variables so that the remainder of the request - minus files - can go through. So how do I "send it through"? Thanks for the response. I hope this clarifies the question, and that others like us are still monitoring this thread and can clarify how they jumped this hurdle... Regards, - Don

_{Tuesday, January 27, 2004 9:11 AM}

I believe that once you properly setup those "dredman" variables the request will go through smoothly. This is due to the fact that an HttpModule is part of the processing pipeline and not the end of it. Typically, all requests go through the HttpModule, but since we are "stealing" some of the bytes and put them in a file, the rest of the pipeline (at the other end of the HttpModule) will crash because it did not receive what it was expecting. I have not done that (setting up those variables) because I don't need to do that for my application (try to think about cancelling the upload in the middle ;) ) I use the Request.Redirect() method.. But again, it really depends on your application's requirements

_{Wednesday, February 4, 2004 1:24 PM}

Nobody mentioned the SoftArtisans FileUp component. It has mixed managed and unmanged code, so you have to use it through Interop, but it can do very large file transfers with progress indication and has an HttpModule written already - just in case you dont feel like writing and maintaining this code from scratch. I have been developing with it from the early ASP days without a hitch, and it now does a great job with .NET file transfers. They just released version 5 that contains the HttpModule. There is a free eval at http://fileup.softartisans.com

_{Thursday, February 5, 2004 3:30 PM}

Thank you deltron, do you know whether it supports cancelling the upload in the middle of the process? I can't find that on their web site and too lazy to ildasm it ;) Thanks again

_{Monday, February 9, 2004 3:29 PM}

Great code! After a lot of trying, i finally found your code to help out, thx for that! I hope you can find time to answer this question about your code: Could you pleas help out with the assembly reference to 'ProgressData' I really can't figure it out.

_{Tuesday, February 10, 2004 9:12 AM}

ProgressData was not discussed in this thread. It's a really simple class. All you have to do is to keep track of variable such as the total bytes of the file and the number of bytes received so far, etc. Using such variables, you can then display a progress indicator of the upload.

_{Wednesday, February 11, 2004 7:18 AM}

Thx for you're help! I've got a little bit further. So I've got all the way untill creating an instance of the ProgressMonitor. Then calling for the method addProgress. It has an parameter (int)....but wants to recieve a parameter of the type progressData. Here's where i need an other good tip. How do i go from here so the int gets converted to a ProgressData-type. Can you help ?

_{Wednesday, February 11, 2004 8:58 AM}

Great translation to c# Thx for that. I do have a little problem though: I cant seem to find the reason your code keeps redirecting to : http://localhost/WebApplication1/WebForm1.aspx&uid=F6795F04-109F-4409-8F29-972C1FD84D7C I can't seem to find where this aquerystring is built. Did I mis something, or did i forget to add something.....I'm getting quite annoyed with my own incompetence, please help!

_{Wednesday, February 11, 2004 9:24 AM}

Good to hear that you are making progress at solving this :D. One thing you should note, though, is that it's not my code that is posted here. Many people posted "partial" solutions here and yes.. not all of these partial stuff actually work! So you should feel free to change the parameter requirement for the addProgress(..) method. As for the querystring in: http://localhost/WebApplication1/WebForm1.aspx&uid=F6795F04-109F-4409-8F29-972C1FD84D7C The code that you are using is probably appending it inside the HttpModule. Is it happening after the upload is finished? or as soon as the upload starts? In any case, you'll need a way to communicate the "upload ID" to the server. Just think about the case where you have multiple users uploading at the same time... the upload ID lets the server distinguish between them.

_{Thursday, February 26, 2004 3:10 AM}

For all of you who just want a quick solution to uploading, my component is now available at http://dnu.rasmus.hartvig.name FOR FREE. If you want the FULL source code, I am charging a symbolic amount of 10 USD. This is simply because I’ve spent ages on this component, and I really do believe that this amount is justifiable. It gets a student pizza on the table. BR, Rasmus

_{Thursday, February 26, 2004 12:30 PM}

Thanks for sharing your component. Looking forward to see what features it has :D

_{Monday, March 1, 2004 4:31 PM}

Another good feature to add would be the ability to upload multiple files. I think that abcupload advertises such a feature but I haven't seen it in action... Thanks, Shawn

_{Tuesday, March 2, 2004 3:08 AM}

Actually I think it already does that...just haven't tested it. BR Rasmus

_{Friday, March 19, 2004 11:34 AM}

Rasmus - Nice work...Just bought you a couple of beers - purchased your source code. How do I get the entire filestream? I want to insert the files I upload into a database. Hope you (or anyone else) can help.

_{Thursday, April 15, 2004 12:24 PM}

Where did the control go, not longer able to download it or purchase it (which is what I was going to do)...? Thanks, Shawn

_{Thursday, April 15, 2004 1:24 PM}

Hi all After the data has been stripped out of the request is it still possible to access things like form values in the request. It seems when I try to do anything with the request after an upload it kinda makes asp.net crash almost. And if I try to access the form values before uploading the file it causes the upload to fail. any input would be great. cheers Dan

_{Thursday, April 15, 2004 4:17 PM}

Please let this thread rest in peace. It is too long. The same questions are being asked over and over again. Look here instead: http://www.asp.net/Forums/ShowPost.aspx?tabindex=1&PostID=531432

_{Sunday, April 18, 2004 9:46 PM}

You are lucky... I paid for the control, which has a timing problem which causes the uplaod to crash.. Went to site.. but no-more!! While not an expensive control, still expected better than this...

_{Thursday, April 29, 2004 10:22 AM}

There's no timing problem with the Rasmus code (assuming you actually managed to grab it like I did). All you have to do make sure is in the setting of your web.config file. maxRequestLength refers to the maximum amount of bytes to be transferred (in this case its around 80Mb). It can be a problematic area as hackers can wreak havoc with this, so be careful. ExecutionTimeout is set to 900 seconds to give people with slow connection the chance to actually complete the upload - again, change to suit your application.

_{Monday, May 3, 2004 5:15 PM}

Sorry, but there is a significant problem with the Rasmus code! It is not related to timeouts, but rather to timing between the async processing threads..... The problem is pretty basic and fundermental in nature. To test simply hit abort during an upload...

_{Wednesday, May 5, 2004 3:44 PM}

Here is an article about this topic. I hope it helps. http://www.codeproject.com/useritems/FileUploadProgressBar.asp#xx812928xx

_{Thursday, May 13, 2004 4:12 AM}

Hi, How do I get the correct file name for the uploaded file? Thank you very much Simon

_{Wednesday, July 28, 2004 3:12 PM}

Rasmus, Many thanks for all the valuable information to you and all others who contributed to this discussion. I tried your link today with no luck. Is there a problem with the website ? Sameh

_{Friday, September 24, 2004 9:58 AM}

A really cool solution for upload large file with progress bar to IIS whthout any client software installation, and easy to use. http://www.bestcomy.net

_{Friday, September 24, 2004 2:18 PM}

Nice! Uh, hmm, your server doesn't seem to be working though? Do you release that one with full source code?

_{Monday, September 27, 2004 5:49 AM}

no full source provided, anyone interested on this demo can visit http://www.bestcomy.net to download. If you can't connet to this site, you can contact me at [email protected]

_{Monday, September 27, 2004 6:16 AM}

imho there should be no commercial interest involved in this thread. I mean, how many folks have used the information from this thread to write an upload component and now try to sell it back to the folks here? That's not really fair, so keep away.

_{Monday, September 27, 2004 2:18 PM}

Nice to see this thread is back again. I have followed the posts in this thread and developed an upload component (I can't share the code and it's not for sale). The information available in this thread is enough to get the upload component working. Thanks to all the contributers

_{Thursday, October 14, 2004 9:41 AM}

It seems that Microsoft had already improved upon its file upload control in .NET Framework 2 Beta. I found the uploaded files are no longer cached entirely in memory,meaning it's streamed to disk.

_{Tuesday, November 16, 2004 10:29 AM}

I found this program. It works great. He includes the full source. Right now it is free, but who knows. http://krystalware.com/blog/

_{Thursday, November 18, 2004 12:07 PM}

Hi, I am also planning to do tha same work - uploading files from client to web server to file server without loosing the session time out can you please send me the code snippet or example how can I write the httphandler and httpmodule to achieve this. Thanks in advance, my email is [email protected] Shashi

_{Thursday, November 18, 2004 12:08 PM}

Hi, I am also planning to do tha same work - uploading files from client to web server to file server without loosing the session time out can you please send me the code snippet or example how can I write the httphandler and httpmodule to achieve this. Thanks in advance Shashi

_{Thursday, November 18, 2004 12:39 PM}

Hi, My name is shashi and I am working on a large project which requires some kind of file upload from client to web server to file server. I have read your article on Asp.NET forums and would like to ask you for the help. Can you please let me know or send me the code how to implement this (uploding files) with progress bar or keeping the session alive while the upload happens. OR Can you point me to the right url where I can get all the information to implement this feature with example/s and code. I appreciate your help. Thanks Shashi

_{Thursday, November 18, 2004 12:43 PM}

Hi, My name is shashi and I am working on a large project which requires some kind of file upload from client to web server to file server. As you mentioned I am willing to pay you more than $10 USD. Please help me out. I have read your article on Asp.NET forums and would like to ask you for the help. Can you please let me know or send me the code how to implement this (uploding files) with progress bar or keeping the session alive while the upload happens. OR Can you point me to the right url where I can get all the information to implement this feature with example/s and code. I appreciate your help. Thanks Shashi

_{Thursday, December 2, 2004 4:27 AM}

With the help off all the code I found here, I have been trying to implement this. I have a lot, but i'm running into a problem saving the file: I can catch the raw postback info: 7d4204b3f043c Content-Dis-data; name="__VIEWSTATE" dDwtMTI3OTMzNDM4NDs7Pp/0hcX8BMxmXMGOJJC27tgAEod8 7d4204b3f043c Content-Dis-data; name="file"; filename="P:\logo.gif" Content-Type: image/gif GIF89an (lot of code for gif image) 7d4204b3f043c-- Now i need to filter out the headers to be able to save it to a gif image(on disk). I tryd this by converting everything to string and filtering out the info. This worked, but for some reason my imagedata gets corrupted. Original: 00000000h: 47 49 46 38 39 61 14 01 6E 00 F7 00 00 F7 F7 F7 ; GIF89a..n.÷..÷÷÷ 00000010h: FF FB FF E7 E7 E7 D6 D3 D6 EF EB EF CE CB CE AD ; ÿûÿçççÖÓÖïëïÎËέ 00000020h: 14 00 DE DB DE 18 45 AD 18 49 B5 10 34 84 10 3C ; ..ÞÛÞ.E­.Iµ.4„.< After filter: 00000000h: 47 49 46 38 39 61 14 01 6E 00 77 00 00 77 77 77 ; GIF89a..n.w..www 00000010h: 7F 7B 7F 67 67 67 56 53 56 6F 6B 6F 4E 4B 4E 2D ; {gggVSVokoNKN- 00000020h: 14 00 5E 5B 5E 18 45 2D 18 49 35 10 34 04 10 3C ; ..^[^.E-.I5.4..< I'm thinking that it is the bytes --> string --> bytes convertion that currupts the info, so i was wondering if anyone knows a way in wich I can parse these headers out, just using byte[]. I think i would need a sort of byte[].Split(byte[] originalArray, byte[] splitOnFindingThis); Or is there a better way? I was also thinking, maybe i could split on \r\n, but that would also split up the file itself, so i don't think that is a good idea. Can anyone help me in the right direction?

_{Thursday, December 2, 2004 10:56 AM}

Turned out i shouldn't use ascii but default encoding. This gives corrupt data back: string stringBuffer = System.Text.Encoding.ASCII.GetString( buffer ); byte[] backBuffer = System.Text.Encoding.ASCII.GetBytes( stringBuffer ); This gives good data: string stringBuffer = System.Text.Encoding.Default.GetString( buffer ); byte[] backBuffer = System.Text.Encoding.Default.GetBytes( stringBuffer ); Looks like it is System.Text.CodePageEncoding encoding...

_{Thursday, February 10, 2005 6:40 AM}

Hi Code Rage, Tried out your stuff... Really cool! But got a bit of a problem. How do I get hold of the file that the user is trying to upload? The one in your code uses a self guid generated file to write to the server. But I need to have the original filename that was being uploaded. Thx.

_{Wednesday, July 27, 2005 9:28 AM}

http://www.asp.net/ControlGallery/ControlDetail.aspx?Control=2748&tabindex=2
Try out this Progress Bar is very good featrure.

_{Tuesday, August 30, 2005 7:08 PM}

You might also be interested in NeatUpload.  I recently release version 1.0.  Here's a short description:

NeatUpload allows ASP.NET developers to stream uploaded files to disk and allows users to monitor upload progress. It is open source and works under Mono's XSP/mod_mono as well as Microsoft's ASP.NET implementation. It features 2 custom controls:  InputFile allows the user to select a file to upload, and ProgressBar displays the upload progress either inline or in a popup. ProgressBar even allows users without JavaScript to see upload progress.

Hope that helps,

--Dean

_{Tuesday, November 1, 2005 1:41 PM}

Like many other posters here, I'm developing an application that will need to upload collections of potentially large files.

Is this where it's come to rest for 2.0 - third-party solutions are still required? ASP.NET 2.0, on its own, will choke, tim eout  or hopelessly fragment its memory if you try to use the built-in classes with files of gigabyte size?

Is there no way to get the client to initiate FTP without a java applet or some other installed component?

_{Friday, November 4, 2005 2:25 PM}

I've just recently stumbled upon the asp.net file upload forum.

I'm having problem where the readentitybody returns 0.

Not really sure if the problem was solved on the forum.

My readentitybody does work...until it gets to the end of a file, where the last "chunk" of the file doesn't get to the server.  it just returns 0 for
readentitybody, but never "hangs".

If anyone has any tips, please let me know.
Thank You.

_{Monday, March 13, 2006 10:26 PM}

Well this have been an exhausting thread and I would like to extended my gratitude to CodeRage, ups, and everyone else that had intellectual feedback on this topic.  No doubt this thread will be visited countless more times before it is closed and locked by the Admins so for anyone that comes to this thread and decides to post questions let me recap the entire thread so that you dont waste your keystrokes.

-It is to your advantage to understand both C# and VB.Net when reading this thread as examples are mixed between both languages

-Both CodeRages and ups's code work, however, I found that if I ran the input fields as server controls I was able to get a more worthwhile HTTP Request.  (ContentType, filename, etc)

-Noone on here has provided a "canned solution" for this application, however, there is enough information within this thread to assemble an httpmodule that will circumvent that Memory issue of the aspnet_wp.exe

-Before you have a useable file, you must convert the encoded file into a useable file. (This requires parsing the text file that is created.)

-An answer to UPS's one question, Microsoft didn't reply to this thread because they didnt have to; they had a team of developers figuring out a work around to their code...for free.  You have essentially done them a favor. 

-Lastly, had it not been for this thread I would still be sitting here staring at my C++ books cotemplating whether or not I really wanted to write an ISAPI Filter for IIS so for that I am greatly thankful and as such I will post my COMPLETE SOURCE CODE here within the next couple of days, however unsupported, as I do not have enough time throughout the day to answer questions.  (Some of the features that I am implementing obviously not everyone will use, but you will get the general idea)

Regards,

-Doug

_{Tuesday, March 21, 2006 5:40 PM}

While most of the thread starters have had their solution for awhile (years removed by now) I only stumbled upon this thread a week or so ago and, using this thread as reference, was able to correctly put together an HTTPModule that solves the problem proposed at the begining of the thread.

I know that my previous post promised source code, however, there is about 500 lines of code altogether but I will elaborate on some areas that were not clear to me until I got straight down into coding.  (and there may be an email address at the end of this post where you can contact me at ;])

First off, it doesn't matter which code base you use (CodeRage! or Bjorn.B posts on page 2) because you should be modifing that code anyway.  (Both code sets do what they are supposed but if you have read this entire thread the changes you should make are obvious, but ill explain.)

The text file that is created in both the above examples are nice to see what a raw HTTP Request looks like because it defines the pattern you have to match to parse out the information but the truth of the matter is, after you have this system working those files become a moot point and you can remove the FileStream variable.

There is where I spent a good amount of my time testing and working on things and, maybe someone can clarifiy something for me; after I created the text file I converted that text file back into a byte array but, for some reason, the file data was corrupted.  (Byte value 0 was replaced with byte value 32?) Before I realized this, my initial thought was to parse the text file, ya right.  Remember that asp.net sets aside 10 bytes of Data for a string variable + 2 * the string length; ok not so bad if you are working with extremely small files but when you go to parse that string, say using substring(), a copy of the original string is made so everytime you call substring another copy is made, you can see why you will run out of Memory quickly.

After I realized that I was going to have to parse this information some other way, I needed to deal with my corrupted data and had read Turok-Han's post about System.Encoding.Default.GetBytes() as he was having the same problem I messed with this class, changing between UTF, ASCII, etc but to no avail, not only was my data corrupt but my byte array length was ALWAYS shorter then what my image length was. (In one case it was larger but significantly larger) 

By this point I was begining to get frustrated and went through and re-read this thread and then, again, it was Bjorn.B that gave me a push in the right direction when he listed the classes that he had created to process his files; he process's his files a byte at a time, but this means you have to parse the entire request a byte at a time because you have no idea where the fields start and end. 

My solution to this was to combine the byte array that held the preloaded body data and the body data into one byte array (completely bypassing a stream to the text file, since my data was getting corrupted when it was converted from bytes to text and then back to bytes i had no use for that text file) and then, myself, created a "DataParser" class that had one method: ParseData that took a byte as an argument.

Up until the previous 3 paragraphs, any person having a remote knowledge of .NET could have moved around CodeRage!'s class (all you had to do was compile and go) but it was dishearting for me to read comments of people spamming this board asking how to Parse the headers.  I am all about sharing knowledge and helping someone out but there is a HUGE difference between that and simply having someone else do your work for you. 

What im trying to say is, attempt to come up with your own solution before you come asking for code, I took the code and subtle direction of this board and was able to come up with a solution, I suggest you do the same or all you will ever be is a second rate programmer who runs to Forums when they have a trivial problem. (I will admit i posted 2 threads wondering about why my data was getting corrupted (AFTER i had done everything I could think of), but I don't think that is a trivial problem)

So the piece of the puzzle that has been missing, the DataParser.

My solution to this was to work with the byte array directly (this is a point where you should my a char array and convert your entire httpRequest to a character array and write both the byte array and character array to a text file, that will show you which byte values are associated with what characters).  So what does any good parser do?  It removes the data you need from a file and gets rid of the rest, but to do this you need to find a pattern in the file you are parsing.  In our case we have an HTTP Request so, if you have run CodeRage! or Bjorn's code, open up one of those text files.

The first line will always be a series of '' (30 to be exact) followed by a series of Alpha and Numeric characters, this is your leading boundary the. (hyphen (-) has a byte value of 45 fyi) The end of each line is denoted by byte value 13 and then 10 the nearest i can guess is this is a Carriage Return and a line feed.  So that is how you determine the end of your lines and the begning and end of your boundaries.

Each subsequent line (except your actual data) after the Boundary will begin with a C (byte value 67)  e.g.: Content-Disposition or Content-Type.  The latter line will only exist if the data you are working with is an uploaded file. Each value on the Content-Disposition line is delimited by a (;)  (byte value 59)  this is how you determine what you are working with (file name, control id, content dispo) and then you parse your data from there.

Content-Type does not have a (;) but there is a space (byte value 32) between content-type and the file type so there is your delimiter to look for.

Regardless if you are working with a file or with text from a text box, there is always a line break between the data and the data information (13, 10) this is how you can tell when you start working with your data.

This is the tricky part, all of the delimiting values i have mentioned thus far (45,13,10) all will exist inside of your data so what I did was I created a loop that went through the entire data field and incrimented a integer for each pass if my loop didnt find 10 '-' together.  If the loop found 10 '-' I exited the loop and the integer that I had been incrimenting is my file length (in bytes) so I created another byte array and set its length to that of my counting integer and went through the loop one more time, this time adding bytes to the array i had just created.  And guess what?  You now have the file and its size!  At this point you could write the array to disk and save the file with appropriate extension and your done...that doesnt fit my needs as I am saving my files directly to a database.  Being that I already have the file stored in a byte array i created yet another class called DataAccess and it to contained only 1 method, proccess files.  It excepts 3 arguments, file type (string), length(integer), and data (byte).

From here it is as simple as connecting to a database and passing the information in.  Im not going to go into how to retrieve the data or what you need for that, you need to figure SOMETHING out ;]

If you have read this post in its entireity, I have all but done the work for you.  I have told you how to parse the data, what your delimiters are, and so on; the only thing I haven't done is code for you.  As I said, I think you should try to solve this problem using all the reference in this thread, you are a programmer and that is what you do: Solve problems. 

If you are unable to solve the problem or if you are just lazy, you can contact me at [email protected] and I will share my source code with you, and that is ALL I will do. I will not support it, I will not answer any questions you have about it, I will let you look at it and get an idea how it works.  It is NOT a complete solution, there is no upload progress bar there is no option to save to disk, you must adjust it accordingly. 

My hats off to the thread starters and all of the commenters, but I think this thread is offically closed. =]

-Regards,

Doug

_{Thursday, March 23, 2006 2:46 AM}

Obviously, by this point, there is a lot of information on the web about implementing this sort of solution, and we are actually using the same approach at my company, so I am not going to comment on that.  Suffice to say, this thread has proven to be very valuable.
The only problem with the code provided in this thread is that it does not work in IIS6 since the worker process used there is a subclass of ISAPIWorkerRequestInProc, so going directly to the base class gives you ISAPIWorkerRequestInProc and not ISAPIWorkerRequest.  This can easily be fixed by walking up the inheritance tree until you get to ISAPIWorkerRequest instead of blindly going to the base class.

One problem that I had in my implementation specifically, was ReadEntityBody() returning 0 after a timeout under certain conditions.  I've spent quite a bit of time trying to find the solution in various search engines, but unfortunately left empty-handed.  It seems that a lot of people were having the same problem, but all of their posts either went unanswered or were answered with complete BS.  Two particular replies come to mind.  One is from an MVP who said "That's Internet for You..." and another from MS Support who said "Use Request.Files instead".  Very helpful indeed :)  I am sure that people who were having these problems eventually resolved them, but nobody bothered to post the answer.  And it would have saved me 3 hours...  Oh well.  Hopefully, I can clear things up a little, so that the next poor soul does not have to repeat it.

The short version:

Make sure you are not referencing any FORM variables before you attempt to read the entity body.

The long version:

Simply put, this is a bug in the implementation and not in the .NET Framework / IIS.  The WorkerProcess is not meant to be accessed from IHttpModule/HttpContext.  It is marked internal for a reason.  By using reflection go get access to an inaccessible property, we open up the Pandora's Box for problems like this.  Unfortunately, in ASP.Net 1.0/1.1 this seems to be the only option.

This behavior occurs if you access FORM variables on the HttpRequest before attempting to call ReadEntityBody().  When the form is submitted using "multipart/form-data", all FORM variables contained in the message body - the FORM field you are querying might well be defined AFTER the file contents in the HTTP post body.  So, in order to be able to read the FORM field value, it has to load the entire request body first.  The getter for HttpRequest.Form initializes the field values collection by calling FillInFormCollection() which in turn calls GetMultipartContent():

1. private MultipartContentElement[] GetMultipartContent()
2. {
3.       if (this._multipartContentElements == null)
4.       {
5.             byte[] buffer1 = this.GetMultipartBoundary();
6.             if (buffer1 == null)
7.             {
8.                   return new MultipartContentElement[0];
9.             }
10.             byte[] buffer2 = this.GetEntireRawContent();
11.             if (buffer2 == null)
12.             {
13.                   return new MultipartContentElement[0];
14.             }
15.             this._multipartContentElements = HttpMultipartContentTemplateParser.Parse(buffer2, buffer2.Length, buffer1, this.ContentEncoding);
16.       }
17.       return this._multipartContentElements;
18. }

When this method is called for the first time, it initializes _multipartContentElements by calling GetEntireRawContent() which does exactly what we are doing in our file upload module - it reads the rest of HTTP post body using ReadEntityBody().  The difference is that since GetEntireRawContent() does not reset WorkerProcess's _contentAvailLength, _contentTotalLength, _preloadedContent and _preloadedContentRead (nor should it), the next time we call ReadEntityBody(), there's nothing left to read in the pipeline!

Hope this helps,
Michael Leibman
Lead Software Developer
eProject Inc.
http://www.eproject.com

_{Thursday, March 23, 2006 2:28 PM}

NOTE:  This also applies to Request["..."] and Request.Params["..."] syntax.

_{Thursday, March 23, 2006 3:02 PM}

One more thing.  You also need to turn TRACING off in the web.config since it essentially reads all request fields itself.

_{Wednesday, April 12, 2006 8:55 PM}

i can get an instance of the HttpWorkerRequest fine when the app is running on Full trust, but when its running at High or Medium i get a security error.

anyone figured out how to get it working with High/Medium trust?

_{Wednesday, June 21, 2006 12:46 AM}

Hi MaduraiKaran and all .net friends,

Here is my code for resolving the HTTP Head Filtering issue. I've test edit under VS2003 and just for your reference:

Imports System
Imports System.IO
Imports System.Web

Public Class UploadModule

    Implements IHttpModule

    Public Sub New()
        MyBase.New()
    End Sub

    Public Sub Dispose() Implements System.Web.IHttpModule.Dispose
    End Sub

    Public Sub Init(ByVal context As System.Web.HttpApplication) Implements System.Web.IHttpModule.Init
        AddHandler context.BeginRequest, AddressOf BeginRequest
    End Sub

    Private Sub BeginRequest(ByVal source As Object, ByVal e As EventArgs)

        'Get new Upload Data Analysis
        Dim FAnalysisUploadData As New AnalysisUploadData

        'Get Received Data
        Dim HttpApp As HttpApplication = CType(source, HttpApplication)
        Dim theHttpContext As HttpContext = HttpApp.Context

        If InStr(theHttpContext.Request.ContentType, "multipart/form-data") = 0 Then
            Exit Sub
        End If

        ' Get the HttpWorkerRequest Object!!!
        Dim HttpWR As HttpWorkerRequest = CType(theHttpContext.GetType.GetProperty("WorkerRequest", Reflection.BindingFlags.Instance Or Reflection.BindingFlags.NonPublic).GetValue(theHttpContext, Nothing), HttpWorkerRequest)

        If HttpWR.HasEntityBody Then

            Try

                'Const DefaultBufferLength As Integer = 65536
                Const DefaultBufferLength As Integer = 1024 * 2

                Dim Buffer As Byte()
                Dim Received As Integer = 0
                Dim TotalReceived As Integer = 0
                Dim ContentLength As Integer = CType(HttpWR.GetKnownRequestHeader(HttpWorkerRequest.HeaderContentLength), Integer)

                Buffer = HttpWR.GetPreloadedEntityBody()

                FAnalysisUploadData.AnalysisReceivedData(Buffer, Buffer.Length)

                Received = Buffer.Length
                TotalReceived += Received

                If Not HttpWR.IsEntireEntityBodyIsPreloaded Then

                    'Read Data in DefaultBufferLength size
                    While (ContentLength - TotalReceived) >= DefaultBufferLength
                        Buffer = New Byte(DefaultBufferLength) {}
                        Received = HttpWR.ReadEntityBody(Buffer, DefaultBufferLength)
                        FAnalysisUploadData.AnalysisReceivedData(Buffer, Received)
                        TotalReceived += Received
                    End While

                    'Read the Remain Data
                    Buffer = New Byte(DefaultBufferLength) {}
                    Received = HttpWR.ReadEntityBody(Buffer, (ContentLength - TotalReceived))
                    FAnalysisUploadData.AnalysisReceivedData(Buffer, Received)
                    TotalReceived += Received
                End If

                'Finish AnalysisRecievedData
                Buffer = New Byte(DefaultBufferLength) {}
                FAnalysisUploadData.UploadSplitHeadSize = 0
                FAnalysisUploadData.AnalysisReceivedData(Buffer, DefaultBufferLength + 1)

            Catch ex As Exception

            Finally
                FAnalysisUploadData.CloseOpenFiles()
            End Try

            theHttpContext.Application("Result") = FAnalysisUploadData.LogStr

            theHttpContext.Response.Redirect("uploadform.aspx")

        End If

    End Sub

End Class

Public Class AnalysisUploadData
    Public LogStr As String

    Dim AnalysisCount As Integer = 0
    Dim AnalysisByte() As Byte
    Dim IsFileOperatingNow As Boolean = False

    Dim UploadSessionID As String
    Dim UploadSplitChar, UploadFileBeginChar, UploadEndChar As String
    Public UploadSplitHeadSize As Integer = 300

    Dim FileName As String
    Dim FilePath As String
    Dim theFileStream As FileStream

    Public Sub AnalysisReceivedData(ByVal ReceivedByte() As Byte, ByVal ReceivedByteNumber As Integer)

        Dim I As Integer

        AnalysisCount = AnalysisCount + 1
        If AnalysisCount = 1 Then
            'Get UploadSessionID
            For I = 29 To 29 + 12
                UploadSessionID = UploadSessionID & Chr(ReceivedByte(I))
            Next

            'Get UploadSplitChar
            UploadSplitChar = "" & UploadSessionID

            'Get UploadEndChar
            UploadEndChar = "" & UploadSessionID & "--"

        End If

        'Get the AnalysisByte and AnalysisStr
        Dim PreviousAnalysisByteLength As Integer = 0
        Try
            PreviousAnalysisByteLength = AnalysisByte.Length
        Catch ex As Exception
        End Try

        ReDim Preserve AnalysisByte(PreviousAnalysisByteLength + ReceivedByteNumber - 1)
        For I = PreviousAnalysisByteLength To PreviousAnalysisByteLength + ReceivedByteNumber - 1
            AnalysisByte(I) = ReceivedByte(I - PreviousAnalysisByteLength)
        Next
        Dim AnalysisStr As String
        Dim StandAnalysisStrLength As Integer = AnalysisByte.Length
        For I = 0 To StandAnalysisStrLength - 1
            AnalysisStr = AnalysisStr & Chr(AnalysisByte(I))
        Next

        'Read and analysis each UploadSplitChar one by one
        Dim IfNeedBreakDoWhile As Boolean

        Dim thePos As Integer
        thePos = InStr(AnalysisStr, UploadSplitChar)

        If thePos > 0 Then

            Do While (thePos > 0)

                'Finish Previous Operation
                If IsFileOperatingNow Then
                    'Write the remain byte and close the file
                    theFileStream.Write(AnalysisByte, 0, thePos - 1)
                    theFileStream.Flush()
                    theFileStream.Close()

                    IsFileOperatingNow = False
                Else
                    'Ignore the data
                End If

                'Make sure the split str's start pos is not in AnalysisTail
                If thePos >= Len(AnalysisStr) - UploadSplitHeadSize Then
                    ResizeToRightByte(AnalysisByte, Len(AnalysisStr) - thePos + 1)
                    Exit Do
                End If

                'Check whether the New Split is a file
                AnalysisStr = Right(AnalysisStr, Len(AnalysisStr) - thePos + 1)

                If IsItAnUploadFile(AnalysisStr, FileName) Then
                    FilePath = "c:\upload\ & FileName

                    'Delete the current exist file
                    If System.IO.File.Exists(FilePath) Then
                        System.IO.File.Delete(FilePath)
                    End If

                    'Create FileStream Object
                    theFileStream = New FileStream(FilePath, FileMode.CreateNew, FileAccess.Write, FileShare.Read)
                    IsFileOperatingNow = True

                    'Write File
                    PutDataToFile(AnalysisStr, StandAnalysisStrLength, IfNeedBreakDoWhile)

                    'Judge whether need to exit do, due cannot find the file end when writting the file data
                    If IfNeedBreakDoWhile Then
                        Exit Do
                    End If
                Else
                    'Remove the upload split head
                    AnalysisStr = Right(AnalysisStr, Len(AnalysisStr) - 50)
                End If

                thePos = InStr(AnalysisStr, UploadSplitChar)

            Loop
        Else
            If IsFileOperatingNow Then
                'Write Analysis Byte and Leave 50 bytes

                theFileStream.Write(AnalysisByte, 0, StandAnalysisStrLength - 50)

                ResizeToRightByte(AnalysisByte, 50)
                IsFileOperatingNow = True
            Else
                'Ignore Last Received Byte because it's not file data
            End If
        End If

    End Sub

    Public Sub CloseOpenFiles()

        If IsFileOperatingNow Then
            theFileStream.Flush()
            theFileStream.Close()

            IsFileOperatingNow = False
        End If

        theFileStream = Nothing

    End Sub

    Private Function IsItAnUploadFile(ByVal HeadStr As String, ByRef FileName As String) As Boolean
        FileName = ""

        If Len(HeadStr) > 300 Then HeadStr = Left(HeadStr, 300)

        Dim thePos As Integer
        thePos = InStr(HeadStr, "filename=""")

        If thePos = 0 Or thePos > 110 Then
            Return False
        End If

        HeadStr = Right(HeadStr, Len(HeadStr) - thePos - 9)

        thePos = InStr(HeadStr, """")
        FileName = Left(HeadStr, thePos - 1)
        thePos = InStrRev(FileName, "\)

        FileName = Right(FileName, Len(FileName) - thePos)

        Return True

    End Function

    Private Sub ResizeToRightByte(ByRef theByte() As Byte, ByVal RightSize As Integer)

        Dim ByteLength As Integer
        ByteLength = theByte.Length

        If RightSize >= ByteLength Then Return

        Dim I As Integer
        For I = 0 To RightSize - 1
            theByte(I) = theByte(ByteLength - RightSize + I)
        Next

        ReDim Preserve theByte(RightSize - 1)

    End Sub

    Private Sub PutDataToFile(ByRef AnalysisStr As String, ByVal StandAnalysisStrLength As Integer, ByRef NeedBreakDoWhile As Boolean)

        NeedBreakDoWhile = False

        Dim DataStartID, DataEndID As Integer

        Dim thePos As Integer
        thePos = InStr(AnalysisStr, Chr(13) & Chr(10) & Chr(13) & Chr(10))

        If thePos > 0 Then

            AnalysisStr = Right(AnalysisStr, Len(AnalysisStr) - thePos - 3)
            DataStartID = StandAnalysisStrLength - Len(AnalysisStr)

            thePos = InStr(AnalysisStr, UploadSplitChar)

            'LogStr = LogStr & "AnalysisStr=" & AnalysisStr & vbCrLf & "

" & vbCrLf

            If thePos > 0 Then
                'Found the File End
                AnalysisStr = Right(AnalysisStr, Len(AnalysisStr) - thePos + 3)
                DataEndID = StandAnalysisStrLength - Len(AnalysisStr) - 1

                theFileStream.Write(AnalysisByte, DataStartID, DataEndID - DataStartID + 1)
                theFileStream.Flush()
                theFileStream.Close()

                IsFileOperatingNow = False
                NeedBreakDoWhile = False
            Else
                'Did not find the file end, leave 50 length for AnaysisByte and exit do
                '  "7d6ab36c03d2" length is 41
                DataEndID = StandAnalysisStrLength - 50 - 1

                theFileStream.Write(AnalysisByte, DataStartID, DataEndID - DataStartID + 1)

                ResizeToRightByte(AnalysisByte, 50)
                IsFileOperatingNow = True
                NeedBreakDoWhile = True
            End If

        End If

    End Sub
End Class

_{Tuesday, July 11, 2006 11:40 PM}

I don’t want to crack on anyone in this thread. It has been extremely good reading knowledge.  However, this last example, while it was nice is extremely slow. Why people like to convert byte arrays to strings and do comparisons I have no idea.  Not only does it take a crap load of CPU, it posts files are retarded slow speeds. There are other problems with the above example I do not care to comment on, but can cause file corruption.

I rewrote this using much more efficient byte array searching, including the ability to look at the last buffer to see if patterns exist with the two byte array buffers combined.  It works REALLY nice and have wrote a progress indicator for it too. 

I plan on writing a full article on it and referencing everyone that contributed to this.  If you would like my "FREE" software, I will email you a zip of it.  I’m not going to go and sell this just because I was able to get it to work.   I’m still doing more testing on it, but I have been able to upload huge files with no errors at all on my production server

Email me at travis at lvfbody.com and I will get you what I have so far.  Please mention something in the subject about my “File Upload Module”. Oh, and I believe in code comments so other people can see what the heck I am doing.

Like I said, I am not trying to bash on anyone, but many people on this forum and others just are doing things so inefficiently.  If you can help improve my code, that would be great. I look forward to writing something up to put online.

Oh yea, I developed this all in .net 2.0. If you want to use it with 1.1, you will have to change some things in it.  Should be minor.  Thanks again to everyone how has contributed to this literally over the years!

_{Sunday, July 16, 2006 10:31 PM}

Sorry it took so long for me to get this up.  I wrote up a project on codeproject.com

http://www.codeproject.com/useritems/UpldFileToDiskProgressBar.asp

I look forward to your input.  Enjoy everyone and thanks again for everyone help.

_{Thursday, September 28, 2006 5:00 PM}

For those who can't use this technique because their hosting service is less than full trust (Reflection is not allowed), I wrote a tutorial on how to use flash to upload multiple files to a HttpHandler.

http://www.codeproject.com/useritems/FlashUpload.asp

It is pretty simple.  Something that TravisWhidden pointed out was that in the HttpPostedFile class definition http://msdn.microsoft.com/en-us/library/system.web.httppostedfile.aspx it says "By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk, rather than held in server memory."  So the code basically just shows how to use the flash and asp.net together to accomplish a nice uploader interface. Hope this is useful. And thanks for all the great contributions.

_{Friday, May 4, 2007 2:48 AM}

Hi Guys,

 
Do you know any solution that can use FTP in ActiveX control & upload the file in ASP.NET?

_{Tuesday, May 8, 2007 5:07 AM}

I have a page there i want to upload a csv file, and the content will be put into database.

now if that file is 10-20 MB+, it takes lots of time, plus some time request timeout occurs.

how to address this problem ?

_{Friday, August 10, 2007 12:00 PM}

 On a funny side, i've got it working (teste with file up to 2gig) with the normal file upload component in 2.0 by setting a few thing before uploading the file.

for infos about these Property:

http://msdn.microsoft.com/en-us/library/e1f13641.aspx

maxRequestLength="2000"

executionTimeout="90"

shutdownTimeout="90"

requestLengthDiskThreshold="256"

I simply modify them before uploading and change it back after. 

It sure not as nice as other solutions and offer no progress bar but just wanted to share the easy fix.

Was hella nice to read all this topic too. Thx to all who shared.  

_{Thursday, December 4, 2008 9:13 AM}

Hi,

     I need to upload large files like  > 1 gb .I am new to ASP.NET. I want to use HttpModule with Chunk method . Any one please post the full source code or URL info.

     I am expecting your replay.

Regards

Ram