Coversion from filestream to memorystream slow

Question

_{Thursday, March 15, 2018 5:57 PM}

I am trying to convert filestream to memorystream using the below code which is taking too much of time.

 {
      byte[] buffer = new byte[1024 * 64];
      int nread = 0;
      while ((nread = fileInfo.Stream.Read(buffer, 0, buffer.Length)) > 0)
      {
          memory.Write(buffer, 0, nread);
      }
      memory.Seek(0, SeekOrigin.Begin);
      docByteArray = memory.ToArray();
      docBase64String = Convert.ToBase64String(docByteArray);
  }

I get the filestream from SharePoint using csom. Below is the code sample:

 fileInfo = Microsoft.SharePoint.Client.File.OpenBinaryDirect(spContext, fileRef.ToString());

Any suggestions on how to make it master.

All replies (2)

_{Thursday, March 15, 2018 6:13 PM}

You are reading 64K bytes at a time. Depending upon the size of the file this could take a long time. On top of that you're using a memory stream which is going to be allocating an ever larger array itself. If you need to copy one stream to another then consider using CopyTo instead. Depending upon the source stream this may take a while anyway. If you look part way down the SO posting here then you'll see an extension method that does basically this.

Why are you copying the file into memory to begin with? It looks like you're trying to base64 encode the file but oftentimes that isn't necessary. You didn't mention what you intend to do with the array once you're done but most calls that support base64 arrays also have an overload for streams. Have you looked to see if one is available? 

Michael Taylor http://www.michaeltaylorp3.net

_{Friday, March 16, 2018 10:11 AM}

Your memory-mapped file might be getting written to disk. I believe that memory-mapped files that are not backed by a permanent disk file are backed by the pagefile; the same file used for virtual storage. If the system it is executing in has insufficient physical memory then it will use the pagefile. That can interfere with the performance of the rest of the system too; your system might be "thrashing". You can try using the performance analyzer. There are many ways ("counters"?) to look at virtual memory and I don't know the details of which one to look at to determine if the system is affected by paging caused by the memory-mapped file.

Back in the day when Windows was very young I was an application developer using IBM mainframe computers. I thought it was good to use virtual storage for temporary storage. The technical support people sure did not like that but they never really explained to me what the problem was. If the performance of your application is degraded by the use of virtual storage (indirectly by using the memory-mapped file) then you can see that happening with the performance analyzer.

Sam Hobbs
SimpleSamples.Info