Exception of type 'System.OutOfMemoryException' was thrown.stack: at System.Text.StringBuilder.ToString() for big file

Question

_{Wednesday, August 12, 2015 5:29 PM}

I have very big file with size bigger than 200 MB. I need to manipulate some data in file, moctly string comparison between two files.

During that I get above exception 

 Exception of type 'System.OutOfMemoryException' was thrown.stack:   at System.Text.StringBuilder.ToString()

Can somebody explain me to better way to write code so I will not get this error in future. This is very urgent.

Piece of code I am using 

 using (StreamReader sReader = new StreamReader(newFilePath))
            {
                char[] ediChar = null;

                while (sReader.Peek() >= 0)
                {
                    ediChar = new char[4096];
                    sReader.Read(ediChar, 0, ediChar.Length);
                    sb.Append(ediChar);
                }

**    message = sb.ToString();**
                //char SegDelimiter = message[105];

I am getting error at line **  message = sb.ToString();**

All replies (11)

_{Wednesday, August 12, 2015 8:32 PM ✅Answered}

It is this

In short... its not for huge data (meaning it has another purpose), but its in virtual memory... so the limitations differ (if you load it into memory again (via StringBuilder etc.) it wont make a difference of course).

It is a way of sharing files with multiple applications/processes/threads

But i cannot help you there... cause i never had to use it.

Edit: Aslo qouting your code (from the new Exception you get in "GetSHA512") 

"StringBuilder sb = new StringBuilder(1024 * 1024 * 320);" you are using Capacity ;-)

I don't know your purpose of the code, but if you need to merge or compare files, maybe you should do it line by line (with StreamReader.ReadLine()) and work out some algorithm to compare or merge or whatever.

Edit2:

I just had another thought, maybe you can make use of LevelsteinDistance for your Caomparisment (if you do Comparisment) or just a hashing of the lines, or even both, it would reduce the amount of text you need to keep in memory

_{Wednesday, August 12, 2015 10:17 PM ✅Answered}

I added this to test removed and made 64 bit and looks like it bypassed that wrror

_{Wednesday, August 12, 2015 5:43 PM}

Hello,

Work Around would be to split the string and operate on it.

just take the substring of your string builder, :)

let me know if this works for you.

_{Wednesday, August 12, 2015 5:52 PM}

I agree ... but it will change my whole process. As this component is written to compare two files and working well till now. So I am looking some easier option something like allocate max memory for this operation . So my existing production vendors will not get impact.

   Please let me know if any other better solution, than substring.

_{Wednesday, August 12, 2015 6:42 PM}

HI,Each character requires 2 bytes (as a char in .NET is a UTF-16 code unit). So by the time you've reached 800 million characters, that's 1.6GB of contiguous memory required1. Now when the StringBuilder needs to resize itself, it has to create another array of the new size (which I believe tries to double the capacity) - which means trying to allocate a 3.2GB array.

I believe that the CLR (even on 64-bit systems) can't allocate a single object of more than 2GB in size. (That certainly used to be the case.) My guess is that your StringBuilder is trying to double in size, and blowing that limit

_{Wednesday, August 12, 2015 7:14 PM}

Yes it could be as I said other places it worked like till 150 MB file I tested. So what could be solution other than breking files, as it change process for me 

_{Wednesday, August 12, 2015 7:19 PM}

Do you receive any error if you read the string using a single line: message = File.ReadAllText( newFilePath )?

_{Wednesday, August 12, 2015 7:27 PM}

I did not try to you think it will efficient?

I just try to change my code like 

 StringBuilder sb = new StringBuilder(1024 * 1024 * 300);
            string message = null;

            using (StreamReader sReader = new StreamReader(newFilePath))
            {
                char[] ediChar = null;

                while (sReader.Peek() >= 0)
                {
                    ediChar = new char[4096];
                    sReader.Read(ediChar, 0, ediChar.Length);
                    sb.Append(ediChar);
                }

                message = sb.ToString();

Now I am not getting error but I am getting error 

:Exception of type 'System.OutOfMemoryException' was thrown.stack:   at System.Text.StringBuilder..ctor(String value, Int32 startIndex, Int32 length, Int32 capacity)

at following code, can somebody help what and where it exactly.

  internal static string GetSHA512(string text)
        {
            UnicodeEncoding UE = new UnicodeEncoding();
            byte[] hashValue;
            byte[] message = UE.GetBytes(text);

            SHA512Managed hashString = new SHA512Managed();
            StringBuilder sb = new StringBuilder(1024 * 1024 * 320);
            string hex = "";

            hashValue = hashString.ComputeHash(message);
            foreach (byte x in hashValue)
            {
              
                sb.Append(String.Format("{0:x2}", x));
            }

            hex = sb.ToString();
            return hex;
        }

_{Wednesday, August 12, 2015 8:09 PM}

I am no expert in this kinda stuff (big memory loads and such) but mabe the direction will be enough:

Try using Memory Mapped Files for work that has to hold loads of data.

Try using Streams whenever possible (so the data does not have to be hold in memory)

Try creating the StringBuilder without a capacity (empty constructor, let it do what it does best)

_{Wednesday, August 12, 2015 8:25 PM}

Till now I was using stringbuilder without any capacity and get this error.

I am using string only when I need to compare data, and I can not avoid it . I thing there it self it start blowing.

BTW what is 

 Memory Mapped Files for work that has to hold loads of data?

_{Thursday, August 13, 2015 7:50 AM}

Please mark helpful posts and select the answere that worked best for you