Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Saturday, June 28, 2014 4:12 AM
Suppose the following lines in text file to which i have to read
INFO 2014-03-31 00:26:57,829 332024549ms Service1 startmethod - FillPropertyColor end
INFO 2014-03-31 00:26:57,829 332024549ms Service1 getReports_Dataset - getReports_Dataset started
INFO 2014-03-31 00:26:57,829 332024549ms Service1 cheduledGeneration - SwitchSchedu
INFO 2014-03-31 00:26:57,829 332024549ms Service1 cheduledGeneration - SwitchScheduledGeneration limitId,
I use the FileStream method to read the text file because the text file size having size over 1 GB. I have to read the files into chunks like initially
in first run of program this would read two lines i.e up to "getReports_Dataset started" of second line. In next run it should read from 3rd line. I did the code but unable to get desired output.Problem is that my code
doesn't give the exact chunk from where i have to start read text in next run. And second problem is while reading text lines .. don't give a complete line..i.e. some part is missing in lines.
int lastReadPosition = getLastReadPosition();
using (FileStream fStream = new FileStream(logFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (System.IO.StreamReader rdr = new System.IO.StreamReader(fStream))
{
rdr.BaseStream.Seek(readPosition, SeekOrigin.Begin);
while (numCharCount > 0)
{
int numChars = rdr.ReadBlock(block, 0, block.Length);
string blockString = new string(block);
lines = blockString.Split(Convert.ToChar('\r'));
lines[0] = fragment + lines[0];
fragment = lines[lines.Length - 1];
foreach (string line in lines)
{
lstTextLog.Add(line);
if (lstTextLog.Contains(fragment))
{
lstTextLog.Remove(fragment);
}
numProcessedChar++;
}
numCharCount--;
}
SetLastPosition(numProcessedChar, logFilePath);
}
All replies (7)
Sunday, June 29, 2014 8:39 AM âś…Answered
If the file is fixed. i.e. no other lines are added by your or other processes, then you can also try the enumerator returned by File.ReadLines:
IEnumerable<string> lines = File.ReadLines( logFilePath );
IEnumerator<string> en = lines.GetEnumerator();
Then you can use en.MoveNext and en.Current on different stages of your program. The current position is remembered by the enumerator.
Saturday, June 28, 2014 12:41 PM
Hello,
This forum is for discussions and questions regarding profiles and Microsoft's recognition system on the MSDN and TechNet sites. It is not for products/technologies.
As it's off-topic here, I am moving the question to the Where is the forum for... forum.
Karl
When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
My Blog: Unlock PowerShell
My Book: Windows PowerShell 2.0 Bible
My E-mail: -join ('6F6C646B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})
Saturday, June 28, 2014 12:41 PM
Hello,
Please ask in the Visual C# Language forum on MSDN.
Karl
When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
My Blog: Unlock PowerShell
My Book: Windows PowerShell 2.0 Bible
My E-mail: -join ('6F6C646B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})
Saturday, June 28, 2014 3:50 PM
It's not clear why you are reading in chunks rather than, for example, using a StreamReader.ReadLine to get full lines. You could have an array (or list) of strings and read say 10 lines at a time then process them. But I don't know what the processing is intended to do - you don't say what you are trying to do.
If you read blocks then unless all lines are the same length you will have to deal with partial lines. Is that what fragment is supposed to handle? That's not clear from the code posted.
Regards David R
The great thing about Object Oriented code is that it can make small, simple problems look like large, complex ones.
Object-oriented programming offers a sustainable way to write spaghetti code. - Paul Graham.
Every program eventually becomes rococo, and then rubble. - Alan Perlis
The only valid measurement of code quality: WTFs/minute.
Saturday, June 28, 2014 10:02 PM | 1 vote
The ReadBlock has to be before the while loop, not after. See the change below.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
string logFilePath = "";
long lastReadPosition = 0;
long readPosition = 0;
char[] block = new char[1024];
long numCharCount = 0;
string[] lines;
string fragment = "";
List<string> lstTextLog = new List<string>();
long numProcessedChar = 0;
using (FileStream fStream = new FileStream(logFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (System.IO.StreamReader rdr = new System.IO.StreamReader(fStream))
{
rdr.BaseStream.Seek(readPosition, SeekOrigin.Begin);
int numChars = rdr.ReadBlock(block, 0, block.Length);
while (numCharCount > 0)
{
string blockString = new string(block);
lines = blockString.Split(Convert.ToChar('\r'));
lines[0] = fragment + lines[0];
fragment = lines[lines.Length - 1];
foreach (string line in lines)
{
lstTextLog.Add(line);
if (lstTextLog.Contains(fragment))
{
lstTextLog.Remove(fragment);
}
numProcessedChar++;
}
numCharCount--;
}
//SetLastPosition(numProcessedChar, logFilePath);
}
}
}
}
jdweng
Sunday, June 29, 2014 4:33 PM
@Riced: I REALLY like the line in your signature: "The only valid measurement of code quality: WTFs/minute.*" *
Boy! If only I coulda had a nickel for every time someone said that about MY code ...
Wednesday, November 9, 2016 3:38 AM
This blog entry has a great solution that incorporates parallel-processing as well:
http://cc.davelozinski.com/code/c-sharp-code/read-lines-in-batches-process-in-parallel