How to replace char in 2GB Text file with Powershell?

Question

_{Thursday, June 12, 2014 5:36 AM}

I would like to replace all "^" char with ";" so that File get correct CSV format.

File size is 2GB. Visual Studio was not able to open the file, but could not process it.

I was able to replace only one "^" char per row with following script. Every single row do have such char 10 times.

I need help so that every single"^" char will be replaced.

(Get-Content FileName.txt) | ForEach-Object { $_ -replace "^", "," } | Set-Content FileName2.csv

Kenny_I

All replies (7)

_{Thursday, June 12, 2014 11:26 AM ✅Answered | 2 votes}

Try this:

$InFile = 'Filename.txt'
$OutFile = 'Filename2.txt'

filter replace-chars { $_ -replace '\^',';' }

if (test-path $OutFile)
 { Clear-Content $OutFile }

Get-Content $InFile -ReadCount 1000 | 
 replace-chars  | 
 Add-Content $OutFile

The carat (^) is reserved in regex and needs to be escaped to do a literal match.

That will process the file 1000 lines at a time (rather than 1).   That should be much faster than processing one line at a time,  and still keep your memory consumption under control.

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Friday, June 13, 2014 7:08 AM}

I know this is the PowerShell forum, but why don't you just use Notepad++ for this? Notepad++ is easily able to process a 2GB file and replace all your ^ with ; (just put the cursor at the first character, press ctrl + h, fill in the variables, press ok and... tadaa).

PS. Don't get me wrong, I love PowerShell, but easy things shouldn't be made more difficult ;)

_{Friday, June 13, 2014 10:42 AM | 1 vote}

If I understand correctly, you're just looking to change the delimiter in what is already a 'csv' file, if that is correct, you could try this

Import-Csv -Delimiter '^' -Path FileName.txt | Export-Csv -Path FileName2.csv -Delimiter ',' -NoTypeInformation

Note that this may make your file bigger as it will quote each entry, but doesn't have the problem of a simple text replace which may break the data if there are commas in any existing unquoted fields.

_{Friday, June 13, 2014 1:10 PM}

The answer is still simple: because the PS coder got confused when using a parameter ( -ReadCount ) he doesn't understand well.

Care to speculate on why someone who claims to understand -Reacount would propose a solution that uses the default of 1 on a 1.5GB file, knowing it's going to make it horribly slow?

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Saturday, June 14, 2014 4:13 AM}

If I understand correctly, you're just looking to change the delimiter in what is already a 'csv' file, if that is correct, you could try this

Import-Csv -Delimiter '^' -Path FileName.txt | Export-Csv -Path FileName2.csv -Delimiter ',' -NoTypeInformation

Note that this may make your file bigger as it will quote each entry, but doesn't have the problem of a simple text replace which may break the data if there are commas in any existing unquoted fields.

another reason for this approach over a simple textual replace should be obivious; what if the original file contains quoted fields containing a "^" character?

Al Dunbar -- remember to 'mark or propose as answer' or 'vote as helpful' as appropriate.

_{Sunday, June 15, 2014 1:52 AM}

If I understand correctly, you're just looking to change the delimiter in what is already a 'csv' file, if that is correct, you could try this

Import-Csv -Delimiter '^' -Path FileName.txt | Export-Csv -Path FileName2.csv -Delimiter ',' -NoTypeInformation

Note that this may make your file bigger as it will quote each entry, but doesn't have the problem of a simple text replace which may break the data if there are commas in any existing unquoted fields.

another reason for this approach over a simple textual replace should be obivious; what if the original file contains quoted fields containing a "^" character?

Al Dunbar -- remember to 'mark or propose as answer' or 'vote as helpful' as appropriate.

Text replacement using the -replace operator is a regex operation.  The first rule of regex is "know your data".  If you don't know whether a particular pattern may or may not appear in the data then you probably shouldn't be using a regex.  OTOH, if you do know what the possible patterns will be, then you can take advantage of using -replace in combination with -ReadCount, which makes the -replace an array operator, eliminating the iteration through the individual members of each ReadCount array.

[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

_{Monday, July 7, 2014 10:22 AM}

Hi Kenny_I,

I’m writing to just check in to see if the suggestions were helpful. If you need further help, please feel free to reply this post directly so we will be notified to follow it up.

If you have any feedback on our support, please click here.

Best Regards,

Anna Wang

TechNet Community Support