30

I'm currently working on some search and replace operation that I'm trying to automate using powershell. Unfortunately I recognized yesterday that we've different file encodings in our codebase (UTF8 and ASCII). Because we're doing these search and replace operations in a different branch I can't change the file encodings at this stage.

If I'm running the following lines it changes all files to UCS-2 Little Eindian even though my default powershell encoding is set to iso-8859-1 (Western European (Windows)).

$content = Get-Content $_.Path
$content -replace 'myOldText' , 'myNewText' | Out-File $_.Path

Is there a way to prevent powershell from changing the file's encoding?

1 Answer 1

40

Out-File has a default encoding unless overriden with the -Encoding parameter:

What I've done to solve this is to try to get the original file's encoding by reading trying to read it's byte order mark and using it as the-Encoding parameter value.

Here's an example processing a bunch of text file paths, getting the original encoding, processing the content and writing it back to file with the original's encoding.

function Get-FileEncoding {
    param ( [string] $FilePath )

    [byte[]] $byte = get-content -Encoding byte -ReadCount 4 -TotalCount 4 -Path $FilePath

    if ( $byte[0] -eq 0xef -and $byte[1] -eq 0xbb -and $byte[2] -eq 0xbf )
        { $encoding = 'UTF8' }  
    elseif ($byte[0] -eq 0xfe -and $byte[1] -eq 0xff)
        { $encoding = 'BigEndianUnicode' }
    elseif ($byte[0] -eq 0xff -and $byte[1] -eq 0xfe)
         { $encoding = 'Unicode' }
    elseif ($byte[0] -eq 0 -and $byte[1] -eq 0 -and $byte[2] -eq 0xfe -and $byte[3] -eq 0xff)
        { $encoding = 'UTF32' }
    elseif ($byte[0] -eq 0x2b -and $byte[1] -eq 0x2f -and $byte[2] -eq 0x76)
        { $encoding = 'UTF7'}
    else
        { $encoding = 'ASCII' }
    return $encoding
}

foreach ($textFile in $textFiles) {
    $encoding = Get-FileEncoding $textFile
    $content = Get-Content -Encoding $encoding
    # Process content here...
    $content | Set-Content -Path $textFile -Encoding $encoding
}

Update Here is an example of getting the original file encoding using the StreamReader class. The example reads the first 3 bytes of the file so that the CurrentEncoding property gets set based on the result of its internal BOM detection routine.

http://msdn.microsoft.com/en-us/library/9y86s1a9.aspx

The detectEncodingFromByteOrderMarks parameter detects the encoding by looking at the first three bytes of the stream. It automatically recognizes UTF-8, little-endian Unicode, and big-endian Unicode text if the file starts with the appropriate byte order marks. Otherwise, the UTF8Encoding is used. See the Encoding.GetPreamble method for more information.

http://msdn.microsoft.com/en-us/library/system.text.encoding.getpreamble.aspx

$text = @" 
This is
my text file
contents.
"@

#Create text file.
[IO.File]::WriteAllText($filePath, $text, [System.Text.Encoding]::BigEndianUnicode)

#Create a stream reader to get the file's encoding and contents.
$sr = New-Object System.IO.StreamReader($filePath, $true)
[char[]] $buffer = new-object char[] 3
$sr.Read($buffer, 0, 3)  
$encoding = $sr.CurrentEncoding
$sr.Close()

#Show the detected encoding.
$encoding

#Update the file contents.
$content = [IO.File]::ReadAllText($filePath, $encoding)
$content2 = $content -replace "my" , "your"

#Save the updated contents to file.
[IO.File]::WriteAllText($filePath, $content2, $encoding)

#Display the result.
Get-Content $filePath
3
  • I already thought about it but there must be a easier way, mustn't it? But that works for me now. Thanks Andy!
    – Pete
    Commented Feb 3, 2012 at 3:34
  • 2
    @Pete You're going to have to get the encoding. There's no cmdlet that will get it for you. I updated my answer adding a different approach. Both ways use BOM detection. Commented Feb 3, 2012 at 5:20
  • Set-Content -Path BOM_Utf32.txt -Value $null -Encoding UTF32 writes UTF-32, little-endian BOM i.e. FF FE 00 00 byte sequence. However, function Get-FileEncoding returns Unicode. On the other hand, 00 00 FE FF byte sequence is recognized as UTF32 but as per The Unicode Consortium this is UTF-32, big-endian BOM. Am I wrong? Where's the error?
    – JosefZ
    Commented May 5, 2016 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.