391

Out-File seems to force the BOM when using UTF-8:

$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "UTF8" $MyPath

How can I write a file in UTF-8 with no BOM using PowerShell?

Update 2021

PowerShell has changed a bit since I wrote this question 10 years ago. Check multiple answers below, they have a lot of good information!

6
  • 37
    BOM = Byte-Order Mark. Three chars placed at the beginning of a file (0xEF,0xBB,0xBF) that look like ""
    – Signal15
    Commented Nov 26, 2014 at 16:50
  • 70
    This is incredibly frustrating. Even third party modules get polluted, like trying to upload a file over SSH? BOM! "Yeah, let's corrupt every single file; that sounds like a good idea." -Microsoft.
    – MichaelGG
    Commented Apr 1, 2015 at 20:48
  • 10
    The default encoding is UTF8NoBOM starting with Powershell version 6.0 learn.microsoft.com/en-us/powershell/module/…
    – Paul S
    Commented Jul 9, 2019 at 14:48
  • 4
    Talk about breaking backwards compatibility...
    – Dragas
    Commented Jan 13, 2020 at 15:31
  • 1
    I feel like it should be noted that while a BOM in a UTF-8 file does make a lot of systems choke, it is explicitly valid in the Unicode UTF-8 spec to include one.
    – Bacon Bits
    Commented Jun 28, 2022 at 14:10

21 Answers 21

313

Using .NET's UTF8Encoding class and passing $False to the constructor seems to work:

$MyRawString = Get-Content -Raw $MyPath
$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
[System.IO.File]::WriteAllLines($MyPath, $MyRawString, $Utf8NoBomEncoding)
14
  • 68
    Ugh, I hope that's not the only way.
    – Scott Muc
    Commented May 24, 2011 at 6:16
  • 146
    One line [System.IO.File]::WriteAllLines($MyPath, $MyFile) is enough. This WriteAllLines overload writes exactly UTF8 without BOM. Commented Nov 8, 2011 at 19:42
  • 6
    Created an MSDN feature request here: connect.microsoft.com/PowerShell/feedbackdetail/view/1137121/…
    – Groostav
    Commented Feb 18, 2015 at 20:08
  • 8
    Note that WriteAllLines seems to require $MyPath to be absolute.
    – sschuberth
    Commented Jan 4, 2017 at 15:38
  • 22
    @xdhmoore WriteAllLines gets the current directory from [System.Environment]::CurrentDirectory. If you open PowerShell and then change your current directory (using cd or Set-Location), then [System.Environment]::CurrentDirectory will not be changed and the file will end up being in the wrong directory. You can work around this by [System.Environment]::CurrentDirectory = (Get-Location).Path. Commented Sep 30, 2017 at 19:00
115

The proper way as of now is to use a solution recommended by @Roman Kuzmin in comments to @M. Dudley answer:

[IO.File]::WriteAllLines($filename, $content)

(I've also shortened it a bit by stripping unnecessary System namespace clarification - it will be substituted automatically by default.)

8
  • 4
    This (for whatever reason) did not remove the BOM for me, where as the accepted answer did
    – Liam
    Commented Jun 17, 2016 at 10:31
  • 1
    @Liam, probably some old version of PowerShell or .NET?
    – ForNeVeR
    Commented Jun 17, 2016 at 14:58
  • 2
    I believe older versions of the .NET WriteAllLines function did write the BOM by default. So it could be a version issue.
    – codewario
    Commented Jan 23, 2017 at 16:38
  • 3
    Confirmed with writes with a BOM in Powershell 3, but without a BOM in Powershell 4. I had to use M. Dudley's original answer.
    – chazbot7
    Commented Oct 30, 2017 at 22:31
  • 7
    So it works on Windows 10 where it's installed by default. :) Also, suggested improvement: [IO.File]::WriteAllLines(($filename | Resolve-Path), $content) Commented Jan 12, 2018 at 7:05
83

I figured this wouldn't be UTF, but I just found a pretty simple solution that seems to work...

Get-Content path/to/file.ext | out-file -encoding ASCII targetFile.ext

For me this results in a utf-8 without bom file regardless of the source format.

7
  • 18
    This worked for me, except I used -encoding utf8 for my requirement.
    – Just Rudy
    Commented Jan 12, 2017 at 14:53
  • 2
    Thank you very much. I am working with dump logs of a tool - which had tabs inside it. UTF-8 was not working. ASCII solved the problem. Thanks. Commented Apr 7, 2017 at 5:50
  • 81
    Yes, -Encoding ASCII avoids the BOM problem, but you obviously only get 7-bit ASCII characters. Given that ASCII is a subset of UTF-8, the resulting file is technically also a valid UTF-8 file, but all non-ASCII characters in your input will be converted to literal ? characters.
    – mklement0
    Commented Apr 7, 2017 at 13:51
  • 15
    Warning: Definitely not. This deletes all non-ASCII characters and replaces them with question marks. Don't do this or you will lose data! (Tried with PS 5.1 on Windows 10)
    – ygoe
    Commented Apr 13, 2022 at 11:51
  • 4
    10000% agree with @ygoe. This solution should be avoided. There are much better answers here, such as the accepted answer--and of course, the fact that newer versions of the x-plat PowerShell use no BOM by default; but for those using Desktop Edition, see the accepted answer. Commented Sep 6, 2023 at 19:00
67

Note: This answer applies to Windows PowerShell; by contrast, in the cross-platform PowerShell Core edition (v6+), UTF-8 without BOM is the default encoding, across all cmdlets.

  • In other words: If you're using PowerShell (Core), i.e. version 6 or higher, you get BOM-less UTF-8 files by default (which you can also explicitly request with -Encoding utf8 / -Encoding utf8NoBOM, whereas you get with-BOM encoding with -utf8BOM).

  • If you're running Windows 10 or above and you're willing to switch to BOM-less UTF-8 encoding system-wide - which has far-reaching consequences, however - even Windows PowerShell can be made to use BOM-less UTF-8 consistently - see this answer.


To complement M. Dudley's own simple and pragmatic answer (and ForNeVeR's more concise reformulation):

  • A simple, (non-streaming) PowerShell-native alternative is to use New-Item, which (curiously) creates BOM-less UTF-8 files by default even in Windows PowerShell:

    # Note the use of -Raw to read the file as a whole.
    # Unlike with Set-Content / Out-File *no* trailing newline is appended.
    $null = New-Item -Force $MyPath -Value (Get-Content -Raw $MyPath)
    
    • Note: To save the output from arbitrary commands in the same format as Out-File would, pipe to Out-String first; e.g.:

       $null = New-Item -Force Out.txt -Value (Get-ChildItem | Out-String) 
      
  • For convenience, below is advanced custom function Out-FileUtf8NoBom, a pipeline-based alternative that mimics Out-File, which means:

    • you can use it just like Out-File in a pipeline.
    • input objects that aren't strings are formatted as they would be if you sent them to the console, just like with Out-File.
    • an additional -UseLF switch allows you use Unix-format LF-only newlines ("`n") instead of the Windows-format CRLF newlines ("`r`n") you normally get.

Example:

(Get-Content $MyPath) | Out-FileUtf8NoBom $MyPath # Add -UseLF for Unix newlines

Note how (Get-Content $MyPath) is enclosed in (...), which ensures that the entire file is opened, read in full, and closed before sending the result through the pipeline. This is necessary in order to be able to write back to the same file (update it in place).
Generally, though, this technique is not advisable for 2 reasons: (a) the whole file must fit into memory and (b) if the command is interrupted, data will be lost.

A note on memory use:

  • M. Dudley's own answer and the New-Item alternative above require that the entire file contents be built up in memory first, which can be problematic with large input sets.
  • The function below does not require this, because it is implemented as a proxy (wrapper) function (for a concise summary of how to define such functions, see this answer).

Source code of function Out-FileUtf8NoBom:

Note: The function is also available as an MIT-licensed Gist, and only the latter will be maintained going forward.

You can install it directly with the following command (while I can personally assure you that doing so is safe, you should always check the content of a script before directly executing it this way):

# Download and define the function.
irm https://gist.github.com/mklement0/8689b9b5123a9ba11df7214f82a673be/raw/Out-FileUtf8NoBom.ps1 | iex
function Out-FileUtf8NoBom {

  <#
  .SYNOPSIS
    Outputs to a UTF-8-encoded file *without a BOM* (byte-order mark).

  .DESCRIPTION

    Mimics the most important aspects of Out-File:
      * Input objects are sent to Out-String first.
      * -Append allows you to append to an existing file, -NoClobber prevents
        overwriting of an existing file.
      * -Width allows you to specify the line width for the text representations
        of input objects that aren't strings.
    However, it is not a complete implementation of all Out-File parameters:
      * Only a literal output path is supported, and only as a parameter.
      * -Force is not supported.
      * Conversely, an extra -UseLF switch is supported for using LF-only newlines.

  .NOTES
    The raison d'être for this advanced function is that Windows PowerShell
    lacks the ability to write UTF-8 files without a BOM: using -Encoding UTF8 
    invariably prepends a BOM.

    Copyright (c) 2017, 2022 Michael Klement <[email protected]> (http://same2u.net), 
    released under the [MIT license](https://spdx.org/licenses/MIT#licenseText).

  #>

  [CmdletBinding(PositionalBinding=$false)]
  param(
    [Parameter(Mandatory, Position = 0)] [string] $LiteralPath,
    [switch] $Append,
    [switch] $NoClobber,
    [AllowNull()] [int] $Width,
    [switch] $UseLF,
    [Parameter(ValueFromPipeline)] $InputObject
  )

  begin {

    # Convert the input path to a full one, since .NET's working dir. usually
    # differs from PowerShell's.
    $dir = Split-Path -LiteralPath $LiteralPath
    if ($dir) { $dir = Convert-Path -ErrorAction Stop -LiteralPath $dir } else { $dir = $pwd.ProviderPath }
    $LiteralPath = [IO.Path]::Combine($dir, [IO.Path]::GetFileName($LiteralPath))
    
    # If -NoClobber was specified, throw an exception if the target file already
    # exists.
    if ($NoClobber -and (Test-Path $LiteralPath)) {
      Throw [IO.IOException] "The file '$LiteralPath' already exists."
    }
    
    # Create a StreamWriter object.
    # Note that we take advantage of the fact that the StreamWriter class by default:
    # - uses UTF-8 encoding
    # - without a BOM.
    $sw = New-Object System.IO.StreamWriter $LiteralPath, $Append
    
    $htOutStringArgs = @{}
    if ($Width) { $htOutStringArgs += @{ Width = $Width } }

    try { 
      # Create the script block with the command to use in the steppable pipeline.
      $scriptCmd = { 
        & Microsoft.PowerShell.Utility\Out-String -Stream @htOutStringArgs | 
          . { process { if ($UseLF) { $sw.Write(($_ + "`n")) } else { $sw.WriteLine($_) } } }
      }  
      
      $steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
      $steppablePipeline.Begin($PSCmdlet)
    }
    catch { throw }

  }

  process
  {
    $steppablePipeline.Process($_)
  }

  end {
    $steppablePipeline.End()
    $sw.Dispose()
  }

}
3
  • 1
    Example usage from a utf8BOM file to plain utf8: $null = New-Item -Force "\$env:ProgramData\ssh\administrators_authorized_keys" -Value (Get-Content -Path "\$env:ProgramData\ssh\administrators_authorized_keys" | Out-String)
    – nhooyr
    Commented Dec 15, 2022 at 13:37
  • 2
    @nhooyr, it's better to use $null = New-Item -Force $MyPath -Value (Get-Content -Raw $MyPath) (much faster, and preserves the existing newline format) - I've updated the answer.
    – mklement0
    Commented Dec 15, 2022 at 13:57
  • -Encoding utf8 added the BOM in my case. I used -Encoding Default instead to convert to UTF-8 without BOM (Windows 11)
    – Max M
    Commented Jul 11 at 7:05
36

Starting from version 6 powershell supports the UTF8NoBOM encoding both for set-content and out-file and even uses this as default encoding.

So in the above example it should simply be like this:

$MyFile | Out-File -Encoding UTF8NoBOM $MyPath
4
  • 5
    Nice. FYI check version with $PSVersionTable.PSVersion
    – KCD
    Commented Oct 29, 2019 at 2:48
  • 2
    Worth noting that in PowerShell [Core] v6+ -Encoding UTF8NoBOM is never required, because it is the default encoding.
    – mklement0
    Commented Oct 25, 2020 at 20:58
  • learn.microsoft.com/en-us/powershell/module/… seems to confirm this - but having issues scraping redirected dos output to files and emitting back out. Notepad++ gives different encodings (UCS-2 LE BOM on one, and UTF8-BOM on the other!) on two of files I emit - when driving a screen scraping automation/redirection to an temp outfile and extracting substrings. I am driving from a CSV for different arguments for the same command, also setting encoding everywhere - but Powershell seems to be ignoring that.
    – JGFMK
    Commented Mar 29, 2023 at 11:32
  • Be sure to check out this answer and my comment under the question itself here too if you are pulling your hair out over Powershell encoding not working as expected (Set-Content and Out-File giving different encoding answers despite specifically requesting utf8!). stackoverflow.com/a/5596984/495157
    – JGFMK
    Commented Mar 29, 2023 at 13:47
21

Here's an alternative to the accepted answer.

The advantage of this approach is that it's compatible with IO.FileInfo objects (from functions like Get-Item) and relative paths.

  1. Create a Text.UTF8Encoding object

    • While Text.UTF8Encoding is capable of inserting a BOM, it doesn't by default
  2. Call the object's GetBytes method to convert a string into bytes

    • Ensure the target string isn't actually a string array$stringVar.Count should equal 1
  3. Write the byte array to your target with Set-Content -Encoding Byte

# This is a reusable class instance object
$utf8 = New-Object Text.UTF8Encoding

$GCRaw = Get-Content -Raw -PSPath $MyPath
Set-Content -Value $utf8.GetBytes($GCRaw) -Encoding Byte -PSPath $MyPath

This can be shortened by letting -Value be inferred by position and, additionally, by creating the Text.UTF8Encoding object from within the argument.

$GCRaw = Get-Content $MyPath -Raw

Set-Content ([Text.UTF8Encoding]::new().GetBytes($GCRaw)) -Encoding Byte -PSPath $MyPath

#NOTE#
# (New-Object Text.UTF8Encoding).GetBytes($GCRaw))
# can be used instead of
# ([Text.UTF8Encoding]::new().GetBytes($GCRaw))
# For code intended to be compact, I recommend the latter,
# not just because it's not as long, but also because its
# lack of whitespace makes it visually more distinct.
1
  • 1
    Nice - works great with strings (which may be all that is needed and certainly meets the requirements of the question). In case you need to take advantage of the formatting that Out-File, unlike Set-Content, provides, pipe to Out-String first; e.g., $MyFile = Get-ChildItem | Out-String
    – mklement0
    Commented Oct 25, 2020 at 21:06
8

important!: this only works if an extra space or newline at the start is no problem for your use case of the file
(e.g. if it is an SQL file, Java file or human readable text file)

one could use a combination of creating an empty (non-UTF8 or ASCII (UTF8-compatible)) file and appending to it (replace $str with gc $src if the source is a file):

" "    |  out-file  -encoding ASCII  -noNewline  $dest
$str  |  out-file  -encoding UTF8   -append     $dest

as one-liner

replace $dest and $str according to your use case:

$_ofdst = $dest ; " " | out-file -encoding ASCII -noNewline $_ofdst ; $src | out-file -encoding UTF8 -append $_ofdst

as simple function

function Out-File-UTF8-noBOM { param( $str, $dest )
  " "    |  out-file  -encoding ASCII  -noNewline  $dest
  $str  |  out-file  -encoding UTF8   -append     $dest
}

using it with a source file:

Out-File-UTF8-noBOM  (gc $src),  $dest

using it with a string:

Out-File-UTF8-noBOM  $str,  $dest
  • optionally: continue appending with Out-File:

    "more foo bar"  |  Out-File -encoding UTF8 -append  $dest
    
2
  • UTF8 != ASCII. So no, this wouldn't work in all cases. All ASCII can be converted to UTF8, but not ALL UTF8 can be converted to ASCII and back again--i.e., UTF8 is NOT (all) round-trippable to ASCII and back. In fact, this can be dangerous. Commented Sep 6, 2023 at 18:57
  • @fourpastmidnight that's why I mentioned this in the beginning :) Commented Sep 8, 2023 at 5:48
8

Old question, new answer:

While the "old" powershell writes a BOM, the new platform-agnostic variant does behave differently: The default is "no BOM" and it can be configured via switch:

-Encoding

Specifies the type of encoding for the target file. The default value is utf8NoBOM.

The acceptable values for this parameter are as follows:

  • ascii: Uses the encoding for the ASCII (7-bit) character set.
  • bigendianunicode: Encodes in UTF-16 format using the big-endian byte order.
  • oem: Uses the default encoding for MS-DOS and console programs.
  • unicode: Encodes in UTF-16 format using the little-endian byte order.
  • utf7: Encodes in UTF-7 format.
  • utf8: Encodes in UTF-8 format.
  • utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
  • utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)
  • utf32: Encodes in UTF-32 format.

Source: https://learn.microsoft.com/de-de/powershell/module/Microsoft.PowerShell.Utility/Out-File?view=powershell-7 Emphasis mine

0
6

This script will convert, to UTF-8 without BOM, all .txt files in DIRECTORY1 and output them to DIRECTORY2

foreach ($i in ls -name DIRECTORY1\*.txt)
{
    $file_content = Get-Content "DIRECTORY1\$i";
    [System.IO.File]::WriteAllLines("DIRECTORY2\$i", $file_content);
}
2
  • This one fails without any warning. What version of powershell should I use to run it? Commented Sep 8, 2013 at 13:34
  • 5
    The WriteAllLines solution works great for small files. However, I need a solution for larger files. Every time I try to use this with a larger file I'm getting an OutOfMemory error. Commented Mar 25, 2015 at 15:44
4

For PowerShell 5.1, enable this setting:

Control Panel, Region, Administrative, Change system locale, Use Unicode UTF-8 for worldwide language support

Then enter this into PowerShell:

$PSDefaultParameterValues['*:Encoding'] = 'Default'

Alternatively, you can upgrade to PowerShell 6 or higher.

https://github.com/PowerShell/PowerShell

1
  • 2
    To spell it out: This is a system-wide setting that makes Windows PowerShell default to BOM-less UTF-8 across all cmdlets, which may or may not be desired, not least because the feature is still in beta (as of this writing) and can break legacy console applications - see this answer for background information.
    – mklement0
    Commented Dec 8, 2020 at 17:23
2

If you want to use [System.IO.File]::WriteAllLines(), you should cast second parameter to String[] (if the type of $MyFile is Object[]), and also specify absolute path with $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), like:

$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Set-Variable MyFile
[System.IO.File]::WriteAllLines($ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath), [String[]]$MyFile, $Utf8NoBomEncoding)

If you want to use [System.IO.File]::WriteAllText(), sometimes you should pipe the second parameter into | Out-String | to add CRLFs to the end of each line explictly (Especially when you use them with ConvertTo-Csv):

$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | Set-Variable tmp
[System.IO.File]::WriteAllText("/absolute/path/to/foobar.csv", $tmp, $Utf8NoBomEncoding)

Or you can use [Text.Encoding]::UTF8.GetBytes() with Set-Content -Encoding Byte:

$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding $False
Get-ChildItem | ConvertTo-Csv | Out-String | % { [Text.Encoding]::UTF8.GetBytes($_) } | Set-Content -Encoding Byte -Path "/absolute/path/to/foobar.csv"

see: How to write result of ConvertTo-Csv to a file in UTF-8 without BOM

1
  • Good pointers; suggestions/: the simpler alternative to $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($MyPath) is Convert-Path $MyPath; if you want to ensure a trailing CRLF, simply use [System.IO.File]::WriteAllLines() even with a single input string (no need for Out-String).
    – mklement0
    Commented Feb 19, 2018 at 16:05
2

I would say to use just the Set-Content command, nothing else needed.

The powershell version in my system is :-

PS C:\Users\XXXXX> $PSVersionTable.PSVersion | fl


Major         : 5
Minor         : 1
Build         : 19041
Revision      : 1682
MajorRevision : 0
MinorRevision : 1682

PS C:\Users\XXXXX>

So you would need something like following.

PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX> Get-Content .\Downloads\finddate.txt | Set-Content .\Downloads\anotherfile.txt
PS C:\Users\XXXXX> Get-Content .\Downloads\anotherfile.txt
Thursday, June 23, 2022 5:57:59 PM
PS C:\Users\XXXXX>

Now when we check the file as per the screenshot it is utf8. anotherfile.txt

PS: To answer on the comment query on foreign character issue. The contents from file "testfgnchar.txt" which having the foreign characters, was copied to "findfnchar2.txt" using the following command.

PS C:\Users\XXXXX> Get-Content .\testfgnchar.txt | Set-Content findfnchar2.txt
PS C:\Users\XXXXX>

screen-shot is here.

Note: Currently, there are newer versions of PowerShell exists, than the one I used during answer.

16
  • 1
    This seemed to work at first, but this actually uses the user’s ANSI code page and replaces other symbols with closest equivalents (e. g. š → s) or question marks. Using set-content -encoding utf8 works.
    – Chortos-2
    Commented Mar 21, 2023 at 13:44
  • @Chortos-2, thanks for commenting. I am more concerned to save the file in "UTF-8" format strictly. If I use "set-content -encoding utf8", it saves the file in "UTF-8-BOM" format Commented Apr 5, 2023 at 8:11
  • 1
    Ah, that’s true; I didn’t notice. But then that means this command is completely unsuitable for the task, because without -encoding, it doesn’t use UTF-8 at all, neither with BOM nor without.
    – Chortos-2
    Commented Apr 6, 2023 at 11:19
  • @Chortos-2, Each system or user will have their own language set during installation. The answer I have shared is for Language English(United States) which is on my system and same is also showing in my Regional Language. I feel this language is most commonly used during installation. Hence that could be the blocker here, but could not help further. What Language system is showing in your system? Press Windows Key and I on your machine to find out the same. Commented Apr 7, 2023 at 12:11
  • 1
    The point is that this command uses the ANSI code page, not UTF-8, which was explicitly requested in the question (unless you’ve set ANSI to be UTF-8 as per Zombo’s answer). On your English system, try echo āčķʃλшא⁴ℝ→⅛≈あ子 | set-content file.txt, and you’ll see none of the characters were preserved. The same problem is pointed out for other PowerShell command in the comments to other answers. It’s certainly good to know that set-content defaults to encoding Latin in single bytes, but it is very different from UTF-8 as originally requested.
    – Chortos-2
    Commented Apr 8, 2023 at 21:00
2

I have created the following code for easy logging. It creates an UTF-8 file without BOM when executed in PowerShell 5. It works as expected for me. Feel free to customize it to your needs :-)

Function myWriteLog{
# $LogFilePath has to be defined before calling the function 
# And, "$SciptName=$MyInvocation.MyCommand.Name" has to be set before calling the function 
    Param( [Parameter(Mandatory=$true, ValueFromPipeline=$true)]
           [string]$content
         )

# disallow a NULL or an EMPTY value 
if ([string]::IsNullOrEmpty($content.Trim())){
    throw "Found 'EMPTY or NULL': you must provide a nonNull and nonEmpty string to function ""myWriteLog"""
    return 0
} else { 
    if((Test-Path $LogFilePath) -eq $False){

        # Creates the file, please note that option  "-NoNewline" has to be set
        "" | Out-file -FilePath $LogFilePath -Encoding ascii -Force -NoNewline
    
        # Create a string as a line separator for a file header
        $t ="".PadLeft(("Logfile for : $SciptName").Length,"#")
        Add-Content -path $LogFilePath -value "$t"
        Add-Content -path $LogFilePath -value "Logfile for : $SciptName"
        Add-Content -path $LogFilePath -value "LogFile Created: $(Get-date -F "yyyy-MM-dd-HH-mm-ss")"
        Add-Content -path $LogFilePath -value "$t"
        Add-Content -path $LogFilePath -value ""
        
        #and now add the content to 
        Add-Content -path $LogFilePath -value "$(Get-date -F "yyyy-MM-dd-HH-mm-ss") : $content" -Encoding UTF8 -force
    }else{
        Add-Content -path $LogFilePath -value "$(Get-date -F "yyyy-MM-dd-HH-mm-ss") : $content" -Encoding UTF8 -force
    }
}

}

1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Mar 29 at 18:35
1

Change multiple files by extension to UTF-8 without BOM:

$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False)
foreach($i in ls -recurse -filter "*.java") {
    $MyFile = Get-Content $i.fullname 
    [System.IO.File]::WriteAllLines($i.fullname, $MyFile, $Utf8NoBomEncoding)
}
1
    [System.IO.FileInfo] $file = Get-Item -Path $FilePath 
    $sequenceBOM = New-Object System.Byte[] 3 
    $reader = $file.OpenRead() 
    $bytesRead = $reader.Read($sequenceBOM, 0, 3) 
    $reader.Dispose() 
    #A UTF-8+BOM string will start with the three following bytes. Hex: 0xEF0xBB0xBF, Decimal: 239 187 191 
    if ($bytesRead -eq 3 -and $sequenceBOM[0] -eq 239 -and $sequenceBOM[1] -eq 187 -and $sequenceBOM[2] -eq 191) 
    { 
        $utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False) 
        [System.IO.File]::WriteAllLines($FilePath, (Get-Content $FilePath), $utf8NoBomEncoding) 
        Write-Host "Remove UTF-8 BOM successfully" 
    } 
    Else 
    { 
        Write-Warning "Not UTF-8 BOM file" 
    }  

Source How to remove UTF8 Byte Order Mark (BOM) from a file using PowerShell

1

One technique I utilize is to redirect output to an ASCII file using the Out-File cmdlet.

For example, I often run SQL scripts that create another SQL script to execute in Oracle. With simple redirection (">"), the output will be in UTF-16 which is not recognized by SQLPlus. To work around this:

sqlplus -s / as sysdba "@create_sql_script.sql" |
Out-File -FilePath new_script.sql -Encoding ASCII -Force

The generated script can then be executed via another SQLPlus session without any Unicode worries:

sqlplus / as sysdba "@new_script.sql" |
tee new_script.log

Update: As others have pointed out, this will drop non-ASCII characters. Since the user asked for a way to "force" conversion, I assume they do not care about that as perhaps their data does not contain such data.

If you care about the preservation of non-ASCII characters, this is not the answer for you.

5
  • 8
    Yes, -Encoding ASCII avoids the BOM problem, but you obviously only get support for 7-bit ASCII characters. Given that ASCII is a subset of UTF-8, the resulting file is technically also a valid UTF-8 file, but all non-ASCII characters in your input will be converted to literal ? characters.
    – mklement0
    Commented Feb 19, 2018 at 17:03
  • 1
    This answer needs more votes. The sqlplus incompatibility with BOM is a cause of many headaches.
    – Amit Naidu
    Commented Mar 8, 2018 at 0:06
  • 3
    @AmitNaidu No, this is the wrong answer, because it won't work if the text has any non-ascii characters: any accents, umlauts, oriental/cryllic, etc. Commented Feb 10, 2022 at 5:19
  • @JoelCoehoorn This is a correct answer according to what the user asked. Since the user asked for a way to "force", they're not expecting any issues or don't care probably because the source doesn't use any non-ASCII characters. For those who do care about the preservation of those characters, this will not work. Commented Feb 11, 2022 at 18:43
  • No, the OP did not ask for a way to "force". They said PowerShell out-file forces BOM. Commented Jun 17 at 8:34
1

Used this method to edit a UTF8-NoBOM file and generated a file with correct encoding-

$fileD = "file.xml"
(Get-Content $fileD) | ForEach-Object { $_ -replace 'replace text',"new text" } | out-file "file.xml" -encoding ASCII

I was skeptical at this method at first, but it surprised me and worked!

Tested with powershell version 5.1

1

If your first line does not contain anything fancy that doesn't require UTF8, the following will create an UTF8 file without BOM on stock Windows 10 Powershell:

$file = get-content -path "C:\temp\myfile.txt" -Encoding UTF8

# do some stuff.

$file[0] | out-file "C:\temp\mynewfile.txt" -Encoding ascii
$file | select -skip 1 | out-file "C:\temp\mynewfile.txt" -append utf8

This uses 2 lines to create the new file. The first one uses -encoding ascii to force UTF8, but it will be limited to 7-bit ascii. With a textfile, this is usually not an issue, otherwise you'd probably choose byte encoding anyway.

The second command appends the rest, but skips the first line as we already parsed that one with full UTF8 support.

0

I have the same error in the PowerShell and used this isolation and fixed it

$PSDefaultParameterValues['*:Encoding'] = 'utf8'
2
  • Just setting the utf is not enough. You also need to specify -encoding default. eg: $filecontent | Out-File $Filename -Encoding default. Do note that this reencodes the file with utf8, so a file that was not encoded properly will change.
    – LPChip
    Commented Sep 11, 2023 at 12:13
  • bom is the three byte code at the beginning of a file to indicate the type of a file, and has nothing really to do with the encoding.
    – Tolga
    Commented May 5 at 18:27
-3

Could use below to get UTF8 without BOM

$MyFile | Out-File -Encoding ASCII
6
  • 4
    No, it will convert the output to current ANSI codepage (cp1251 or cp1252, for example). It is not UTF-8 at all!
    – ForNeVeR
    Commented Oct 5, 2015 at 15:05
  • 1
    Thanks Robin. This may not have worked for writing a UTF-8 file without the BOM but the -Encoding ASCII option removed the BOM. That way I could generate a bat file for gvim. The .bat file was tripping up on the BOM.
    – Greg
    Commented Dec 10, 2015 at 22:34
  • 3
    @ForNeVeR: You're correct that encoding ASCII is not UTF-8, but it's als not the current ANSI codepage - you're thinking of Default; ASCII truly is 7-bit ASCII encoding, with codepoints >= 128 getting converted to literal ? instances.
    – mklement0
    Commented Jan 21, 2016 at 6:01
  • 1
    @ForNeVeR: You're probably thinking of "ANSI" or "extended ASCII". Try this to verify that -Encoding ASCII is indeed 7-bit ASCII only: 'äb' | out-file ($f = [IO.Path]::GetTempFilename()) -encoding ASCII; '?b' -eq $(Get-Content $f; Remove-Item $f) - the ä has been transliterated to a ?. By contrast, -Encoding Default ("ANSI") would correctly preserve it.
    – mklement0
    Commented Jan 21, 2016 at 15:07
  • 3
    @rob This is the perfect answer for everybody who just doesn't need utf-8 or anything else that is different to ASCII and is not interested in understanding encodings and the purpose of unicode. You can use it as utf-8 because the equivalent utf-8 characters to all ASCII characters are identical (means converting an ASCII-file to an utf-8-file results in an identical file (if it gets no BOM)). For all who have non-ASCII characters in their text this answer is just false and misleading.
    – TNT
    Commented Aug 25, 2016 at 19:25
-4

This one works for me (use "Default" instead of "UTF8"):

$MyFile = Get-Content $MyPath
$MyFile | Out-File -Encoding "Default" $MyPath

The result is ASCII without BOM.

3
  • 3
    Per the Out-File documentation specifying the Default encoding will use the system's current ANSI code page, which is not UTF-8, as I required. Commented May 6, 2015 at 13:21
  • This does seem to work for me, at least for Export-CSV. If you open the resulting file in a proper editor, the file encoding is UTF-8 without BOM, and not Western Latin ISO 9 as I would have expected with ASCII
    – eythort
    Commented Aug 5, 2016 at 11:00
  • 1
    Many editors open the file as UTF-8 if they can't detect the encoding.
    – emptyother
    Commented Jul 22, 2017 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.