Note: This answer applies to Windows PowerShell; by contrast, in the cross-platform PowerShell Core edition (v6+), UTF-8 without BOM is the default encoding, across all cmdlets.
In other words: If you're using PowerShell (Core), i.e. version 6 or higher, you get BOM-less UTF-8 files by default (which you can also explicitly request with -Encoding utf8
/ -Encoding utf8NoBOM
, whereas you get with-BOM encoding with -utf8BOM
).
If you're running Windows 10 or above and you're willing to switch to BOM-less UTF-8 encoding system-wide - which has far-reaching consequences, however - even Windows PowerShell can be made to use BOM-less UTF-8 consistently - see this answer.
To complement M. Dudley's own simple and pragmatic answer (and ForNeVeR's more concise reformulation):
A simple, (non-streaming) PowerShell-native alternative is to use New-Item
, which (curiously) creates BOM-less UTF-8 files by default even in Windows PowerShell:
# Note the use of -Raw to read the file as a whole.
# Unlike with Set-Content / Out-File *no* trailing newline is appended.
$null = New-Item -Force $MyPath -Value (Get-Content -Raw $MyPath)
Note: To save the output from arbitrary commands in the same format as Out-File
would, pipe to Out-String
first; e.g.:
$null = New-Item -Force Out.txt -Value (Get-ChildItem | Out-String)
For convenience, below is advanced custom function Out-FileUtf8NoBom
, a pipeline-based alternative that mimics Out-File
, which means:
- you can use it just like
Out-File
in a pipeline.
- input objects that aren't strings are formatted as they would be if you sent them to the console, just like with
Out-File
.
- an additional
-UseLF
switch allows you use Unix-format LF-only newlines ("`n"
) instead of the Windows-format CRLF newlines ("`r`n"
) you normally get.
Example:
(Get-Content $MyPath) | Out-FileUtf8NoBom $MyPath # Add -UseLF for Unix newlines
Note how (Get-Content $MyPath)
is enclosed in (...)
, which ensures that the entire file is opened, read in full, and closed before sending the result through the pipeline. This is necessary in order to be able to write back to the same file (update it in place).
Generally, though, this technique is not advisable for 2 reasons: (a) the whole file must fit into memory and (b) if the command is interrupted, data will be lost.
A note on memory use:
- M. Dudley's own answer
and the
New-Item
alternative above require that the entire file contents be built up in memory first, which can be problematic with large input sets.
- The function below does not require this, because it is implemented as a proxy (wrapper) function (for a concise summary of how to define such functions, see this answer).
Source code of function Out-FileUtf8NoBom
:
Note: The function is also available as an MIT-licensed Gist, and only the latter will be maintained going forward.
You can install it directly with the following command (while I can personally assure you that doing so is safe, you should always check the content of a script before directly executing it this way):
# Download and define the function.
irm https://gist.github.com/mklement0/8689b9b5123a9ba11df7214f82a673be/raw/Out-FileUtf8NoBom.ps1 | iex
function Out-FileUtf8NoBom {
<#
.SYNOPSIS
Outputs to a UTF-8-encoded file *without a BOM* (byte-order mark).
.DESCRIPTION
Mimics the most important aspects of Out-File:
* Input objects are sent to Out-String first.
* -Append allows you to append to an existing file, -NoClobber prevents
overwriting of an existing file.
* -Width allows you to specify the line width for the text representations
of input objects that aren't strings.
However, it is not a complete implementation of all Out-File parameters:
* Only a literal output path is supported, and only as a parameter.
* -Force is not supported.
* Conversely, an extra -UseLF switch is supported for using LF-only newlines.
.NOTES
The raison d'être for this advanced function is that Windows PowerShell
lacks the ability to write UTF-8 files without a BOM: using -Encoding UTF8
invariably prepends a BOM.
Copyright (c) 2017, 2022 Michael Klement <[email protected]> (http://same2u.net),
released under the [MIT license](https://spdx.org/licenses/MIT#licenseText).
#>
[CmdletBinding(PositionalBinding=$false)]
param(
[Parameter(Mandatory, Position = 0)] [string] $LiteralPath,
[switch] $Append,
[switch] $NoClobber,
[AllowNull()] [int] $Width,
[switch] $UseLF,
[Parameter(ValueFromPipeline)] $InputObject
)
begin {
# Convert the input path to a full one, since .NET's working dir. usually
# differs from PowerShell's.
$dir = Split-Path -LiteralPath $LiteralPath
if ($dir) { $dir = Convert-Path -ErrorAction Stop -LiteralPath $dir } else { $dir = $pwd.ProviderPath }
$LiteralPath = [IO.Path]::Combine($dir, [IO.Path]::GetFileName($LiteralPath))
# If -NoClobber was specified, throw an exception if the target file already
# exists.
if ($NoClobber -and (Test-Path $LiteralPath)) {
Throw [IO.IOException] "The file '$LiteralPath' already exists."
}
# Create a StreamWriter object.
# Note that we take advantage of the fact that the StreamWriter class by default:
# - uses UTF-8 encoding
# - without a BOM.
$sw = New-Object System.IO.StreamWriter $LiteralPath, $Append
$htOutStringArgs = @{}
if ($Width) { $htOutStringArgs += @{ Width = $Width } }
try {
# Create the script block with the command to use in the steppable pipeline.
$scriptCmd = {
& Microsoft.PowerShell.Utility\Out-String -Stream @htOutStringArgs |
. { process { if ($UseLF) { $sw.Write(($_ + "`n")) } else { $sw.WriteLine($_) } } }
}
$steppablePipeline = $scriptCmd.GetSteppablePipeline($myInvocation.CommandOrigin)
$steppablePipeline.Begin($PSCmdlet)
}
catch { throw }
}
process
{
$steppablePipeline.Process($_)
}
end {
$steppablePipeline.End()
$sw.Dispose()
}
}