In the world of scripting and automation, consistent data handling is paramount. Developers and system administrators often grapple with character encoding issues, particularly when dealing with diverse datasets or international characters. PowerShell, a powerful command-line shell and scripting language, traditionally had default encoding behaviors that could lead to unexpected results, such as garbled text or data corruption, especially when piping output or writing to files. This challenge highlights the critical need for a reliable and universal encoding standard. This article delves into the essential process of changing PowerShell’s default output encoding to UTF-8, a standard that ensures broad compatibility and prevents data integrity issues, paving the way for more robust and globally-aware scripts. We’ll explore why UTF-8 is the superior choice, how PowerShell handles encoding, and provide clear, actionable steps to implement this crucial change across your environment.
Understanding the Imperative: Why UTF-8 is Essential for PowerShell
Character encoding defines how characters are represented in digital form. While older systems might default to ANSI or locale-specific encodings, these often fall short when scripts interact with diverse data sources or users globally. Imagine a script processing log files from different operating systems or displaying user-submitted data containing special characters; without a universal encoding, these characters can quickly become question marks or unreadable symbols. This common problem undermines data accuracy and complicates troubleshooting.
UTF-8, or Unicode Transformation Format - 8-bit, stands as the dominant character encoding for the web and modern computing environments. It’s a variable-width encoding that can represent every character in the Unicode character set, which includes virtually all characters from all writing systems in the world. Its widespread adoption means that by aligning PowerShell’s output to UTF-8, you’re ensuring compatibility with web services, databases, cross-platform applications, and internationalized text files. This shift dramatically reduces the likelihood of encoding-related errors, making your scripts more reliable and your data more consistent.
The move to UTF-8 is not just about avoiding errors; it’s about future-proofing your scripts and workflows. As systems become more interconnected and global, the ability to handle any character seamlessly becomes a fundamental requirement. Historically, PowerShell’s default encoding varied depending on the version and operating system locale, often leading to inconsistencies when scripts were moved between environments. Standardizing on UTF-8 provides a predictable and universal baseline for all your PowerShell operations, from simple console output to complex file manipulations and API interactions. It streamlines development and reduces the overhead of constantly converting or specifying encoding for individual commands.
PowerShell’s Encoding Landscape and the Need for Change
Out of the box, different versions of PowerShell have historically used varying default encodings for different cmdlets and scenarios, which can be a source of frustration. For instance, Out-File and Set-Content might default to UTF-16 (UCS-2 Little Endian) in Windows PowerShell 5.1, while PowerShell Core (PowerShell 6 and 7+) shifted towards UTF-8 without a Byte Order Mark (BOM) for many cmdlets. This inconsistency means that a script working perfectly on one machine might produce garbled output or corrupt files on another if the target system expects a different encoding.
The challenge arises particularly when you’re piping text-based output from one command to another, or when writing to files that are consumed by other applications expecting a specific encoding. If PowerShell writes output using one encoding and another application reads it expecting a different one, the data integrity is compromised. This is a common pitfall that often goes unnoticed until critical data is lost or misinterpreted. For example, a CSV file generated by PowerShell might contain non-ASCII characters that appear incorrectly when opened in Excel if the encoding isn’t handled properly.
To ensure cross-platform compatibility and prevent character corruption, explicitly changing PowerShell’s default output encoding to UTF-8 is a best practice. This involves configuring PowerShell to consistently use UTF-8 without a Byte Order Mark (BOM) for its output streams, ensuring that text data is uniformly represented regardless of the operating system or locale. This standardization is crucial for maintaining data integrity when scripts interact with APIs, generate logs, or process data across diverse systems.
According to Microsoft’s documentation, “PowerShell Core (versions 6 and higher) uses UTF-8 encoding by default for all commands and cmdlets that use character encoding, for example, Get-Content, Set-Content, and Out-File.” While this is a significant improvement, many users still rely on Windows PowerShell 5.1, and even in newer versions, specific scenarios or legacy scripts might require explicit configuration to guarantee UTF-8 behavior. Therefore, understanding how to enforce UTF-8 across all PowerShell environments remains a vital skill for any serious administrator or developer.
Implementing a consistent UTF-8 encoding strategy in PowerShell involves a few key configuration points. The most effective way to achieve this is by leveraging the $PSDefaultParameterValues preference variable and configuring your PowerShell profile. These settings provide a robust and persistent solution, ensuring that your environment is always set up for optimal character handling.
Configuring $PSDefaultParameterValues for Cmdlets
The $PSDefaultParameterValues preference variable allows you to specify default values for cmdlet parameters. This is incredibly powerful for enforcing encoding standards. By setting specific encoding for cmdlets like Out-File, Set-Content, and Add-Content, you can ensure they always use UTF-8 unless explicitly overridden. It’s important to specify UTF8NoBOM for maximum compatibility, as the Byte Order Mark can sometimes cause issues with applications that don’t expect it.
- Open your PowerShell profile: If you don’t have one, you can create it. Type notepad $PROFILE in your PowerShell console and press Enter. This will open your profile script in Notepad. If the file doesn’t exist, Notepad will ask if you want to create it.
- Add the following lines to your profile script: These lines set the default encoding for common cmdlets. ```
$PSDefaultParameterValues[‘Out-File:Encoding’] = ‘utf8NoBOM’ $PSDefaultParameterValues[‘Set-Content:Encoding’] = ‘utf8NoBOM’ $PSDefaultParameterValues[‘Add-Content:Encoding’] = ‘utf8NoBOM’ $PSDefaultParameterValues[’
Question & Answer :
By default, when you redirect the output of a command to a file or pipe it into something else in PowerShell, the encoding is UTF-16, which isn’t useful. I’m looking to change it to UTF-8.
It can be done on a case-by-case basis by replacing the >foo.txt syntax with | out-file foo.txt -encoding utf8 but this is awkward to have to repeat every time.
The persistent way to set things in PowerShell is to put them in \Users\me\Documents\WindowsPowerShell\profile.ps1; I’ve verified that this file is indeed executed on startup.
It has been said that the output encoding can be set with $PSDefaultParameterValues = @{‘Out-File:Encoding’ = ‘utf8’} but I’ve tried this and it had no effect.
https://blogs.msdn.microsoft.com/powershell/2006/12/11/outputencoding-to-the-rescue/ which talks about $OutputEncoding looks at first glance as though it should be relevant, but then it talks about output being encoded in ASCII, which is not what’s actually happening.
How do you set PowerShell to use UTF-8?
Note:
The next section applies primarily to Windows PowerShell.
- See the section after it for the cross-platform PowerShell (Core) 7 edition.
In both cases, the information applies to making PowerShell use UTF-8 for reading and writing files.
- By contrast, for information on how to send and receive UTF-8-encoded strings to and from external programs, see this answer.
A system-wide switch to BOM-less UTF-8 is possible nowadays (since recent versions of Windows 10): see this answer, but note the following caveats:
The feature has far-reaching consequences, because both the OEM and the ANSI code page are then set to 65001, i.e. UTF-8; also, the feature is still considered a beta feature as of this writing (Windows 11 22H2).
- Notably, it can change the interpretation of preexisting BOM-less files, including source code, if they contain non-ASCII characters, by any application (not just PowerShell), because they are then (potentially mis-)decoded as UTF-8.
Additionally, in Windows PowerShell, it takes effect only for those file-writing cmdlets that default to the ANSI code page, notably Set-Content, and therefore notably not for Out-File / >; the bottom section lists the default encoding of all cmdlets.
- While you can use $PSDefaultParameterValues[’:Encoding’] = ‘utf8’, as explained below, to make all cmdlets default to UTF8, file-writing cmdlets will then invariably create UTF-8 files with BOM in Windows PowerShell; this is not a concern in PowerShell 7, which consistently defaults to BOM-less UTF-8 to begin with.
The Windows PowerShell perspective:
In v5.1 (and also in PowerShell 7), where > and » are effectively aliases of Out-File, you can set the default encoding for > / » / Out-File via the $PSDefaultParameterValues preference variable:
- $PSDefaultParameterValues[‘Out-File:Encoding’] = ‘utf8’
- Note:
In Windows PowerShell (the legacy, Windows-only, ships-with-Windows edition whose latest and last version is 5.1), this invariably creates UTF-8 files with a BOM.
- Many Unix-based utilities do not recognize this BOM (see bottom); see this post for workarounds that create BOM-less UTF-8 files.
In PowerShell (Core) 7, BOM-less UTF-8 is the default (see next section), but if you do want a BOM there, you can use ‘utf8BOM’
In v5.0 or below, you cannot change the encoding for > / », but, on v3 or higher, the above technique does work for explicit calls to Out-File.
(The $PSDefaultParameterValues preference variable was introduced in v3.0).In v3.0 or higher, if you want to set the default encoding for all cmdlets that support
an -Encoding parameter (which in v5.1 and PowerShell 7 includes > and »), use:- $PSDefaultParameterValues[’:Encoding’] = ‘utf8’
- As noted, this will make all file-writing cmdlets create UTF-8 files with BOM in Windows PowerShell.
If you place this command in your $PROFILE, cmdlets such as Out-File and Set-Content will use UTF-8 encoding by default, but note that this makes it a session-global setting that will affect all commands / scripts that do not explicitly specify an encoding via their -Encoding parameter.
Similarly, be sure to include such commands in your scripts or modules that you want to behave the same way, so that they indeed behave the same even when run by another user or a different machine; however, to avoid a session-global change, use the following form to create a local copy of $PSDefaultParameterValues:
- $PSDefaultParameterValues = @{ ‘*:Encoding’ = ‘utf8’ }
For a summary of the wildly inconsistent default character encoding behavior across many of the Windows PowerShell standard cmdlets, see the bottom section.
The $OutputEncoding preference variable is unrelated to the issue at hand: it only applies to how PowerShell communicates with external programs (it determines what encoding PowerShell uses when sending strings to them, and defaults to ASCII(!) in Windows PowerShell and BOM-less UTF-8 in PowerShell 7) and therefore has nothing to do with the encoding that the output redirection operators and PowerShell cmdlets use to save to files.
Optional reading: The cross-platform perspective: PowerShell (Core) 7:
PowerShell is now cross-platform, via its PowerShell (Core) 7 edition, whose encoding - sensibly - defaults to BOM-less UTF-8, in line with Unix-like platforms.
This means that source-code files without a BOM are assumed to be UTF-8, and using > / Out-File / Set-Content defaults to BOM-less UTF-8; explicit use of the utf8 -Encoding argument too creates BOM-less UTF-8, but you can opt to create files with the pseudo-BOM with the utf8bom value.
If you create PowerShell scripts with an editor on a Unix-like platform and nowadays even on Windows with cross-platform editors such as Visual Studio Code and Sublime Text, the resulting *.ps1 file will typically not have a UTF-8 pseudo-BOM:
- This works fine in PowerShell Core.
- It may break on Windows PowerShell, if the file contains non-ASCII characters; if you do need to use non-ASCII characters in your scripts, save them as UTF-8 with BOM.
Without the BOM, Windows PowerShell (mis)interprets your script as being encoded in the legacy “ANSI” codepage (determined by the system locale for pre-Unicode applications; e.g., Windows-1252 on US-English systems).
Conversely, files that do have the UTF-8 pseudo-BOM can be problematic on Unix-like platforms, as they cause Unix utilities such as cat, sed, and awk - and even some editors such as gedit - to pass the pseudo-BOM through, i.e., to treat it as data.
- This may not always be a problem, but definitely can be, such as when you try to read a file into a string in bash with, say, text=$(cat file) or text=$(<file) - the resulting variable will contain the pseudo-BOM as the first 3 bytes.
Inconsistent default encoding behavior in Windows PowerShell:
Regrettably, the default character encoding used in Windows PowerShell is wildly inconsistent; the cross-platform PowerShell Core edition, as discussed in the previous section, has commendably put and end to this.
Note:
The following doesn’t aspire to cover all standard cmdlets.
Googling cmdlet names to find their help topics now shows you the PowerShell Core version of the topics by default; use the version drop-down list above the list of topics on the left to switch to a Windows PowerShell version.
Historically, the documentation frequently incorrectly claimed that ASCII is the default encoding in Windows PowerShell; fortunately, this has since been corrected.
Cmdlets that write:
Out-File and > / » create “Unicode” - UTF-16LE - files by default - in which every ASCII-range character (too) is represented by 2 bytes - which notably differs from Set-Content / Add-Content (see next point); New-ModuleManifest and Export-CliXml also create UTF-16LE files.
Set-Content (and Add-Content if the file doesn’t yet exist / is empty) uses ANSI encoding (the encoding specified by the active system locale’s ANSI legacy code page, which PowerShell calls Default).
Export-Csv indeed creates ASCII files, as documented, but see the notes re -Append below.
Export-PSSession creates UTF-8 files with BOM by default.
New-Item -Type File -Value currently creates BOM-less(!) UTF-8.
The Send-MailMessage help topic also claims that ASCII encoding is the default - I have not personally verified that claim.
Start-Transcript invariably creates UTF-8 files with BOM, but see the notes re -Append below.
Re commands that append to an existing file:
» / Out-File -Append make no attempt to match the encoding of a file’s existing content. That is, they blindly apply their default encoding, unless instructed otherwise with -Encoding, which is not an option with » (except indirectly in v5.1+, via $PSDefaultParameterValues, as shown above). In short: you must know the encoding of an existing file’s content and append using that same encoding.
Add-Content is the laudable exception: in the absence of an explicit -Encoding argument, it detects the existing encoding and automatically applies it to the new content.Thanks, js2010. Note that in Windows PowerShell this means that it is ANSI encoding that is applied if the existing content has no BOM, whereas it is UTF-8 in PowerShell Core.
This inconsistency between Out-File -Append / » and Add-Content, which also affects PowerShell Core, is discussed in GitHub issue #9423.
Export-Csv -Append partially matches the existing encoding: it blindly appends UTF-8 if the existing file’s encoding is any of ASCII/UTF-8/ANSI, but correctly matches UTF-16LE and UTF-16BE.
To put it differently: in the absence of a BOM, Export-Csv -Append assumes UTF-8 is, whereas Add-Content assumes ANSI.Start-Transcript -Append partially matches the existing encoding: It correctly matches encodings with BOM, but defaults to potentially lossy ASCII encoding in the absence of one.
Cmdlets that read (that is, the encoding used in the absence of a BOM):
Get-Content and Import-PowerShellDataFile default to ANSI (Default), which is consistent with Set-Content.
ANSI is also what the PowerShell engine itself defaults to when it reads source code from files.By contrast, Import-Csv, Import-CliXml and Select-String assume UTF-8 in the absence of a BOM, and so does the switch statement with its -File parameter.
