Clarify UTF8 and UTF8BOM/UTF8NoBOM #4021

Activity

1. sdwheeler commented on Mar 20, 2019

   

   Sean Wheeler (sdwheeler)

   on Mar 20, 2019

   Contributor

   More actions

   The change is covered in the release notes and there is a detailed article about encoding.

   • https://docs.microsoft.com/en-us/powershell/scripting/whats-new/what-s-new-in-powershell-core-60?view=powershell-6#default-encoding-is-utf-8-without-a-bom-except-for-new-modulemanifest
   • https://docs.microsoft.com/en-us/powershell/scripting/components/vscode/understanding-file-encoding?view=powershell-6
2. Sean Wheeler (sdwheeler)

   closed this as completedon Mar 20, 2019
3. Sean Wheeler (sdwheeler)

   added

   resolution-answeredStatus - closed by existing documentationStatus - closed by existing documentation

   on Mar 20, 2019
4. willxie-eng commented on Mar 20, 2019

   

   William Xie (willxie-eng)

   on Mar 20, 2019

   Author

   More actions

   Thanks for the links, Sean! My gripe was that there's both UTF8 and UTF8NoBOM, which feels like UTF8NoBOM is redundant.

   👍React with 👍2Reacted by hilari0n and Alexandr Buhlak
5. sdwheeler commented on Mar 20, 2019

   

   Sean Wheeler (sdwheeler)

   on Mar 20, 2019

   Contributor

   More actions

   UTF8 and UTF8NoBOM are different. UTF8 has a byte-order-mark (BOM) at the beginning of the file. UTF8NoBOM does not. The BOM is not always compatible across applications and platforms.
6. arfmach commented on Jan 3, 2020

   

   Alexandre Machado (arfmach)

   on Jan 3, 2020

   More actions

   In fact, UTF8BOM is not recognized by Out-File cmdlet. I'm using PowerShell version 5.1.18362.145 and the output is

   Out-File : Não é possível validar o argumento no parâmetro 'Encoding'. O argumento "UTF8BOM" não pertence ao conjunto "unknown;string;unicode;bigendianunicode;utf8;utf7;utf32;ascii;default;oem" especificado pelo atributo ValidateSet.
   Forneça um argumento que esteja no conjunto e tente o comando novamente.
   No linha:1 caractere:20
   + Out-File -Encoding UTF8BOM Teste.txt
   +                    ~~~~~~~
       + CategoryInfo          : InvalidData: (:) [Out-File], ParameterBindingValidationException
       + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.OutFileCommand
   

   👍React with 👍1Reacted by Kent Rasmussen
7. hilari0n commented on Mar 1, 2021

   

   hilari0n

   on Mar 1, 2021

   Last edited by hilari0n

   More actions

   I don't understand why this question was dismissed/closed. I don't see any comments actually explaining/addressing the issue.
   The documentation was mentioning 3 distinct options (it was for PowerShell 6, I believe), and it still does for PowerShell versions 7.0, 7.1 and 7.2 (emphasis mine):

   • utf8: Encodes in UTF-8 format.
   • utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
   • utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)

   None of those document versions mention, if the first option ("utf8") will encode with or without Byte Order Mark (or if this behavior is dependent on platform or whatever).
   The documents linked by Sean Wheeler (@sdwheeler) describe that the default encoding has changed (to "UTF8NoBOM") and how encoding works in PowerShell in general. None of those two mention if "utf8" encoding in PowerShell uses BOM or not. The second document mentions "utf8" as one without BOM, but in the VSCode context, not PowerShell.
   This document: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.1#character-encoding-in-windows-powershell
   Mentions "UTF8" encoding as one with BOM, but it actually states, that this is for PowerShell 5.1, so it fails to list "utf8BOM" or "utf8NoBOM". In that light it's hard to assess if it applies to PowerShell 7+ in any way.

   All in all, there seems to be no consistent and clear document for PowerShell 7+ addressing the issue raised by @WilliamXieMSFT.
   If I'm mistaken, please share the links or appropriate quotes from already linked documents.

   Edit:
   And just as I have posted, I have found some relevant mention in the document I have linked - in a section for PowerShell Core, below the one I linked before.
   It does explain how BOM works for all 3 options:

   • utf8: Encodes in UTF-8 format (no BOM).
   • utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
   • utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)

   I fail to understand why documentation for Out-File (and others, e.g. Export-Csv) can't also be clear on that, i.e. that the option utf8 does not use BOM for PowerShell Core (and it did use BOM for Windows PowerShell). Esp. that this option apparently underwent an important change.

   👍React with 👍2Reacted by MikeRosoft and the-working-rene
8. me-suzy commented on May 23, 2021

   

   Doar EU (me-suzy)

   on May 23, 2021

   More actions

   hello. I try to convert a file from UTF-8 to UTF-8 BOM, and the code in Powershell is not working, gat an error that say:

   "Unable to match the identifier name
   utf8NoBOM to a valid enumerator name. Specify one of the following enumerator names and try again:
   Unknown, String, Unicode, Byte, BigEndianUnicode, UTF8, UTF7, UTF32, Ascii, Default, Oem, BigEndianUTF32""

   This is the powershell code:

   $a = "C:/Folder1/TEST_ro.txt"
    $b = "C:/Folder1/TEST_ro-2.txt"
    (Get-Content -path $a) | Set-Content -Encoding UTF8BOM -Path $b
   
9. sdwheeler commented on May 23, 2021

   

   Sean Wheeler (sdwheeler)

   on May 23, 2021

   Last edited by sdwheeler

   Contributor

   More actions

   Doar EU (@me-suzy) PowerShell 6+ Supports the following encodings:

   • ascii: Uses the encoding for the ASCII (7-bit) character set.
   • bigendianunicode: Encodes in UTF-16 format using the big-endian byte order.
   • bigendianutf32: Encodes in UTF-32 format using the big-endian byte order.
   • oem: Uses the default encoding for MS-DOS and console programs.
   • unicode: Encodes in UTF-16 format using the little-endian byte order.
   • utf7: Encodes in UTF-7 format.
   • utf8: Encodes in UTF-8 format.
   • utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
   • utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)
   • utf32: Encodes in UTF-32 format.

   Windows PowerShell 5.1 (and earlier) supports:

   • Ascii Uses ASCII (7-bit) character set.
   • BigEndianUnicode Uses UTF-16 with the big-endian byte order.
   • BigEndianUTF32 Uses UTF-32 with the big-endian byte order.
   • Byte Encodes a set of characters into a sequence of bytes.
   • Default Uses the encoding that corresponds to the system's active code page (usually ANSI).
   • Oem Uses the encoding that corresponds to the system's current OEM code page.
   • String Same as Unicode.
   • Unicode Uses UTF-16 with the little-endian byte order.
   • Unknown Same as Unicode.
   • UTF7 Uses UTF-7.
   • UTF8 Uses UTF-8. (with BOM)
   • UTF32 Uses UTF-32 with the little-endian byte order.

   Note that this does not include utf8NoBOM.

   👍React with 👍1Reacted by MikeRosoft
10. me-suzy commented on May 23, 2021

    

    Doar EU (me-suzy)

    on May 23, 2021

    Last edited by me-suzy

    More actions

    ok, I search on internet, and I find 2 SOLUTIONS. Very easy, one by using REGEX , the other is done by Python Script, just using Notepad++

    https://community.notepad-plus-plus.org/topic/21200/change-save-encoding-how-to-convert-800-txt-files-utf-8-to-utf-8-bom
11. MikeRosoft commented on Jan 10, 2022

    

    MikeRosoft

    on Jan 10, 2022

    More actions

    To fully answer the question (by directly testing the following commands in powershell.exe and pwsh.exe on my system):

    $data=@{}
    $data.Foo='Foo'
    $data.Bar='Bar'
    $data | ConvertTo-Json | Out-File 'c:\temp\data.json' -Encoding utf8
    

    In Windows Powershell (version 5.1) this writes the file with BOM.
    In Powershell Core (version 7.2.1) this writes the file without BOM.

    👍React with 👍1Reacted by Olav Rønnestad Birkeland
12. the-working-rene commented on Aug 20, 2024

    

    the-working-rene

    on Aug 20, 2024

    More actions

    In the current documentation for the Set-Content and Add-Content commands, it is still not clear, what exactly "utf8" means (with or without BOM).

    utf8: Encodes in UTF-8 format.
    utf8BOM: Encodes in UTF-8 format with Byte Order Mark (BOM)
    utf8NoBOM: Encodes in UTF-8 format without Byte Order Mark (BOM)

    Thanks to the comments here and after trying out, "utf8" is equivalent to "utf8NoBOM" - a little hint for that behavior would be really helpful in those documentations. Especially because the behavior changed between PowerShell 5.1 and PowerShell 6.