Terminal should force pseudoconsole host into UTF-8 codepage by default

It's 2019, after all. Maybe we should introduce a flag that starts up the pseudoconsole host in codepage 65001 so that we make good on our promise of "emoji just work and everything else works like it should too," and use WT as a real opportunity to push the boundaries here.

  🌛
💪 💪
  👖

maintainer note, Aug 2023

It's {{current_year}}, after all

Also, we want to take into account arbitrary codepages, ala #15678

I_am_okay_with_this.jpg

Does cmd support batch scripts in codepage 65001 now?

+1 for running *nix tools with CJK outputs

For example: fc-list in texlive

A work-around in the meanwhile is ... enable BETA: Use Unicode UTF-8 for worldwide language support in the "Control Panel \ Clock and Region \ Region \ Administrative \ Change system locale..." dialog box and rebooting. Do note the BETA prefix.

I'm on board with this, esp. if we add a "disableAutoCp65001" ^{(boy that needs a better name)} setting to disable this behavior, set to false by default.

Maybe we need to somehow enable localized messages in CMD while codepage is 65001.
Now it is forced to be English.

A work-around in the meanwhile is ... enable BETA: Use Unicode UTF-8 for worldwide language support in the "Control Panel \ Clock and Region \ Region \ Administrative \ Change system locale..." dialog box and rebooting. Do note the BETA prefix.

This changes the system code page to 65001, if you have any ANSI application, they will be forced to use UTF-8.
It should be fine for English users, but for CJK users, that will be a big trouble.

Is there any plan to add an environment variable for "UTF-8 mode"?

For example, Python has PYTHONUTF8. Git for Windows uses LANG. Some tools use the GetConsoleCP.
It seems there is no standard way to indicate "I want to use UTF-8 in this session!".

Is the ConsoleCP the best way?
When we launch a GUI application from console, the ConsoleCP can be used too?

I still want to know how many real world console apps are there expects legacy codepages (especially ones that outputs legacy codepage strings like GBK), and is it a good idea to break these.

Because that's why system-wide UTF-8 is still labeled as beta.

Given that many modern Windows command line apps are ported from the *nix world, and the modern principle seems to be command line apps should use English, I guess it is acceptable?

I still want to know how many real world console apps are there expects legacy codepages (especially ones that outputs legacy codepage strings like GBK), and is it a good idea to break these.

Because that's why system-wide UTF-8 is still labeled as beta.

System wide setting breaks legacy applications. That's why we need per-session option instead.

Given that many modern Windows command line apps are ported from the *nix world, and the modern principle seems to be command line apps should use English, I guess it is acceptable?

Some modern CLI applications (Python and Go) use WriteConsoleW to write to console. But Python still use legacy encoding for PIPE.
And some modern applications from Unix use UTF-8 always. Cygwin/MSYS also use UTF-8 (they use environment variables like LANG).

To use such applications, we want UTF-8 session in VSCode terminal and Windows Terminal.

Since std input works well with UTF-8 encoding today #14745, may be it's worth to add some sort of a syntactic sugar to push/pop the initial state of the system code page for new Windows console applications? Wrap it up in a single API call or put it into a specific header file (e.g. <iostream>) to reduce following boilerplate code at the beginning

#define UTF8_EVERYWHERE
#include <iostream>
#include <string>

// Put this block inside <iostream> on windows?
#ifdef _WIN32
    #ifdef UTF8_EVERYWHERE
        #include "windows.h"
        namespace winapi_cp_state
        {
            static UINT ou_state = GetConsoleOutputCP(); // Save original system code pages.
            static UINT in_state = GetConsoleCP();       //
            static void set_page(UINT out, UINT in) { SetConsoleOutputCP(out); SetConsoleCP(in); }
            static void set_page() { set_page(ou_state, in_state); }
            static int _state = (set_page(CP_UTF8, CP_UTF8), ::atexit(set_page)); // Set to UTF-8 and always restore original system code pages at exit.
        }
    #endif
#endif

// x-platform code
int main()
{
    std::cout << "Test: あああ🙂🙂🙂日本👌中文👍Кириллица\n"; // Make sure you save your project file with 65001(UTF-8) encoding.
    std::cout << "Enter text: ";
    std::string utf8;
    std::cin >> utf8;
    std::cout << "UTF-8 text: " << utf8 << std::endl;
    return 0;
}

The #define UTF8_EVERYWHERE key is used to indicate the programmer’s intention to use UTF-8 encoding instead of original system code page.
This would be extremely helpful for newbie console programmers. All of them are completely confused with text encodings. In addition, cross-platform compatibility is achieved automatically.

xref some discussion in #15504

pretty sure our plan was to do:

compatibility.defaultToutf8 in the Terminal settings, which sets
a --flag to conpty to tell it to default to CP65001 instead.

Now it's just a matter of plumbing, and deciding if we really want to do the --flags thing or the --defaultToUtf8 thing.

More notes

conhost --codepage 65001 to start in utf-8, or accept an arbitrary one.
conhost --codepage WITHOUT AN ARG to use the one in HKCU/Console/Codepage

feat: add IME support infrastructure and Win32 test harness

It's a great pain in the neck that we have to add the following code to every $PSPROFILE:

[Console]::InputEncoding = [Console]::OutputEncoding = [System.Text.Encoding]::UTF8

Coding Agent CLIs abuse Unicode-only characters, so the priority of this issue has been increased recently.

"BETA: Use Unicode UTF-8 for worldwide language support" is still dangerous if you rely on non-OSS (especially proprietary especially legacy) applications. If you could stop relying on such applications, you should consider ditching Windows and migrating to Linux-based OSes. 🤥

Add configurable code pages to terminal profiles

Navigation Menu

Repository navigation

Terminal should force pseudoconsole host into UTF-8 codepage by default #1802

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Terminal should force pseudoconsole host into UTF-8 codepage by default #1802

Description

Activity

miniksa commented on Jul 3, 2019

MicheleCicciottiWork commented on Jul 4, 2019

hcoona commented on Jul 5, 2019

rivy commented on Jul 6, 2019

zadjii-msft commented on Jul 8, 2019

driver1998 commented on Sep 26, 2019

driver1998 commented on Oct 4, 2019

methane commented on Jan 22, 2020

20 remaining items

driver1998 commented on Jul 26, 2020

methane commented on Jul 27, 2020

o-sdn-o commented on May 20, 2023

zadjii-msft commented on Jul 5, 2023

tats-u commented on Apr 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions