Skip to content

[BugFix] Read build-hook generator output as UTF-8 so dev_install works on CJK Windows - #7830

Open
b84330808 wants to merge 1 commit into
openbq-org:developfrom
b84330808:bugfix/build-hook-utf8-output
Open

b84330808 wants to merge 1 commit into
openbq-org:developfrom
b84330808:bugfix/build-hook-utf8-output

Conversation

@b84330808

Copy link
Copy Markdown

Pull Request the OpenBB Platform

Description

  • Summary of the change/ bug fix.

On Windows with a CJK code page (here cp932, Japanese), python openbb_platform/dev_install.py stops while building openbb-ecb:

  File "...\openbb_platform\providers\ecb\hatch_build.py", line 109, in initialize
    for line in proc.stdout:
UnicodeDecodeError: 'cp932' codec can't decode byte 0x94 in position 132: illegal multibyte sequence

The bls, ecb, eia, imf and oecd build hooks run the cache generator with text=True, so they decode its output with the locale encoding. uv runs PEP 517 builds with PYTHONIOENCODING=utf-8:backslashreplace, so the generator writes UTF-8, and the first non-ASCII character breaks the read. For ECB and IMF that is the em dash in the final summary line, after the whole download (about six minutes for ECB). cp936, cp949 and cp950 fail the same way. Without uv's PYTHONIOENCODING, the generator writes to the pipe with cp932 instead, and print of that em dash raises UnicodeEncodeError in the generator.

This sets PYTHONIOENCODING=utf-8 for the generator and reads its output with encoding="utf-8", errors="replace" in all five hooks.

How has this been tested?

Built the ECB wheel with uv build --wheel from a copy of openbb_platform/providers/ecb without the cache file, on Windows 11 with the cp932 code page:

  • develop: fails with the UnicodeDecodeError above (exit 2).

  • this branch: Successfully built openbb_ecb-2.0.0-py3-none-any.whl, with the summary line printed intact.

  • All five hooks: ran initialize() with a stub generator that prints the same kind of em dash line under PYTHONIOENCODING=utf-8:backslashreplace, as uv sets it. On develop each raises UnicodeDecodeError: 'cp932' codec can't decode byte 0x94; on this branch each reads the output and marks the cache file.

ruff check and ruff format --check pass on the five files.

Checklist

  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have adhered to the GitFlow naming convention and my branch name is in the format of feature/feature-name or hotfix/hotfix-name.
  • I ensure that I am following the CONTRIBUTING guidelines.

The bls, ecb, eia, imf and oecd build hooks read the cache generator's
output with the locale encoding. uv runs PEP 517 builds with
PYTHONIOENCODING=utf-8, so the generator writes UTF-8, and on Windows
with a CJK code page (cp932, cp936, cp949, cp950) the hook fails with
UnicodeDecodeError as soon as a line holds a non-ASCII character, such
as the em dash in the ECB and IMF summary lines. Building these
providers from source (dev_install.py) then aborts.

Run the generator with PYTHONIOENCODING=utf-8 and decode its output as
UTF-8, replacing undecodable bytes, so the build no longer depends on
the console code page.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant