UnicodeDecodeError cause by system encoding

[WEGFan]

When I try to export to a static image, the following error occurs but I still can get the image correctly.

Exception in thread Thread-4:
Traceback (most recent call last):
  File "D:\Program Files\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "D:\Program Files\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "D:\My Documents\GitHub\github-readme-codestats-widget\venv\lib\site-packages\kaleido\scopes\base.py", line 74, in _collect_standard_error
    val = self._proc.stderr.readline().decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb9 in position 77: invalid start byte

So I debugged the line at val = self._proc.stderr.readline().decode('utf-8') and found that it should be decode as gb2312 because this is my system encoding.

I think this line should be changed to

os_encoding = locale.getpreferredencoding()
val = self._proc.stderr.readline().decode(os_encoding)

[jonmmease]

Hi @WEGFan, thanks a lot for letting us know. This section of code is using a background thread to read the standard error stream produced by Chromium. I'll need to dig into this bit more deeply, but your suggestion makes sense.

We are also explicitly using the UTF-8 encoding when writing JSON image export requests to standard in for the Kaleido C++ executable, and when reading the resulting JSON export results from standard out. Are you running into any issues with the resulting images? Have you tried the SVG format? Thanks!

[WEGFan]

The result images have no issues and exporting SVG format still getting the error.

[jonmmease]

Ok, great. Thanks for the info!

[jonmmease]

@WEGFan do get the same encoding from sys.stderr.encoding? This is a pattern I've seen some other projects use:

import sys
import locale
try:
    encoding = sys.stderr.encoding
except Exception:
    encoding = locale.getpreferredencoding()

getpreferredencoding actually has side-effects by default that I don't fully understand the implications of (https://stackoverflow.com/questions/23743160/locale-getpreferredencoding-why-does-this-reset-string-letters).

[WEGFan]

Nope.

>>> import sys
>>> import locale
>>> sys.stderr.encoding
'utf-8'
>>> locale.getpreferredencoding()
'cp936'

[WEGFan]

I took a look at Python 3 open() document and it says

In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.

And locale.getpreferredencoding() on Python 2

If invoking setlocale is not necessary or desired, do_setlocale should be set to False.

So maybe using locale.getpreferredencoding(False) is safe.

[jonmmease]

Thanks, I just opened #17 which should take care of it. It delays encoding until we actually need to display to the user, then it attemps to decode using sys.stderr.encoding and locale.getpreferredencoding(False), with exception handling.