Try using Kaleido v1 for image tests by emilykl · Pull Request #7412 · plotly/plotly.js

emilykl · 2025-04-30T18:10:23Z

No description provided.

ayjayt · 2025-05-29T15:34:10Z

Hi @emilykl, any interest in assigning me as well?

emilykl · 2025-05-29T16:15:40Z

@ayjayt You're welcome to try on your own branch if you like -- I believe this task is blocked by plotly/plotly.py#5207 though which I'm still testing

This reverts commit 279af99.

This reverts commit 97b5f29.

…s' baselines

emilykl · 2025-05-30T21:18:41Z

@ayjayt There are two figures which are failing consistently with an error in Kaleido v1 -- could you investigate?

These are the two figures:

gl3d_streamtube_reversed_ranges
mathjax

(There are a few others which are failing intermittently but I think that's due to instability in the pipeline, not anything to do with Kaleido)

Here is an archive containing:

The JSON figure representations for the failing figures
A script which attempts to generate an image from each, and prints the traceback on failure.

failing_figures_kaleido_v1.zip

You'll need to pip install a few things in your environment to run the script:

python -m pip install "git+https://github.com/plotly/plotly.py.git@6837831"
python -m pip install "git+https://github.com/plotly/[email protected]#subdirectory=src/py"
python -m pip install numpy

Below is the output generated by running python generate_images.py from inside the root directory of the archive:

Error output

(make_baseline) (base) ekl@Emilys-MacBook-Air-2 failing_figures_kaleido_v1 % python generate_images.py
=============
=== Failed to generate image for gl3d_streamtube_reversed_ranges ===
=============
Traceback (most recent call last):
  File "generate_images.py", line 32, in <module>
    pio.write_image(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/plotly/io/_kaleido.py", line 509, in write_image
    img_data = to_image(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/plotly/io/_kaleido.py", line 379, in to_image
    img_bytes = kaleido.calc_fig_sync(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 145, in calc_fig_sync
    return _async_thread_run(calc_fig, args=args, kwargs=kwargs)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 138, in _async_thread_run
    raise res
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 129, in run
    q.put(asyncio.run(func(*args, **kwargs)))
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 55, in calc_fig
    return await k.calc_fig(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/kaleido.py", line 319, in calc_fig
    spec, full_path = build_fig_spec(fig, path, opts)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/_fig_tools.py", line 144, in build_fig_spec
    name = _next_filename(directory, prefix, ext)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/_fig_tools.py", line 94, in _next_filename
    re_number = re.compile(r"^" + prefix + r"-(\d+)\." + ext + r"$")
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/re.py", line 252, in compile
    return _compile(pattern, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 598, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range _-2 at position 94
=============
=============
=== Failed to generate image for mathjax ===
=============
Traceback (most recent call last):
  File "generate_images.py", line 32, in <module>
    pio.write_image(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/plotly/io/_kaleido.py", line 509, in write_image
    img_data = to_image(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/plotly/io/_kaleido.py", line 379, in to_image
    img_bytes = kaleido.calc_fig_sync(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 145, in calc_fig_sync
    return _async_thread_run(calc_fig, args=args, kwargs=kwargs)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 138, in _async_thread_run
    raise res
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 129, in run
    q.put(asyncio.run(func(*args, **kwargs)))
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/__init__.py", line 55, in calc_fig
    return await k.calc_fig(
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/kaleido.py", line 319, in calc_fig
    spec, full_path = build_fig_spec(fig, path, opts)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/_fig_tools.py", line 144, in build_fig_spec
    name = _next_filename(directory, prefix, ext)
  File "/Users/ekl/code/plotly.js/venv/make_baseline/lib/python3.8/site-packages/kaleido/_fig_tools.py", line 94, in _next_filename
    re_number = re.compile(r"^" + prefix + r"-(\d+)\." + ext + r"$")
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/re.py", line 252, in compile
    return _compile(pattern, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 525, in _parse
    code = _escape(source, this, state)
  File "/Users/ekl/.local/share/uv/python/cpython-3.8.20-macos-aarch64-none/lib/python3.8/sre_parse.py", line 426, in _escape
    raise source.error("bad escape %s" % escape, len(escape))
re.error: bad escape \h at position 3
=============

ayjayt · 2025-06-02T17:06:39Z

@emilykl

plotly/Kaleido#339 makes it work for me

here's a uv script preamble to build in the build steps (will redownload stuff) right in generate_images

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "kaleido",
#     "numpy",
#     "plotly",
# ]
#
# [tool.uv.sources]
# plotly = { git = "https://github.com/plotly/plotly.py.git", rev = "6837831" }
# ///

archmoj · 2025-06-04T14:03:01Z

@emilykl After I viewed most significant changes on my dev branch I suggest you drop the a426afa commit (possibly by force pushing i.e. to reduce the size of changes) and instead commit the changes of the following baselines:

test/image/baselines/mathjax.png
test/image/baselines/ternary-mathjax-title-place-subtitle.png
test/image/baselines/legend_mathjax_title_and_items.png
test/image/baselines/world-cals.png

test/image/baselines/worldcup.png
test/image/baselines/funnel_horizontal_group_basic.png
test/image/baselines/font-decorline-scatter.png
test/image/baselines/text_on_shapes_basic.png
test/image/baselines/indicator_scatter.png

Thank you 🙏

emilykl · 2025-07-28T17:00:23Z

Closing in favor of #7498.

emilykl added 2 commits April 30, 2025 14:10

try using Kaleido v1 for image tests

9a078e3

Update Plotly version

f7f5a03

gvwilson assigned emilykl May 8, 2025

gvwilson added P2 considered for next cycle infrastructure build process etc. labels May 8, 2025

emilykl added 2 commits May 27, 2025 17:13

Merge branch 'master' into use-new-kaleido-for-image-tests

5595393

update version

afafaf2

emilykl added 3 commits May 29, 2025 15:33

update to new pio API and use unreleased Plotly and Kaleido

8c84e4b

typo

14298d8

install chrome for make-baselines jobs

09a0f4f

emilykl assigned ayjayt May 29, 2025

emilykl added 17 commits May 29, 2025 16:07

add apt-get update step

94335b8

does it work with '-browsers' images?

86f0a3e

use cimg instead of circleci

30c39f4

use python instead of python3

279af99

Revert "use python instead of python3"

ecb2c0c

This reverts commit 279af99.

ensurepip

97b5f29

Revert "ensurepip"

18f0a67

This reverts commit 97b5f29.

'python' rather than 'sudo python3'

3f13771

try python3 without sudo

fae9f08

pip install requests

d0fa346

update kaleido ref

03a53b6

temporarily blacklist 'gl3d_streamtube_reversed_ranges' baseline

99eb4b5

temporarily blacklist 'mathjax' baseline

fcafd40

temporarily blacklist 'map_predefined-styles2' baseline

41172b3

remove eps from make_exports

0ec3ba1

temporarily blacklist 'map_predefined-styles1' and 'grid_subplot_type…

ff45414

…s' baselines

temporarily blacklist 'map_fonts-supported-metropolis' baseline

340fd9e

typo

4e518a2

ayjayt mentioned this pull request Jun 2, 2025

Escape user-supplied strings regex plotly/Kaleido#339

Merged

emilykl added 5 commits June 2, 2025 16:35

try with new kaleido with bugfix

2011839

update kaleido version

80fe5cf

temporarily blacklist 'map_fonts-supported-open-sans-weight' baseline

5857a70

typo

53e26b9

temporarily blacklist 'map_fonts-supported-metropolis-weight' baseline

430b195

use single Kaleido instance for making baselines (faster performance)

6bab84d

emilykl force-pushed the use-new-kaleido-for-image-tests branch from b1f16e0 to 6bab84d Compare June 4, 2025 14:58

emilykl added 2 commits June 4, 2025 11:41

commit new baselines with significant changes

4d4d34f

update plotly version in env_image.sh

86f5283

archmoj mentioned this pull request Jul 7, 2025

Andrew/new kaleido #7241

Closed

archmoj mentioned this pull request Jul 17, 2025

Run image tests and test-exports on new CircleCI's cimg/python new system #7471

Merged

Merge branch 'master' into use-new-kaleido-for-image-tests

9085ed5

emilykl closed this Jul 28, 2025

archmoj deleted the use-new-kaleido-for-image-tests branch July 28, 2025 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Try using Kaleido v1 for image tests#7412

Try using Kaleido v1 for image tests#7412
emilykl wants to merge 34 commits intomasterfrom
use-new-kaleido-for-image-tests

emilykl commented Apr 30, 2025

Uh oh!

ayjayt commented May 29, 2025

Uh oh!

emilykl commented May 29, 2025

Uh oh!

emilykl commented May 30, 2025

Uh oh!

ayjayt commented Jun 2, 2025 •

edited

Loading

Uh oh!

archmoj commented Jun 4, 2025

Uh oh!

emilykl commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

emilykl commented Apr 30, 2025

Uh oh!

ayjayt commented May 29, 2025

Uh oh!

emilykl commented May 29, 2025

Uh oh!

emilykl commented May 30, 2025

Uh oh!

ayjayt commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

archmoj commented Jun 4, 2025

Uh oh!

emilykl commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ayjayt commented Jun 2, 2025 •

edited

Loading