Jpeg load performance regression for Pillow 5.1+ from wheels

[kkopachev]

What did you do?

Performance regression when opening (and loading) a jpeg file in Pillow, installed from wheels on x86_64 linux. Affects loading of jpeg files. Regression happened updating from Pillow 5.0.0 to 5.1.0 (up to 5.3.0)

What are your OS, Python and Pillow versions?

• OS: linux, x86_64
• Python: 2.7, 3.7
• Pillow: 5.1.0+

I am not sure where this issue belongs to, seems to be related to wheels building somehow.

Original issue was created in thumbor repo thumbor/thumbor#1102 saying there was a noticeable performance regression. Upon further investigation, I was able to come up with simple reproduction steps.

docker run --rm -it python:2 bash
root@9e6e1f13f44a:/# wget -q wget https://upload.wikimedia.org/wikipedia/commons/b/b4/JPEG_example_JPG_RIP_100.jpg
root@9e6e1f13f44a:/# pip install pillow==5.0.0
Collecting pillow==5.0.0
  Downloading https://files.pythonhosted.org/packages/89/bd/1d9a10f3e8157b7df275740b0782a892a0db387f8286620110c41e5146c7/Pillow-5.0.0-cp27-cp27mu-manylinux1_x86_64.whl (5.8MB)
    100% |████████████████████████████████| 5.9MB 1.5MB/s 
Installing collected packages: pillow
Successfully installed pillow-5.0.0
You are using pip version 18.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@9e6e1f13f44a:/# wget https://upload.wikimedia.org/wikipedia/commons/b/b4/JPEG_example_JPG_RIP_100.jpg
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/# python -m timeit -n1000 -r5 -s'from PIL import Image' 'Image.open("JPEG_example_JPG_RIP_100.jpg").load()'
1000 loops, best of 5: 2.58 msec per loop
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/# pip install pillow==5.1.0
Collecting pillow==5.1.0
  Using cached https://files.pythonhosted.org/packages/00/49/a0483e7308b4b04b5a898789911dbb876d9fea54e7df0453915e47744cfd/Pillow-5.1.0-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: pillow
  Found existing installation: Pillow 5.0.0
    Uninstalling Pillow-5.0.0:
      Successfully uninstalled Pillow-5.0.0
Successfully installed pillow-5.1.0
You are using pip version 18.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/#
root@9e6e1f13f44a:/# python -m timeit -n1000 -r5 -s'from PIL import Image' 'Image.open("JPEG_example_JPG_RIP_100.jpg").load()'
1000 loops, best of 5: 5.77 msec per loop
root@9e6e1f13f44a:/#

In short - Pillow 5.1.0 takes 5.77 msec to open test JPEG file. However Pillow 5.0.0 took 2.58 msec to open the same file.

I did try to build both Pillow versions from source (only enabling libjpeg) and in that case they show same performance.
I thought maybe for Pillow 5.1 you switched to libjpeg9c, so I built both Pillow versions with libjpeg9c - no performance difference.

As people report in original thumbor ticket, they are not able to repro it on OSX, so I assume it's linux only.

As a last resort, I copied libjpeg.so from what was shipped with wheels for Pillow 5.0.0 and copied it over to Pillow 5.1 installed from wheels - no performance difference.

I suspect something wrong with how libjpeg got build starting from Pillow 5.1.0. I tried 5.2.0 and 5.3.0 - both experience same issue.

[hugovk]

I can reproduce on Linux (Ubuntu/Travis CI) and not on macOS (High Sierra).

msec per loop:

Pillow version	2.7 wheel	2.7 source	3.7 wheel	3.7 source
5.0.0	2.75	2.6	n/a	2.94
5.1.0	5.41	2.58	n/a	2.95
5.2.0	5.45	2.61	5.67	2.94
5.3.0	5.44	2.62	5.69	2.94

https://travis-ci.org/hugovk/test/builds/447090663

Wheels are built on https://github.com/python-pillow/pillow-wheels (see
https://github.com/python-pillow/Pillow/blob/master/RELEASING.md for the full process).

• 5.0.0 was released on 2018-01-02 (python-pillow/pillow-wheels@31f13e7)
• 5.1.0 was released on 2018-04-02 (python-pillow/pillow-wheels@1cab7ad)

Here's the diff:

• https://github.com/python-pillow/pillow-wheels/compare/31f13e7058314a5c8ccbae85716bfcebe39bce1d..1cab7ad46cad74bbbf62767bbde6100d3993b0bf
• FreeType 2.8.1 -> 2.9
• multibuild was updated (2017-03-30 -> 2018-02-28)

[kkopachev]

I noticed that .so files for most of libraries in 5.1.0 are smaller than same libs included in 5.0.0

Library	5.0.0	5.1.0
libfreetype	3691288	1254200
libjpeg	1340024	344704
liblcms2	1665552	453104
liblzma	867928	216664
libopenjp2	343024	310256
libpng16	943336	279768
libtiff	1783352	636464
libwebp	3439312	723664
libwebpdemux	95048	29512
libwebpmux	214240	58600
libz	87848	87848

[hugovk]

@homm, Just to let you know, on Linux, installing 5.1.0+ from wheel has slower JPEG load performance than when building from source.

@matthew-brett, Hello, do you think this might be caused by a change in multibuild?

[matthew-brett]

Might be - but I can't think what might have caused that. Have you tried going back to earlier versions of the JPEG libraries?

[hugovk]

We updated JPEG_VERSION from 9b to 9c in May, but this problem showed up between our 2 January and 2 April releases.

python-pillow/pillow-wheels@d6bb3fb

OPENJPEG_VERSION=2.1 hasn't changed since switching to multibuild two years ago.

https://github.com/python-pillow/pillow-wheels/blame/master/config.sh

[matthew-brett]

I'm sorry - I don't know what could have caused that. I can't see anything obvious in the commits between 2 January and 2 April. Maybe worth clearing the ccaches with the travis interface, and trying a rebuild? Could it be the Docker image? Have the Mac builds slowed down as well?

[hugovk]

I cleared the Travis caches for https://travis-ci.org/python-pillow/Pillow/

There are no caches for https://travis-ci.org/python-pillow/pillow-wheels or https://travis-ci.org/python-pillow/docker-images

---

Actually, the Mac wheels are consistent for all versions 5.0.0 -> 5.3.0, and also when building from source, but wheels are about twice as slow as source.

Pillow version	2.7 wheel	2.7 source	3.7 wheel	3.7 source
4.0.0	6.96	3.22	n/a	3.45
4.1.0	6.87	3.15	n/a	3.18
4.2.0	6.98	3.24	n/a	3.36
4.3.0	6.71	3.16	n/a	3.2
5.0.0	7.56	3.31	n/a	3.84
5.1.0	8.05	3.26	n/a	3.37
5.2.0	7.88	3.57	7.55	3.61
5.3.0	7.2	3.5	7.35	3.59

[matthew-brett]

Is it possible that the JPEG library is using things like AVX intrinsics, and so on, when built from source? Is the compiler the same? I mean, maybe clang has got faster, and you see that on your machine, but we are using an old clang on the OSX images we use for building? The gcc on the Manylinux docker image is pretty old too.

[hugovk]

My Mac when building source:

$ clang --version
Apple LLVM version 10.0.0 (clang-1000.10.44.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Linux / Travis / Python 2.7 when building source:

gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

Linux / Travis / Python 3.7 when building source:

gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

[kkopachev]

I rebuild libjpeg passing same CFLAGS as multibuild does and was able to get regression.
https://github.com/matthew-brett/multibuild/blob/devel/configure_build.sh#L30-L32
This also explains why it affects Linux wheels on Pillow 5.1.0+ (as multibuild was updated right before that release) and also explains why OSX wheels has consistently same bad performance (OSX build was always including these flags)

[matthew-brett]

Do you mean that -Wl,-strip-all in CFLAGS introduces the regression, on its own?