Python 3.10: segfault due to invalid pointer

[eclipseo]

On 3.10 alpha 4, pillow fail to run and segfault with "invalid pointer". All projects depending on Pillow also fail to build.

python3 -c 'from PIL import Image'
free(): invalid pointer
Aborted (core dumped)

What are your OS, Python and Pillow versions?

• OS: Fedora Linux Rawhide
• Python: 3.10 alpha4
• Pillow: 8.1.0

[wiredfool]

Can you post the minimal code that you're using to trigger that?

Sorry, I see that it's just import image.

How did you install pillow?

[wiredfool]

It's very likely that there's something wrong with the build, because we do test on 3.10-dev in CI, and the test suite passes.

[eclipseo]

Sorry, I see that it's just import image.

How did you install pillow?

Pillow is being rebuilt as part of a mass-rebuild preparing for Python 3.10. The package python-pillow (https://src.fedoraproject.org/rpms/python%2Dpillow) is being built as usual:

https://src.fedoraproject.org/rpms/python-pillow/blob/master/f/python-pillow.spec

%build
# Build Python 3 modules
CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\
  /usr/bin/python3 setup.py  build --executable="/usr/bin/python3 -s"

%if 0%{?with_docs}
PYTHONPATH=$PWD/build/%py3_libbuilddir make -C docs html BUILDDIR=_build_py3 SPHINXBUILD=sphinx-build-%python3_version
rm -f docs/_build_py3/html/.buildinfo
%endif


%install
# Install Python 3 modules
install -d %{buildroot}/%{py3_incdir}/Imaging
install -m 644 src/libImaging/*.h %{buildroot}/%{py3_incdir}/Imaging
  CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\
  /usr/bin/python3 setup.py  install -O1 --skip-build --root BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 
  rm -rfv BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64/usr/bin/__pycache__


%check
# Check Python 3 modules
ln -s $PWD/Images $PWD/build/%py3_libbuilddir/Images
cp -R $PWD/Tests $PWD/build/%py3_libbuilddir/Tests
cp -R $PWD/selftest.py $PWD/build/%py3_libbuilddir/selftest.py
pushd build/%py3_libbuilddir
PYTHONPATH=$PWD %{__python3} selftest.py
popd

where RPM_OPT_FLAGS = -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
and RPM_LD_FLAGS = -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld

The downstream bug has more info: https://bugzilla.redhat.com/show_bug.cgi?id=1904379
with a Valgrind trace:

==117== Invalid free() / delete / delete[] / realloc()
==117==    at 0x483D9F1: free (vg_replace_malloc.c:538)
==117==    by 0x493CF01: _PyMem_RawFree (obmalloc.c:127)
==117==    by 0x493CF01: PyMem_RawFree (obmalloc.c:595)
==117==    by 0x493CF01: _PyObject_Free (obmalloc.c:1922)
==117==    by 0x493CF01: _PyObject_Free (obmalloc.c:1913)
==117==    by 0x4956999: _Py_Dealloc (object.c:2209)
==117==    by 0x4956999: _Py_DECREF (object.h:448)
==117==    by 0x4956999: _Py_XDECREF (object.h:514)
==117==    by 0x4956999: tupledealloc (tupleobject.c:276)
==117==    by 0x495AFF8: _Py_XDECREF (object.h:514)
==117==    by 0x495AFF8: code_dealloc (codeobject.c:658)
==117==    by 0x4957228: _PyDealloc (object.c:2209)
==117==    by 0x4957228: _Py_DECREF (object.h:448)
==117==    by 0x4957228: func_clear (funcobject.c:593)
==117==    by 0x495708D: func_dealloc (funcobject.c:614)
==117==    by 0x4947966: _Py_Dealloc (object.c:2209)
==117==    by 0x4947966: _Py_DECREF (object.h:448)
==117==    by 0x4947966: _Py_XDECREF (object.h:514)
==117==    by 0x4947966: free_keys_object (dictobject.c:628)
==117==    by 0x49EE0C9: type_clear (typeobject.c:3836)
==117==    by 0x4941864: delete_garbage (gcmodule.c:1018)
==117==    by 0x4941864: gc_collect_main (gcmodule.c:1301)
==117==    by 0x49EFD9B: _PyGC_CollectNoFail.isra.0 (gcmodule.c:2090)
==117==    by 0x49F2E74: interpreter_clear (pystate.c:323)
==117==    by 0x49F19DA: _Py_InterpreterState_Clear (pystate.c:355)
==117==    by 0x49F19DA: finalize_interp_clear (pylifecycle.c:1600)
==117==  Address 0x4b67f40 is 0 bytes inside data symbol "_Py_TrueStruct"

So the reference count of _Py_TrueStruct is reaching 0.

[radarhere]

I've created #5194 to resolve this. Are you able to test that at all?

[eclipseo]

I've created #5194 to resolve this. Are you able to test that at all?

I can confirm that the issue is solved. I can import PIL as expected and build packages depending on Pillow.

[hugovk]

I can also repro something similar on master and confirm it's fixed with the PR, with an older 3.10-dev from pyenv on macOS.

master

Python 3.10.0a0 (heads/master:5b24d15, Sep  2 2020, 09:22:20)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> exit()
python3.10(80947,0x113d475c0) malloc: *** error for object 0x10c038ab0: pointer being freed was not allocated
python3.10(80947,0x113d475c0) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    80947 abort      python3.10

PR

Python 3.10.0a0 (heads/master:5b24d15, Sep  2 2020, 09:22:20)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> exit()

Tests run with pytest pass in both cases.

[radarhere]

The fix for this should now be released as part of Pillow 8.1.1