Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

Synapse's docker images (and CI?) do not pin the specific build requirements of poetry #13849

Closed as not planned
Closed as not planned
Synapse's docker images (and CI?) do not pin the specific build requirements of poetry#13849
Labels
A-DockerDocker images, or making it easier to run Synapse in a container.O-UncommonMost users are unlikely to come across this or unexpected workflowS-MinorBlocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.X-RegressionSomething broke which worked on a previous release

Description

@DMRobertson
DMRobertson
opened on Sep 20, 2022

Synapse's docker file installs a specific version of poetry in build stage 0:

synapse/docker/Dockerfile

Lines 58 to 61 in 08297f2

# We install poetry in its own build stage to avoid its dependencies conflicting with
# synapse's dependencies.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --user "poetry==1.3.2"

and uses it to export a requirements.txt file for the other stages to use (without having to install poetry).

synapse/docker/Dockerfile

Lines 78 to 85 in 08297f2

# Export the dependencies, but only if we're actually going to use the Poetry lockfile.
# Otherwise, just create an empty requirements file so that the Dockerfile can
# proceed.
RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
/root/.local/bin/poetry export --extras all -o /synapse/requirements.txt ${TEST_ONLY_SKIP_DEP_HASH_VERIFICATION:+--without-hashes}; \
else \
touch /synapse/requirements.txt; \
fi

The former step is not fully reproducible1, because pip will use the latest version of poetry's build-time dependencies to install poetry. (As of poetry 1.3.2 those dependencies are the unbounded range poetry-core >= 1.1.0, see here). I advised the reporter to patch the Dockerfile in order to get Synapse 1.61 built again.

We saw this manifest in the report below: a version of poetry-core was released under which previous versions of poetry would fail to build, see #13849 (comment).

It seems like the Python packaging ecosystem doesn't have an obvious way to pin build-time dependencies. (See for instance https://discuss.python.org/t/pinning-build-dependencies/8363 or pypa/pip#9542.) The best approach that I can see is described in pypa/pip#9542 (comment), namely

  • install the build-time dependencies into the environment you're interested in
  • pip install --no-build-isolation ...

Original issue:

Unable to build v1.61.0 docker image with clean caches

Just tried to build v1.61.0 on Linux and Windows and both fail.

I know this worked a few weeks ago and I basically think that OP was right with the version pining.

Motivation: I want to bisect and thus build v1.61.0 .. v1.62.0 for identifying a patch which destroyed the generate call for us (I'll open a new ticket for that one).

Here is the error:

git checkout tags/v1.61.0
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile -t fred .
[+] Building 76.9s (14/25)
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.1s
 => => transferring dockerfile: 5.28kB                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.1s
 => => transferring context: 170B                                                                                                                                                                                                       0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                              1.3s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                                                                        0.0s
 => docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                1.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc                                                                                                                    0.0s
 => => sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 2.00kB / 2.00kB                                                                                                                                          0.0s
 => => sha256:ad87fb03593d1b71f9a1cfc1406c4aafcb253b1dabebf569768d6e6166836f34 528B / 528B                                                                                                                                              0.0s
 => => sha256:1e8a16826fd1c80a63fa6817a9c7284c94e40cded14a9b0d0d3722356efa47bd 2.37kB / 2.37kB                                                                                                                                          0.0s
 => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB                                                                                                                                          0.7s
 => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546                                                                                                                                               0.2s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                       0.3s
 => => transferring context: 7.41MB                                                                                                                                                                                                     0.2s
 => [stage-2 1/5] FROM docker.io/library/python:3.9-slim                                                                                                                                                                                0.2s
 => [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     51.4s
 => [stage-2 2/5] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked   apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5      70.1s
 => [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get update && apt-get install -y git     && rm -rf /var/lib/apt/lists/*            60.6s
 => ERROR [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"                     13.0s
------
 > [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5":
#13 3.856 Collecting git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 3.858   Cloning https://github.com/python-poetry/poetry.git (to revision fb13b3a676f476177f7937ffa480ee5cff9a90a5) to /tmp/pip-req-build-tufn8q4p
#13 3.866   Running command git clone --filter=blob:none --quiet https://github.com/python-poetry/poetry.git /tmp/pip-req-build-tufn8q4p
#13 7.242   Running command git rev-parse -q --verify 'sha^fb13b3a676f476177f7937ffa480ee5cff9a90a5'
#13 7.250   Running command git fetch -q https://github.com/python-poetry/poetry.git fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 7.839   Running command git checkout -q fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.253   Resolved https://github.com/python-poetry/poetry.git to commit fb13b3a676f476177f7937ffa480ee5cff9a90a5
#13 9.720   Installing build dependencies: started
#13 12.36   Installing build dependencies: finished with status 'done'
#13 12.37   Getting requirements to build wheel: started
#13 12.46   Getting requirements to build wheel: finished with status 'done'
#13 12.46   Preparing metadata (pyproject.toml): started
#13 12.77   Preparing metadata (pyproject.toml): finished with status 'error'
#13 12.77   error: subprocess-exited-with-error
#13 12.77
#13 12.77   × Preparing metadata (pyproject.toml) did not run successfully.
#13 12.77   │ exit code: 1
#13 12.77   ╰─> [26 lines of output]
#13 12.77       Traceback (most recent call last):
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
#13 12.77           main()
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
#13 12.77           json_out['return_val'] = hook(**hook_input['kwargs'])
#13 12.77         File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
#13 12.77           return hook(metadata_directory, config_settings)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
#13 12.77           poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 62, in create_poetry
#13 12.77           package = self.configure_package(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 156, in configure_package
#13 12.77           cls._add_package_group_dependencies(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 102, in _add_package_group_dependencies
#13 12.77           cls.create_dependency(
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 366, in create_dependency
#13 12.77           dependency = Dependency(name, constraint, groups=groups)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 66, in __init__
#13 12.77           self.constraint = constraint  # type: ignore[assignment]
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/packages/dependency.py", line 110, in constraint
#13 12.77           self._constraint = parse_constraint(constraint)
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 31, in parse_constraint
#13 12.77           constraint_objects.append(parse_single_constraint(constraint))
#13 12.77         File "/tmp/pip-build-env-yp6fwb9j/overlay/lib/python3.9/site-packages/poetry/core/semver/helpers.py", line 147, in parse_single_constraint
#13 12.77           raise ParseConstraintError(f"Could not parse version constraint: {constraint}")
#13 12.77       poetry.core.semver.exceptions.ParseConstraintError: Could not parse version constraint: (>=20.4.3
#13 12.77       [end of output]
#13 12.77
#13 12.77   note: This error originates from a subprocess, and is likely not a problem with pip.
#13 12.78 error: metadata-generation-failed
#13 12.78
#13 12.78 × Encountered error while generating package metadata.
#13 12.78 ╰─> See above for output.
#13 12.78
#13 12.78 note: This is an issue with the package mentioned above, not pip.
#13 12.78 hint: See above for details.
#13 12.79 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#13 12.79 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"]: exit code: 1

Originally posted by @qknight in #11537 (comment)

Footnotes

  1. The steps which run apt-get update and run the abitrary sh.rustup.sh script aren't reproducible either, if anyone's counting. ↩

Activity

  1. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    I can reproduce this with DOCKER_BUILDKIT=1 docker build --no-cache -f docker/Dockerfile -t fred . on b8bf612

  2. clokep
    changed the title [-]Unable to build v.161.0 docker image with clean caches[/-] [+]Unable to build v1.61.0 docker image with clean caches[/+] on Sep 20, 2022

  3. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    Regressed in #12385 according to a bisection

  4. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    #13 12.77 poetry.core.semver.exceptions.ParseConstraintError: Could not parse version constraint: (>=20.4.3

    I think the version requirement comes from https://github.com/python-poetry/poetry/blob/fb13b3a676f476177f7937ffa480ee5cff9a90a5/pyproject.toml#L54 in the specific version of poetry we use in that version of the lockfile. But what I don't understand is why this is suddenly a problem now, given that we built the images just fine at the time.

  5. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    But what I don't understand is why this is suddenly a problem now,

    Seems to be python-poetry/poetry-core#461.

    Related:

    Frustrating. Goes to show that we should have properly somehow frozen the environment for poetry itself (C.f. #12853). I hate Python packaging.

    I suspect the easiest way to fix this will be to try to rebuild from v1.61 with a patched dockerfile. Let me get back to you on that.

  6. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    Running

    pip install "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5" -v

    in a clean virtualenv includes this line:

      Running command /home/dmr/workspace/dummy-python/.direnv/python-3.10.6/bin/python /tmp/pip-standalone-pip-ieecjtw7/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dxlrb3q_/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poetry-core>=1.0.0'

    which I think comes from here. This pulls in poetry-core 1.2.0 to try and build the pinned version of poetry itself... but this is now broken as python-poetry/poetry#6376 noted.

    We were using an unreleased poetry version so maybe this serves us right. But it's still incredibly frustrating.

  7. DMRobertson commented on Sep 20, 2022

    @DMRobertson
    DMRobertson
    on Sep 20, 2022
    ContributorAuthor

    Given that we have no control over what version pip chooses when it installs the build-backend, I think the only option is to build using poetry 1.2.

    diff --git a/docker/Dockerfile b/docker/Dockerfile
index 7af0e51f9..29010b436 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -55,7 +55,7 @@ RUN \
 # NB: In poetry 1.2 `poetry export` will be moved into a plugin; we'll need to also
 # pip install poetry-plugin-export (https://github.com/python-poetry/poetry-plugin-export).
 RUN --mount=type=cache,target=/root/.cache/pip \
-  pip install --user "poetry-core==1.1.0a7" "git+https://github.com/python-poetry/poetry.git@fb13b3a676f476177f7937ffa480ee5cff9a90a5"
+  pip install --user "poetry-core==1.2.0" "poetry==1.2.1"
 
 WORKDIR /synapse

    This doesn't create a bitwise identical container, but I think it's the best we can do.

    Details ``` $ container-diff diff -t apt -t pip -t sizelayer -t metadata -t history matrixdotorg/synapse:v1.61.0 daemon://fred:latest ERRO[0011] failed to locate python package for corresponding package metadata matrix-synapse ERRO[0011] failed to locate python package for corresponding package metadata matrix-synapse

    -----Apt-----

    Packages found only in matrixdotorg/synapse:v1.61.0: None

    Packages found only in fred:latest: None

    Version differences:
    PACKAGE IMAGE1 (matrixdotorg/synapse:v1.61.0) IMAGE2 (fred:latest)
    -base-files 11.1 deb11u3, 340K 11.1 deb11u2, 340K
    -curl 7.74.0-1.3 deb11u1, 426K 7.74.0-1.3 deb11u3, 429K
    -dpkg 1.20.10, 6.8M 1.20.9, 6.7M
    -gpgv 2.2.27-2 deb11u1, 882K 2.2.27-2, 882K
    -gzip 1.10-4 deb11u1, 242K 1.10-4, 242K
    -libc-bin 2.31-13 deb11u3, 3.6M 2.31-13 deb11u2, 3.6M
    -libc6 2.31-13 deb11u3, 12.5M 2.31-13 deb11u2, 12.5M
    -libcurl4 7.74.0-1.3 deb11u1, 732K 7.74.0-1.3 deb11u3, 739K
    -libexpat1 2.2.10-2 deb11u3, 411K 2.2.10-2 deb11u2, 411K
    -liblzma5 5.2.5-2.1~deb11u1, 277K 5.2.5-2, 277K
    -libpq5 13.7-0 deb11u1, 788K 13.8-0 deb11u1, 786K
    -libssl-dev 1.1.1n-0 deb11u2, 7.8M 1.1.1n-0 deb11u3, 7.8M
    -libssl1.1 1.1.1n-0 deb11u2, 4M 1.1.1n-0 deb11u3, 4M
    -libsystemd0 247.3-7, 865K 247.3-6, 864K
    -libudev1 247.3-7, 282K 247.3-6, 281K
    -libxslt1.1 1.1.34-4, 498K 1.1.34-4 deb11u1, 502K
    -openssl 1.1.1n-0 deb11u2, 1.4M 1.1.1n-0 deb11u3, 1.4M
    -publicsuffix 20211207.1025-0 deb11u1, 330K 20220811.1734-0 deb11u1, 336K
    -sysvinit-utils 2.96-7 deb11u1, 79K 2.96-7, 79K
    -tzdata 2021a-1 deb11u3, 3.3M 2021a-1 deb11u2, 3.3M
    -zlib1g 1:1.2.11.dfsg-2 deb11u1, 166K 1:1.2.11.dfsg-2, 166K

    -----History-----

    Docker history lines found only in matrixdotorg/synapse:v1.61.0:
    -/bin/sh -c #(nop) ADD file:134f25aec8adf83cb940ba073a3409ca85dbb5ae592b704f95193e7d2539a3bc in /
    -/bin/sh -c #(nop) ENV PYTHON_VERSION=3.9.13
    -/bin/sh -c set -eux; savedAptMark="$(apt-mark showmanual)"; apt-get update; apt-get install -y --no-install-recommends dpkg-dev gcc gnupg dirmngr libbluetooth-dev libbz2-dev libc6-dev libexpat1-dev libffi-dev libgdbm-dev liblzma-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev make tk-dev uuid-dev wget xz-utils zlib1g-dev ; wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; gpg --batch --verify python.tar.xz.asc python.tar.xz; command -v gpgconf > /dev/null && gpgconf --kill all || :; rm -rf "$GNUPGHOME" python.tar.xz.asc; mkdir -p /usr/src/python; tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; rm python.tar.xz; cd /usr/src/python; gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; ./configure --build="$gnuArch" --enable-loadable-sqlite-extensions --enable-optimizations --enable-option-checking=fatal --enable-shared --with-system-expat --without-ensurepip ; nproc="$(nproc)"; make -j "$nproc" LDFLAGS="-Wl,--strip-all" ; make install; cd /; rm -rf /usr/src/python; find /usr/local -depth ( ( -type d -a ( -name test -o -name tests -o -name idle_test ) ) -o ( -type f -a ( -name '.pyc' -o -name '.pyo' -o -name 'libpython*.a' ) ) ) -exec rm -rf '{}' + ; ldconfig; apt-mark auto '.' > /dev/null; apt-mark manual $savedAptMark; find /usr/local -type f -executable -not ( -name 'tkinter' ) -exec ldd '{}' ';' | awk '/=>/ { print $(NF-1) }' | sort -u | xargs -r dpkg-query --search | cut -d: -f1 | sort -u | xargs -r apt-mark manual ; apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; rm -rf /var/lib/apt/lists/; python3 --version
    -/bin/sh -c set -eux; for src in idle3 pydoc3 python3 python3-config; do dst="$(echo "$src" | tr -d 3)"; [ -s "/usr/local/bin/$src" ]; [ ! -e "/usr/local/bin/$dst" ]; ln -svT "$src" "/usr/local/bin/$dst"; done
    -/bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=22.0.4
    -/bin/sh -c #(nop) ENV PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py
    -/bin/sh -c #(nop) ENV PYTHON_GET_PIP_SHA256=ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d

    Docker history lines found only in fred:latest:
    -/bin/sh -c #(nop) ADD file:d48a85028743f16ed927507322e3e3beee187fbfadd0b781d4b89de197c64b5b in /
    -/bin/sh -c #(nop) ENV PYTHON_VERSION=3.9.10
    -/bin/sh -c set -eux; savedAptMark="$(apt-mark showmanual)"; apt-get update; apt-get install -y --no-install-recommends dpkg-dev gcc gnupg dirmngr libbluetooth-dev libbz2-dev libc6-dev libexpat1-dev libffi-dev libgdbm-dev liblzma-dev libncursesw5-dev libreadline-dev libsqlite3-dev libssl-dev make tk-dev uuid-dev wget xz-utils zlib1g-dev ; wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; gpg --batch --verify python.tar.xz.asc python.tar.xz; command -v gpgconf > /dev/null && gpgconf --kill all || :; rm -rf "$GNUPGHOME" python.tar.xz.asc; mkdir -p /usr/src/python; tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; rm python.tar.xz; cd /usr/src/python; gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; ./configure --build="$gnuArch" --enable-loadable-sqlite-extensions --enable-optimizations --enable-option-checking=fatal --enable-shared --with-system-expat --with-system-ffi --without-ensurepip ; nproc="$(nproc)"; make -j "$nproc" LDFLAGS="-Wl,--strip-all" ; make install; cd /; rm -rf /usr/src/python; find /usr/local -depth ( ( -type d -a ( -name test -o -name tests -o -name idle_test ) ) -o ( -type f -a ( -name '.pyc' -o -name '.pyo' -o -name '.a' ) ) ) -exec rm -rf '{}' + ; ldconfig; apt-mark auto '.' > /dev/null; apt-mark manual $savedAptMark; find /usr/local -type f -executable -not ( -name 'tkinter' ) -exec ldd '{}' ';' | awk '/=>/ { print $(NF-1) }' | sort -u | xargs -r dpkg-query --search | cut -d: -f1 | sort -u | xargs -r apt-mark manual ; apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; rm -rf /var/lib/apt/lists/*; python3 --version
    -/bin/sh -c set -eux; for src in idle3 pydoc3 python3 python3-config; do dst="$(echo "$src" | tr -d 3)"; [ -s "/usr/local/bin/$src" ]; [ ! -e "/usr/local/bin/$dst" ]; ln -svT "/usr/local/bin/$src" "/usr/local/bin/$dst"; done
    -/bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=21.2.4
    -/bin/sh -c #(nop) ENV PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py
    -/bin/sh -c #(nop) ENV PYTHON_GET_PIP_SHA256=e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a

    -----Metadata-----

    Image metadata differences between matrixdotorg/synapse:v1.61.0 and fred:latest:

    matrixdotorg/synapse:v1.61.0
    -Env: PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,LANG=C.UTF-8,GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568,PYTHON_VERSION=3.9.10,PYTHON_PIP_VERSION=21.2.4,PYTHON_SETUPTOOLS_VERSION=58.1.0,PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py,PYTHON_GET_PIP_SHA256=e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a
    -Labels: org.opencontainers.image.documentation:https://github.com/matrix-org/synapse/blob/master/docker/README.md org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/matrix-org/synapse.git org.opencontainers.image.url:https://matrix.org/docs/projects/server/synapse

    fred:latest
    -Env: PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,LANG=C.UTF-8,GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568,PYTHON_VERSION=3.9.13,PYTHON_PIP_VERSION=22.0.4,PYTHON_SETUPTOOLS_VERSION=58.1.0,PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py,PYTHON_GET_PIP_SHA256=ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d
    -Labels: gitsha1:b8bf61230c0d51231429b2d15973a8fd1cd76906 org.opencontainers.image.documentation:https://github.com/matrix-org/synapse/blob/master/docker/README.md org.opencontainers.image.licenses:Apache-2.0 org.opencontainers.image.source:https://github.com/matrix-org/synapse.git org.opencontainers.image.url:https://matrix.org/docs/projects/server/synapse

    -----Pip-----

    Packages found only in matrixdotorg/synapse:v1.61.0: None

    Packages found only in fred:latest: None

    Version differences:
    PACKAGE IMAGE1 (matrixdotorg/synapse:v1.61.0) IMAGE2 (fred:latest)
    -pip 22.0.4, 6.9M 21.2.4, 5.1M

    -----SizeLayer-----

    Layer size differences between matrixdotorg/synapse:v1.61.0 and fred:latest:
    LAYER SIZE1 SIZE2
    0 326M 323.7M
    1 326M 323.7M
    2 326M 323.7M
    3 326M 323.7M
    4 326M 323.7M
    5 326M 323.7M
    6 326M 323.7M
    7 326M 323.7M
    8 326M 323.7M

    </details>
  8. H-Shay
    added
    A-DockerDocker images, or making it easier to run Synapse in a container.
    S-MajorMajor functionality / product severely impaired, no satisfactory workaround.
    T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.
    X-RegressionSomething broke which worked on a previous release
    O-OccasionalAffects or can be seen by some users regularly or most users rarely
    on Sep 20, 2022

  9. reivilibre commented on Sep 29, 2022

    @reivilibre
    reivilibre
    on Sep 29, 2022
    Contributor

    If we wanted to pin this for the future, we could manually do e.g. pip install poetry==1.2.1 and then pip freeze > poetry_requirements.txt.
    In the Dockerfile, we could pip install -r poetry_requirements.txt instead of installing a poetry version with floating versions of (transitive) dependencies.
    That would give us some pinned deps for Poetry to use in our Dockerfile; sort of like bootstrapping. That won't help with existing Dockerfiles but it would solve this going forward.

  10. DMRobertson commented on Sep 29, 2022

    @DMRobertson
    DMRobertson
    on Sep 29, 2022
    ContributorAuthor

    If we wanted to pin this for the future, we could manually do e.g. pip install poetry==1.2.1 and then pip freeze > poetry_requirements.txt. In the Dockerfile, we could pip install -r poetry_requirements.txt

    Agreed. I was worried about managing poetry's dependencies itself during #11537 but clearly dropped the ball on this one. :(

    However: I'm not sure if pip will use the current virtualenv's contents when fetching a build-backend to install poetry. (That is why we pull in poetry-core 1.2.x, and then we hit python-poetry/poetry#6376, which "made poetry uninstallable via pip" according to python-poetry/poetry#6402).

  11. dklimpel commented on May 6, 2023

    @dklimpel
    dklimpel
    on May 6, 2023
    Contributor

    Is this issue still up to date?

  12. DMRobertson commented on May 9, 2023

    @DMRobertson
    DMRobertson
    on May 9, 2023
    ContributorAuthor

    It is still the case that the dockerfile does not pin poetry's dependencies, yes:

    synapse/docker/Dockerfile

    Lines 60 to 61 in d4eba44

    RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --user "poetry==1.3.2"

  13. richvdh commented on May 16, 2023

    @richvdh
    richvdh
    on May 16, 2023
    Member

    It is still the case that the dockerfile does not pin poetry's dependencies, yes:

    @DMRobertson could you update the issue summary, then? Currently it reads as if there is a specific problem with v1.61, which seems like a difficult thing to care about.

  14. DMRobertson
    changed the title [-]Unable to build v1.61.0 docker image with clean caches[/-] [+]Synapse's docker images (and CI?) do not pin the specific build requirements of poetry[/+] on May 16, 2023
  15. DMRobertson
    added
    S-MinorBlocks non-critical functionality, workarounds exist.
    O-UncommonMost users are unlikely to come across this or unexpected workflow
    and removed
    S-MajorMajor functionality / product severely impaired, no satisfactory workaround.
    O-OccasionalAffects or can be seen by some users regularly or most users rarely
    on May 16, 2023

  16. DMRobertson commented on May 16, 2023

    @DMRobertson
    DMRobertson
    on May 16, 2023
    ContributorAuthor

    I have updated the description; I hope it is comprehensible.

    However on reflection I think we should probably ignore (close?) this:

    • I proposed a patch to the dockerfile as a workaround for the reporter
    • We don't have the time or resources to make Python's packaging ecosystem bulletproof
    • We ourselves are still going to be running pipx install poetry==a.b.c without using any kind of build-system pinning.

    Let's discuss among the team.

  17. DMRobertson commented on May 18, 2023

    @DMRobertson
    DMRobertson
    on May 18, 2023
    ContributorAuthor

    Closing as WONTFIX.

  18. DMRobertson
    closed this as not plannedon May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-DockerDocker images, or making it easier to run Synapse in a container.O-UncommonMost users are unlikely to come across this or unexpected workflowS-MinorBlocks non-critical functionality, workarounds exist.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.X-RegressionSomething broke which worked on a previous release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions