Refactor Docker image and package builds

[ethho]

Dependency on datajoint/djbase as a base image makes it difficult to maintain the datajoint/datajoint image. For example, adding support for Python 3.12 #1165 requires rebuild of the entire "image hierarchy", for which many of the builds are not reproducible. The task for this PR is to refactor the datajoint/datajoint Docker image so that it is lightweight, easy to maintain, and does not depend on the image hierarchy. Includes changes from #1187

[ethho]

Build fails with

Details

❯ PY_VER=3.8 MYSQL_VER=5.7 DISTRO=alpine MINIO_VER=RELEASE.2022-08-11T04-37-28Z HOST_UID=$(id -u) docker compose -f LNX-docker-compose.yml build
WARN[0000] /home/eho/repos/dj/datajoint-python/LNX-docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Building 24.7s (10/10) FINISHED                                                                                      docker:default
 => [app internal] load build definition from Dockerfile                                                                           0.0s
 => => transferring dockerfile: 425B                                                                                               0.0s
 => [app internal] load metadata for docker.io/jupyter/docker-stacks-foundation:latest                                             0.9s
 => [app auth] jupyter/docker-stacks-foundation:pull token for registry-1.docker.io                                                0.0s
 => [app internal] load .dockerignore                                                                                              0.0s
 => => transferring context: 90B                                                                                                   0.0s
 => [app 1/5] FROM docker.io/jupyter/docker-stacks-foundation:latest@sha256:5c5b20673f6ac7649de7cae5df18303936e9a986d0c46354d21a  11.0s
 => => resolve docker.io/jupyter/docker-stacks-foundation:latest@sha256:5c5b20673f6ac7649de7cae5df18303936e9a986d0c46354d21ab180f  0.0s
 => => sha256:5c5b20673f6ac7649de7cae5df18303936e9a986d0c46354d21ab180f66c4e13 772B / 772B                                         0.0s
 => => sha256:088f11eb1e74ae07b0c0cfefb7906e1821c0b84f00524cc8e4011c276fe7d2d7 679B / 679B                                         0.2s
 => => sha256:c2565182fd8265060e6c61839a66692c10a3997f7760d5ac2de101440c1e17db 9.16kB / 9.16kB                                     0.0s
 => => sha256:fd92c719666cac5862b8ea1e034eeb038c7191d52d0ff468c0a90ed68c0420a4 8.82MB / 8.82MB                                     1.0s
 => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B                                           0.1s
 => => sha256:41606cd2ba0f4e4957c031820d7d5a564a24413f0e409aa293dccbd43cfd348b 3.02kB / 3.02kB                                     0.0s
 => => sha256:ef8373d600b0ac4a916da50b72cb116efc56800836a264d1dd8ea8e07e76256d 1.92kB / 1.92kB                                     0.3s
 => => sha256:77e45ee945dc8a1ce57be013482750833e7721903814e42510a3db2a05940aee 4.92kB / 4.92kB                                     0.4s
 => => sha256:a30f89a0af6c7f64872ff93f4a9445b9d25f1206982670a911b2c83b672f443a 151B / 151B                                         0.8s
 => => sha256:dc42adc7eb7398a8589bad907643f9b08467de588b3c2719c78302d5ec480446 276B / 276B                                         0.6s
 => => sha256:abaa8376a6502612b92e1750cee7aa450ef045e23a418077439a392b0c224f2e 104.82MB / 104.82MB                                 9.1s
 => => sha256:aa099bb9e49a60e4a86d1015fa7be3ea1dc1dc87153a0c0767dac5c3df94babb 4.40kB / 4.40kB                                     1.0s
 => => extracting sha256:fd92c719666cac5862b8ea1e034eeb038c7191d52d0ff468c0a90ed68c0420a4                                          0.2s
 => => sha256:822c4cbcf6a6b7660f90516cd166125d55de917a2f18c086eee2abe52c1b7e9a 181B / 181B                                         1.2s
 => => extracting sha256:088f11eb1e74ae07b0c0cfefb7906e1821c0b84f00524cc8e4011c276fe7d2d7                                          0.0s
 => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1                                          0.0s
 => => extracting sha256:ef8373d600b0ac4a916da50b72cb116efc56800836a264d1dd8ea8e07e76256d                                          0.0s
 => => extracting sha256:77e45ee945dc8a1ce57be013482750833e7721903814e42510a3db2a05940aee                                          0.0s
 => => extracting sha256:a30f89a0af6c7f64872ff93f4a9445b9d25f1206982670a911b2c83b672f443a                                          0.0s
 => => extracting sha256:dc42adc7eb7398a8589bad907643f9b08467de588b3c2719c78302d5ec480446                                          0.0s
 => => extracting sha256:abaa8376a6502612b92e1750cee7aa450ef045e23a418077439a392b0c224f2e                                          1.5s
 => => extracting sha256:aa099bb9e49a60e4a86d1015fa7be3ea1dc1dc87153a0c0767dac5c3df94babb                                          0.0s
 => => extracting sha256:822c4cbcf6a6b7660f90516cd166125d55de917a2f18c086eee2abe52c1b7e9a                                          0.0s
 => [app internal] load build context                                                                                              0.1s
 => => transferring context: 1.07MB                                                                                                0.0s
 => [app 2/5] RUN conda install -y -n base -c conda-forge python=${PY_VER} &&     conda clean -afy                                12.0s
 => [app 3/5] COPY --chown=anaconda:anaconda ./setup.py ./datajoint.pub ./requirements.txt /main/                                  0.1s
 => [app 4/5] COPY --chown=anaconda:anaconda ./datajoint /main/datajoint                                                           0.0s
 => ERROR [app 5/5] RUN     pip install --no-cache-dir /main &&     rm -r /main/*                                                  0.6s
------
 > [app 5/5] RUN     pip install --no-cache-dir /main &&     rm -r /main/*:
0.392 Processing /main
0.393   Preparing metadata (setup.py): started
0.466   Preparing metadata (setup.py): finished with status 'error'
0.469   error: subprocess-exited-with-error
0.469   
0.469   × python setup.py egg_info did not run successfully.
0.469   │ exit code: 1
0.469   ╰─> [45 lines of output]
0.469       /opt/conda/lib/python3.11/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
0.469       !!
0.469       
0.469               ********************************************************************************
0.469               Requirements should be satisfied by a PEP 517 installer.
0.469               If you are using pip, you can try `pip install --use-pep517`.
0.469               ********************************************************************************
0.469       
0.469       !!
0.469         dist.fetch_build_eggs(dist.setup_requires)
0.469       Traceback (most recent call last):
0.469         File "<string>", line 2, in <module>
0.469         File "<pip-setuptools-caller>", line 34, in <module>
0.469         File "/main/setup.py", line 26, in <module>
0.469           setup(
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 102, in setup
0.469           _install_setup_requires(attrs)
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 75, in _install_setup_requires
0.469           _fetch_build_eggs(dist)
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/__init__.py", line 80, in _fetch_build_eggs
0.469           dist.fetch_build_eggs(dist.setup_requires)
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 662, in fetch_build_eggs
0.469           return _fetch_build_eggs(self, requires)
0.469                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/installer.py", line 38, in _fetch_build_eggs
0.469           resolved_dists = pkg_resources.working_set.resolve(
0.469                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/pkg_resources/__init__.py", line 829, in resolve
0.469           dist = self._resolve_dist(
0.469                  ^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/pkg_resources/__init__.py", line 865, in _resolve_dist
0.469           dist = best[req.key] = env.best_match(
0.469                                  ^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1135, in best_match
0.469           return self.obtain(req, installer)
0.469                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/pkg_resources/__init__.py", line 1147, in obtain
0.469           return installer(requirement)
0.469                  ^^^^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/installer.py", line 74, in _fetch_build_egg_no_warn
0.469           eggs_dir = os.path.realpath(dist.get_egg_cache_dir())
0.469                                       ^^^^^^^^^^^^^^^^^^^^^^^^
0.469         File "/opt/conda/lib/python3.11/site-packages/setuptools/dist.py", line 707, in get_egg_cache_dir
0.469           os.mkdir(egg_cache_dir)
0.469       PermissionError: [Errno 13] Permission denied: './.eggs'
0.469       [end of output]
0.469   
0.469   note: This error originates from a subprocess, and is likely not a problem with pip.
0.469 error: metadata-generation-failed
0.469 
0.469 × Encountered error while generating package metadata.
0.469 ╰─> See above for output.
0.469 
0.469 note: This is an issue with the package mentioned above, not pip.
0.469 hint: See above for details.
------
failed to solve: process "/bin/bash -o pipefail -c pip install --no-cache-dir /main &&     rm -r /main/*" did not complete successfully: exit code: 1

[ethho]

@tabedzki Added Python 3.12 testing to CI, which should resolve #1165.

[tabedzki]

Great! Thanks for tackling the refactoring issue and officially including Python 3.12 support!

[dimitri-yatsenko]

Before making the release, let's disable the default addition of timestamps to every table as implemented in #1091---it's to disruptive of a change. We can make this configurable through dj.config, or add a specific parameter to the @schema decorator, for example.

[ethho]

Before making the release, let's disable the default addition of timestamps to every table as implemented in #1091---it's to disruptive of a change. We can make this configurable through dj.config, or add a specific parameter to the @schema decorator, for example.

@dimitri-yatsenko I agree. We should implement that patch to #1091 in a separate PR (WIP PR #1188) , which we will make sure to merge before creating a release. Can we test and merge this PR since its changes are unrelated to #1091? Merging this PR will not create a release automatically.