Use airflow user for build segment of docker image - #20744
Merged
Conversation
boring-cyborg
Bot
added
the
area:production-image
Member
Author
|
This is another smaller change separated out from the #20238 |
potiuk
force-pushed
the
build-image-uses-airflow-useer
branch
from
January 7, 2022 11:41
6558bd8 to
d999df3
Compare
1 task
Member
Author
potiuk
force-pushed
the
build-image-uses-airflow-useer
branch
from
January 7, 2022 11:52
d999df3 to
5a3a105
Compare
Member
Author
|
Looks green :) |
potiuk
changed the title
PIP produces a warning when root user is used to run pip install. This is done for a good reason - because installing PIP this way clashes with a number of distro-managed python packages. The warning cannot be disabled even if our use case is legitimate as has been extensively discussed in pypa/pip#10556. However, the advice given by the warning is a bit misleading - it suggests to use virtualenv, but since this is considered a bad practice for container building and because we need to create virtualenvs dynamically inside the image, using virtualenv is a bad solution for us. It's been attempted in apache#19189 and failed. Instead we create an airflow user and use PIP_USER="true" which installs all dependencies in build segment to ~/.local folder from where we can copy it to the main image. That get rids of the warning and at the same time allows us to keep the best practices of building the images.
potiuk
force-pushed
the
build-image-uses-airflow-useer
branch
from
January 8, 2022 16:34
5a3a105 to
67279c6
Compare
Member
Author
|
REbased to latest main. Would love re-review. |
mik-laj
approved these changes
Jan 8, 2022
Contributor
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
Bot
added
the
full tests needed
potiuk
added
the
changelog:skip
potiuk
added a commit
that referenced
this pull request
Jan 22, 2022
PIP produces a warning when root user is used to run pip install. This is done for a good reason - because installing PIP this way clashes with a number of distro-managed python packages. The warning cannot be disabled even if our use case is legitimate as has been extensively discussed in pypa/pip#10556. However, the advice given by the warning is a bit misleading - it suggests to use virtualenv, but since this is considered a bad practice for container building and because we need to create virtualenvs dynamically inside the image, using virtualenv is a bad solution for us. It's been attempted in #19189 and failed. Instead we create an airflow user and use PIP_USER="true" which installs all dependencies in build segment to ~/.local folder from where we can copy it to the main image. That get rids of the warning and at the same time allows us to keep the best practices of building the images. (cherry picked from commit 3feb057)
jedcunningham
pushed a commit
that referenced
this pull request
Jan 27, 2022
PIP produces a warning when root user is used to run pip install. This is done for a good reason - because installing PIP this way clashes with a number of distro-managed python packages. The warning cannot be disabled even if our use case is legitimate as has been extensively discussed in pypa/pip#10556. However, the advice given by the warning is a bit misleading - it suggests to use virtualenv, but since this is considered a bad practice for container building and because we need to create virtualenvs dynamically inside the image, using virtualenv is a bad solution for us. It's been attempted in #19189 and failed. Instead we create an airflow user and use PIP_USER="true" which installs all dependencies in build segment to ~/.local folder from where we can copy it to the main image. That get rids of the warning and at the same time allows us to keep the best practices of building the images. (cherry picked from commit 3feb057)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PIP produces a warning when root user is used to run pip install.
This is done for a good reason - because installing PIP this way
clashes with a number of distro-managed python packages.
The warning cannot be disabled even if our use case is legitimate
as has been extensively discussed in
pypa/pip#10556.
The advice given by the warning is a bit misleading - it suggests
to use virtualenv, but since this is considered a bad practice
for container building and because we need to create virtualenvs
dynamically inside the image, using virtualenv is a bad solution
for us. It's been attempted in #19189 and failed.
Instead we create an airflow user and use PIP_USER="true" which
installs all dependencies in build segment to ~/.local folder
from where we can copy it to the main image.
That get rids of the warning and at the same time allows us to
keep the best practices of building the images.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.