Python pip builder resolve dependency error : numpy==1.20.3

[vynu]

Description:
Python pip builder resolve dependency error : numpy==1.20.3 while running sam build with pandas in lambda app requirements.txt

deep debugging :

ran pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch numpy==1.20.3 by logging into actively running amazon/aws-sam-cli-build-image-python3.8

its downloading with wheel file name : Saved /tmp/samcli/scratch/numpy-1.20.3-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

I guess its failing at : this-line

and I've tried with pip download --only-binary=:all: --no-deps --platform manylinux2014_x86_64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch numpy==1.20.1

this downloaded wheel file : Saved /tmp/samcli/scratch/numpy-1.20.1-cp38-cp38-manylinux2010_x86_64.whl

clearly numpy==1.20.1 does not have any issue in file name

Steps to reproduce the issue:

1. Run sam build --use-container --build-image amazon/aws-sam-cli-build-image-python3.8 --debug with pandas lib dependency in lambda app requirements.txt

Observed result:
PythonPipBuilder:ResolveDependencies - {numpy==1.20.3(wheel)}

Expected result:
build successful

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
linux container env : amazon/aws-sam-cli-build-image-python3.8

[vynu]

FYI:

adding numpy==1.20.1 to your requirements.txt before pandas will not raise any problem

[charris]

See numpy/numpy#18988, the problem is that the naming scheme changed in the manylinux docker images. The immediate AWS problem is:

    _MANYLINUX_COMPATIBLE_PLATFORM = {
        "any",
        "linux_x86_64",
        "manylinux1_x86_64",
        "manylinux2010_x86_64",
        "manylinux2014_x86_64",
    }

The set needs to include the new aliased names.

[mayeut]

The set does not need to include the new names, however, per PEP427, the platform tag is in fact a set off platform tags joined with . so:

--- a/aws_lambda_builders/workflows/python_pip/packager.py
+++ b/aws_lambda_builders/workflows/python_pip/packager.py
@@ -343,7 +343,7 @@ class DependencyBuilder(object):
         wheel = filename[:-4]
         implementation, abi, platform = wheel.split("-")[-3:]
         # Verify platform is compatible
-        if platform not in self._MANYLINUX_COMPATIBLE_PLATFORM:
+        if self._MANYLINUX_COMPATIBLE_PLATFORM.isdisjoint(platform.split(".")):
             return False
 
         lambda_runtime_abi = get_lambda_abi(self.runtime)

should work and it does (following tests are passing with previous diff applied, failing without the diff applied):

--- a/tests/integration/workflows/python_pip/test_python_pip.py
+++ b/tests/integration/workflows/python_pip/test_python_pip.py
@@ -47,8 +47,10 @@ class TestPythonPipWorkflow(TestCase):
 
         if self.runtime == "python2.7":
             expected_files = self.test_data_files.union({"numpy", "numpy-1.15.4.data", "numpy-1.15.4.dist-info"})
-        else:
+        elif self.runtime == "python3.6":
             expected_files = self.test_data_files.union({"numpy", "numpy-1.17.4.dist-info"})
+        else:
+            expected_files = self.test_data_files.union({"numpy", "numpy.libs", "numpy-1.20.3.dist-info"})
         output_files = set(os.listdir(self.artifacts_dir))
         self.assertEqual(expected_files, output_files)
 
--- a/tests/integration/workflows/python_pip/testdata/requirements-numpy.txt
+++ b/tests/integration/workflows/python_pip/testdata/requirements-numpy.txt
@@ -1,2 +1,3 @@
 numpy==1.15.4; python_version == '2.7'
-numpy==1.17.4; python_version >= '3.6'
+numpy==1.17.4; python_version == '3.6'
+numpy==1.20.3; python_version >= '3.7'

[aahung]

pinning the issue to prevent duplicates.

[aahung]

1.4.0 supports PEP 600 tags.

[kannanprasanna]

@aahung , any update on when next aws-sam-cli version will be released with aws-lambda-builders pointing to 1.4.0 (currently seems to be hardcoded to version 1.3.0) or do you recommend pinning aws-lambda-builders 1.4.0 in our requirements file?

[aahung]

AWS SAM CLI 1.24.0 has been released now