Salt Virtualenv is broken when requirements file contains comments

[mced]

Description of Issue

Creation of virtualenv with requirements file doesn't work if requirements file contains depandabot comments.

Setup

salt/app.sls

/home/user/app/venv:
  virtualenv.managed:
    - requirements: /home/user/app/requirements/base.txt
    - user: user
    - python: /usr/bin/python3

requirements/base.txt

...
 psycopg2-binary==2.8.4    # via -r requirements/base.in
...

The comment has been added by dependabot, and seems to be parsed somehow by salt.

requirements/base.in is used by pip-compile to compile a requirements/base.txt

The error resulting:

Comment: An exception occurred in this state: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1889, in call
**cdata['kwargs'])
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1839, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/salt/states/virtualenv_mod.py", line 321, in managed
**kwargs
File "/usr/lib/python2.7/dist-packages/salt/modules/pip.py", line 628, in install
user=user
File "/usr/lib/python2.7/dist-packages/salt/modules/pip.py", line 327, in _process_requirements
reqs = _resolve_requirements_chain(requirement)
File "/usr/lib/python2.7/dist-packages/salt/modules/pip.py", line 257, in _resolve_requirements_chain
chain.extend(_resolve_requirements_chain(_find_req(req_file)))
File "/usr/lib/python2.7/dist-packages/salt/modules/pip.py", line 257, in _resolve_requirements_chain
chain.extend(_resolve_requirements_chain(_find_req(req_file)))
File "/usr/lib/python2.7/dist-packages/salt/modules/pip.py", line 235, in _find_req
with salt.utils.fopen(link) as fh_link:
File "/usr/lib/python2.7/dist-packages/salt/utils/__init__.py", line 1361, in fopen
fhandle = open(*args, **kwargs) # pylint: disable=resource-leakage
IOError: [Errno 2] No such file or directory: 'requirements/base.in'

Versions Report

Salt Version:
           Salt: 2017.7.8
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Oct  8 2019, 14:14:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-173-generic
         system: Linux
        version: Ubuntu 16.04 xenial

I double checked if the parser in https://github.com/saltstack/salt/commits/master/salt/modules/pip.py and https://github.com/saltstack/salt/commits/master/salt/modules/pip.py was fixed in v2018, v2019, v3000, and that's not the case.

[OrangeDog]

Perhaps salt should use the pypa requirements-parsing code?
https://packaging.pypa.io/en/latest/requirements/

[xeacott]

So although i was able to reproduce the exact same error as you on Ubuntu with python2.7, i was able to run this successfully on windows 10 python 3.6 and salt 2019.2.1..

local:
----------
          ID: C://dev//salt//venv
    Function: virtualenv.managed
      Result: True
     Comment: Created new virtualenv
     Started: 13:44:58.792214
    Duration: 56652.166 ms
     Changes:
              ----------
              new:
              packages:
                  ----------
                  new:
                      - urllib3==1.25.8
                      - idna==2.9
                      - MarkupSafe==1.1.1
                      - PyYAML==5.3.1
                      - msgpack==0.6.2
                      - Jinja2==2.11.1
                      - chardet==3.0.4
                      - requests==2.23.0
                      - certifi==2019.11.28
                  old:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  56.652 s

[xeacott]

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: 1.12.2
       cherrypy: 17.4.1
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: 0.28.4
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: 3.8.1
         pygit2: 1.0.1
         Python: 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)]
   python-gnupg: 0.4.4
         PyYAML: 5.3.1
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: 2.0.5
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: cp1252
        machine: AMD64
        release: 10
         system: Windows
        version: 10 10.0.18362 SP0 Multiprocessor Free

[mced]

@xeacott can you tell which information you need ?

[Bjwebb]

I run into the same issue with Python 2.7 and Python 3.6 on Linux. Happy to provide extra info if its useful.