Dependency rooted in a subfolder of a GIT repository

I have searched the issues of this repo and believe that this is not a duplicate.

Feature request

Currently, as far as I can figure, it is impossible to use a GIT repository as a dependency source where the setup.py or pyproject.toml file is not in the root directory but in some sub-folder of the repository.

E.g. GIT repo structure:

GIT root
- pythonmodule
  - setup.py / pyproject.toml
  - module1.py
  - module2.py
- javamodule
  - pom.xml
- cmodule
  - Makefile

In this example I would like to be able to point poetry to the pythonmodule directory inside this repo, for example using poetry add mydep --git https://github.com/foo/bar.git --path path/to/mydep. I don't think this can be accomplished via a separate repository definition, however, the documentation is a bit scarce on that topic.

Same here, i'm able to install a git dependency using pip with its VCS support

pip install "git+https://github.com/nmslib/hnswlib.git@bb2a1d1#egg=hnswlib&subdirectory=python_bindings"

It would be nice if we can do it using poetry add.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

I would love to switch to Poetry and it looks like this is the only thing preventing me. Very excited to see this feature is in progress!

Ready for testing :)

@finswimmer

@finswimmer Thank you for your work on this! Is there any way I can help with getting your PRs ready for review?

Was this functionality implemented? My team relies on poetry and we just started breaking some of our code up into a separate git repo - we would need this to continue to use poetry

Was this functionality implemented? My team relies on poetry and we just started breaking some of our code up into a separate git repo - we would need this to continue to use poetry

As far as I can tell, it has been implemented, but not merged nor released. This still-in-progress implementation is here: #1822.

It isn't fully implemented until now. A lot has changed since I wrote #1822. This PR is now splitted in python-poetry/poetry-core#9 and #2242 But there is still a lot to do. I cannot tell when this will be ready.

@ashnair1

Thank you @ashnair1 and @remram44! It does seem like this is a bit difficult right now. It may be too much of a curve ball for my team so I will probably have to wait for better support for subdirectories.

Thanks again!

I've opened #5172 and python-poetry/poetry-core#288 to address this.

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }

Hi folks! Have anybody already tested it? It should be extremely useful, but it looks i've installed this fork in wrong way and it doesn't work correctly on my machine = (

Suggest edit of title to use subdirectory which is the term used in PIP to help it be found by search.

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }
Hi folks! Have anybody already tested it? It should be extremely useful, but it looks i've installed this fork in wrong way and it doesn't work correctly on my machine = (

This doesn't seem to work even with poetry and poetry-core pre (as of today).

Our "problem dependency" is this:

unicoirn = { git = "https://github.com/unicorn-engine/unicorn.git", branch = "dev", subdirectory="bindings/python"}

It seems it does not enter the subdirectory when building the VCS package. With an error output of:

The following error occurred when trying to handle this error:


  PackageInfoError

  Unable to determine package info for path: /var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd
  
  Command ['/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/bin/python', '-'] errored with the following return code 1, and output: 
  Traceback (most recent call last):
    File "<stdin>", line 9, in <module>
    File "/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/lib/python3.10/site-packages/build/__init__.py", line 208, in __init__
      _validate_source_directory(srcdir)
    File "/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/.venv/lib/python3.10/site-packages/build/__init__.py", line 109, in _validate_source_directory
      raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
  build.BuildException: Source /var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd does not appear to be a Python project: no pyproject.toml or setup.py
  input was : import build
  import build.env
  import pep517
  
  source = '/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/pypoetry-git-unicorntfo2cpdd'
  dest = '/var/folders/0j/v1pqc5ks1v90nj3xf_h715sc0000gn/T/tmp1kapedvm/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pep517.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  
  No fallback setup.py file was found to generate egg_info.

  at ~/.pyenv/versions/3.10.2/lib/python3.10/site-packages/poetry/inspection/info.py:497 in _pep517_metadata
      493│                 # fallback to egg_info if setup.py available
      494│                 cls._log(f"PEP517 build failed: {e}", level="debug")
      495│                 setup_py = path / "setup.py"
      496│                 if not setup_py.exists():
    → 497│                     raise PackageInfoError(
      498│                         path,
      499│                         e,
      500│                         "No fallback setup.py file was found to generate egg_info.",
      501│                     )```

I've tried this but can't even get the subdirectory parameter to work.

I've got:

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

in my pyproject.toml. I'm using a ssh URL for a private GitHub repo. But I get "[repo definition] is not valid under any of the given schemas".

Does this need the pre-release Poetry as well as the poetry-core settings?

Update scikit-decide dependencies

@ashnair1

So I think I've got poetry VCS installing to work with subdirectory kwarg - but I needed to update both poetry and poetry-core ~~to bleeding edge versions~~ first;

$ pip install git+https://github.com/python-poetry/poetry.git@1.2.0b2
$ pip install git+https://github.com/python-poetry/poetry-core.git@1.1.0b2

~~which is the commit where @ashnair1 s PR #5172 merged (using poetry-core HEAD isn't currently working for me)~~

After that, adding

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

and the subdirectory kwarg

my_module = { git = "https://github.com/my_org/my_repo.git", subdirectory = "src/my_module_folder" }

seems to be working for me 🚀

~~NB This would be much nicer if there were some recent releases that contained these changes.~~

Edit. Sorry RE recent release comments maintainers - they do exist, and poetry is a fine tool 👍

seems to be working for me rocket

I think something is missing here since it does not work for me. Here is my environment:

$ pip freeze

poetry @ git+https://github.com/python-poetry/poetry.git@ccbeffc1f1a5809e999cc18c584ed2cee2a8f580
poetry-core==1.1.0b2

$ cat pyproject.toml

...
[tool.poetry.dependencies]
openapi-client = { git = "https://github.com/deribit/deribit-api-clients.git", subdirectory = "python" }

[build-system]
requires = ["poetry-core @ git+https://github.com/python-poetry/poetry-core.git"]
build-backend = "poetry.core.masonry.api"

Trying to install project with: poetry install ends up with:

  • Installing openapi-client (1.0.0 7ae49f6): Failed

  CalledProcessError

  Command '['ROOT/machine/.venv/bin/python', '/home/lukasz/.pyenv/versions/3.10.2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', 'ROOT/machine/.venv', '--upgrade', '--no-deps', 'ROOT/machine/.venv/src/deribit-api-clients']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.10.2/lib/python3.10/subprocess.py:524 in run
       520│             # We don't call process.wait() as .__exit__ does that for us.
       521│             raise
       522│         retcode = process.poll()
       523│         if check and retcode:
    →  524│             raise CalledProcessError(retcode, process.args,
       525│                                      output=stdout, stderr=stderr)
       526│     return CompletedProcess(process.args, retcode, stdout, stderr)
       527│ 
       528│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['ROOT/machine/.venv/bin/python', '/home/lukasz/.pyenv/versions/3.10.2/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', 'ROOT/machine/.venv', '--upgrade', '--no-deps', 'ROOT/machine/.venv/src/deribit-api-clients'] errored with the following return code 1, and output: 
  ERROR: Directory 'ROOT/machine/.venv/src/deribit-api-clients' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

@b1r3k

poetry @ git+https://github.com/python-poetry/poetry.git@ccbeffc1f1a5809e999cc18c584ed2cee2a8f580

@b1r3k ccbeffc is the 1.2.0b2 tag. You don’t have #5172, which has not yet been merged. You need to do:

pip install git+https://github.com/ashnair1/poetry.git@subdir-fix

this is again broken after new installer became the default (1.4.0). I was able to fix it locally by adding following:

if package.source_subdirectory is not None:
    archive = archive / Path(package.source_subdirectory)

after this line: https://github.com/python-poetry/poetry/blob/master/src/poetry/installation/executor.py#L571

fix: 🐛 fix all ImportError exceptions for the current datasets

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Navigation Menu

Repository navigation

Dependency rooted in a subfolder of a GIT repository #755

Feature request

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Dependency rooted in a subfolder of a GIT repository #755

Description

Feature request

Activity

bowendeng commented on Jan 2, 2019

stale commented on Nov 13, 2019

stale commented on Nov 20, 2019

zbarnett commented on Jan 31, 2020

finswimmer commented on Feb 2, 2020

velovix commented on Sep 4, 2020

TheButlah commented on Dec 18, 2020

sinoroc commented on Dec 18, 2020

finswimmer commented on Dec 19, 2020

17 remaining items

falonso-alo commented on Jan 25, 2022

ashnair1 commented on Feb 10, 2022

Felix-neko commented on Feb 14, 2022

rickmark commented on Mar 4, 2022

rickmark commented on Mar 18, 2022

StuartBertram commented on Apr 7, 2022

danodonovan commented on Jul 8, 2022

b1r3k commented on Jul 29, 2022

andersk commented on Jul 31, 2022

nikowatari commented on Mar 1, 2023

github-actions commented on Feb 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions