Skip to content

[BUG] CI/CD external_contributions_tests does not work as expected #114

Description

@davidberenstein1957

Describe the bug

There are two related issues:

  • Incorrect User Role Detection: Some contributors (e.g., @davidberenstein1957, @nifleisch) are not recognized as OWNER, MEMBER, or COLLABORATOR even though we are part of the team.
  • Tests Not Running Against PR Code: The external_contributions_tests suite does not seem to test the actual code introduced in the PR. It runs against a different codebase/version, possibly from a cached or default source retrieved from the --extra-index.

A flow of what happened:

What I did

The author_association field in GitHub's API seems unreliable or affected by:

  • Team visibility settings (private vs. public)
  • Whether a contributor is added directly to the repo
  • Potential GitHub caching issues

We tested changing visibility and adding direct contributor access (thanks @sharpenb) and re-running the test, but no resolution yet. Some values may be cached here too.

No luck so far, perhaps some values are cached.

Expected behavior

  • Contributors should be accurately identified (e.g., as MEMBER).
  • external_contributions_tests should always test the code proposed in the PR branch, not some default or cached source.
  1. Improve Role Detection Logic
  • Use GitHub API fallback methods:
  • Consider logging or debugging role recognition in CI to catch misclassification.
  1. Ensure PR Code is Tested
  • Confirm that external_contributions_tests uses checkout@v3 or newer with ref: ${{ github.event.pull_request.head.ref }}.
  • Avoid dependency resolution via --extra-index that might point to outdated packages.

Screenshots

Environment

  • pruna version:
  • python version:
  • Operating System:

Additional context

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions