Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions .github/workflows/pr-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -106,17 +106,32 @@ jobs:
npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"
}

install_gemini() {
if command -v gemini >/dev/null 2>&1; then
echo "gemini-cli cache hit: $(gemini --version)"
return 0
fi
npm install -g @google/gemini-cli
}

install_gemini_best_effort() {
install_gemini || true
if ! command -v gemini >/dev/null 2>&1; then
echo "::warning::gemini CLI unavailable after best-effort install — Gemini fallback will be disabled."
fi
}

case "$REVIEW_ENGINE" in
claude)
install_claude
# Best-effort install of fallbacks for duck/fallback support
npm install -g @google/gemini-cli || true
install_gemini_best_effort
if ! env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version > /dev/null 2>&1; then
echo "::warning::gh copilot built-in unavailable with user token — rate-limit fallback will be unavailable."
fi
;;
gemini)
npm install -g @google/gemini-cli
install_gemini
install_claude || true
;;
copilot)
Expand All @@ -125,7 +140,7 @@ jobs:
gh extension install github/gh-copilot || true
fi
install_claude || true
npm install -g @google/gemini-cli || true
install_gemini_best_effort
;;
*)
echo "::error::Unknown REVIEW_ENGINE=$REVIEW_ENGINE"
Expand Down
245 changes: 163 additions & 82 deletions docs/pr-review-agent/machine-user-setup.md
Original file line number Diff line number Diff line change
@@ -1,127 +1,208 @@
# Machine User Setup for PR Review Agent

This project authenticates via a **machine user account** with a fine-grained PAT. This replaces the previous GitHub App approach, which could not satisfy CODEOWNERS approval requirements (see [issue #27](https://github.com/petry-projects/.github-private/issues/27)).

## Why Machine User?

- Can be added to an org team listed in CODEOWNERS
- Approvals count as code owner reviews
- Simple PAT-based auth — no JWT generation step needed
- Works identically to a human reviewer from GitHub's perspective

## Step 1: Create the Machine User Account

1. Create a new GitHub account (e.g., `donpetry-bot`) with a shared org email alias
2. Add the account to the `petry-projects` organization
3. Create an org team (e.g., `petry-projects/pr-reviewers`) and add the machine user

## Step 2: Configure CODEOWNERS

In each target repo, add the team to `CODEOWNERS`:
The agent authenticates via a **machine user** GitHub account whose classic
PAT is stored in the workflow secret `DON_PETRY_BOT_GH_PAT`. The bot posts
approvals using this token; its review counts as a CODEOWNERS approval when
the bot is in the team listed in `CODEOWNERS`.

> [!IMPORTANT]
> **Use a classic PAT. Fine-grained PATs do not work for this workflow.**
>
> After the migration to `petry-projects/.github-private`, fine-grained PATs
> fail the GraphQL `addPullRequestReview` mutation with:
>
> ```
> failed to create review: GraphQL: Resource not accessible by personal access token (addPullRequestReview)
> ```
>
> The failure persists even when every obvious gate is satisfied: the org owner
> has approved the token request, fine-grained PATs are allowed for
> `petry-projects`, the bot has Write collaborator access on the target repo,
> and the PAT lists the right repository / permissions. Classic PATs with
> `repo` scope work; fine-grained ones do not. If you ever see the error above
> in a workflow log, the secret is holding a fine-grained token — generate a
> classic one and replace it.

## Why a machine user

- It can be added to an org team referenced in `CODEOWNERS`, so its approvals
satisfy code-owner review requirements.
- It works identically to a human reviewer from GitHub's perspective —
no JWT exchange, no app installation flow, no opaque integration ID.
- A simple PAT-based auth keeps the workflow short.

(GitHub Apps were tried and abandoned: app accounts cannot be in `CODEOWNERS`,
so app-account approvals don't satisfy code-owner requirements. See the
"Trade-offs" table at the bottom for the full comparison.)

## Step 1: Create the machine user account

1. Sign out of your primary account (or use a private window).
2. Create a new GitHub account, e.g. `donpetry-bot`. Use a dedicated email
alias you can route to your inbox (e.g. `you+donpetry-bot@gmail.com`).
3. Verify the email, complete account setup, and turn on 2FA.

## Step 2: Add the bot to the org and the CODEOWNERS team

1. Sign in as your primary account (org owner).
2. **github.com/organizations/petry-projects/settings/members** →
**Invite member** → enter `donpetry-bot` → role: **Member**.
3. Accept the invite from the bot account.
4. Create or open the org team that's referenced in `CODEOWNERS`
(e.g. `petry-projects/pr-reviewers`).
5. Add `donpetry-bot` to that team.
6. On every target repo, ensure the bot is a **Write** collaborator
(org-level team membership grants the role; verify on
`github.com/<org>/<repo>/settings/access`).

In each repo whose PRs the agent reviews, the `CODEOWNERS` file should
reference the team:

```
# .github/CODEOWNERS
* @petry-projects/pr-reviewers
```

Or use path-specific rules as needed.
Or path-specific rules as needed.

## Step 3: Generate a Fine-Grained PAT
## Step 3: Generate a classic PAT for the bot

1. Sign in as the machine user account
2. Go to **Settings → Developer settings → Fine-grained personal access tokens**
3. Create a new token:
- **Token name:** `pr-review-agent`
- **Expiration:** 90 days (set a calendar reminder to rotate)
- **Resource owner:** `petry-projects`
- **Repository access:** All repositories (or select specific repos)
- **Permissions:**
- **Repository → Contents:** Read-only
- **Repository → Pull requests:** Read and write
- **Repository → Commit statuses:** Read-only
- **Repository → Checks:** Read-only
- **Organization → Members:** Read-only
1. Sign in **as the bot** (`donpetry-bot`). The PAT must be created from the
bot's account, not yours — sign out of your primary account first.
2. **Settings → Developer settings → Personal access tokens → Tokens (classic)**
→ **Generate new token (classic)**.
3. Settings:
- **Note:** `pr-review-agent`
- **Expiration:** 1 year (set a calendar reminder to rotate)
- **Scopes:**
- ✅ `repo` (full control of private repos) — required for the
`addPullRequestReview` mutation
- ✅ `workflow` — required when AI delegation pushes workflow-file changes
- ✅ `read:org` — silences `Missing required token scopes: 'read:org'`
in workflow logs and lets the agent resolve `@org/team` mentions in
CODEOWNERS escalations
4. **Generate token** and copy it immediately (you won't see it again).

## Step 4: Store the PAT as an Org Secret
## Step 4: Store the PAT in `petry-projects/.github-private`

```bash
gh secret set DON_PETRY_BOT_GH_PAT --org petry-projects --body "<paste-token>"
```
Sign back in as your primary account, then either via the UI:

**github.com/petry-projects/.github-private/settings/secrets/actions**
→ **New repository secret** (or **Update**) → name `DON_PETRY_BOT_GH_PAT`,
value = the token from Step 3.

Or store at repo level if preferred:
Or via `gh`:

```bash
gh secret set DON_PETRY_BOT_GH_PAT --repo petry-projects/.github-private --body "<paste-token>"
gh secret set DON_PETRY_BOT_GH_PAT \
--repo petry-projects/.github-private \
--body "<paste-token>"
```

### Verify the secret is set:
Verify:

```bash
gh secret list --repo petry-projects/.github-private
gh secret list --repo petry-projects/.github-private | grep DON_PETRY_BOT_GH_PAT
```

Should show `DON_PETRY_BOT_GH_PAT` in the list.
### Secret naming reference

## Step 5: Test the Setup
| Secret | Purpose |
|--------|---------|
| `DON_PETRY_BOT_GH_PAT` | Machine user / bot reviewer PAT. Used by `pr-review.yml`, `repair-pr-approvals.yml`, `daily-pr-review-health.yml`, and `claude.yml` for GitHub API authentication as the bot. |
| `GH_PAT` | User PAT with a Copilot subscription. Exposed as `COPILOT_GITHUB_TOKEN` in `pr-review.yml` for Copilot CLI access. Do **not** store the bot reviewer PAT here. |

```bash
# Test dry-run
gh workflow run fix-stuck-prs.yml --repo petry-projects/.github-private -f dry_run=true

# Check the logs
gh run view <run-number> --repo petry-projects/.github-private --log | grep "Logged in"
```
## Step 5: Verify the setup

Should show the machine user account name.
Trigger a one-shot dry-run of the PR review workflow:

## Step 6: Clean Up Old GitHub App Secrets
```bash
gh workflow run pr-review.yml \
--repo petry-projects/.github-private \
-f dry_run=true
```

Once the migration is validated:
Then read the run's `Install review engine CLIs` step. The first command is
`gh auth status`, which should report the bot's login (not yours) and the
three scopes from Step 3:

```bash
gh secret delete APP_ID --repo petry-projects/.github-private
gh secret delete APP_INSTALLATION_ID --repo petry-projects/.github-private
gh secret delete APP_PRIVATE_KEY --repo petry-projects/.github-private
```
Logged in to github.com account donpetry-bot
- Token scopes: 'read:org', 'repo', 'workflow'
```

Optionally uninstall/delete the GitHub App from [org settings](https://github.com/organizations/petry-projects/settings/apps).
If the login is your primary account, the secret is holding the wrong PAT —
recreate it from the bot's account (Step 3) and update the secret (Step 4).

## PAT Rotation
If the run reaches the `[approve] Posting APPROVED review...` line and
returns the `Resource not accessible by personal access token` error, the
secret is holding a fine-grained PAT (see the warning at the top).

Fine-grained PATs have a configurable expiry (90-day recommended). To rotate:
## PAT rotation

1. Sign in as the machine user
2. Generate a new fine-grained PAT with the same scopes
3. Update the secret: `gh secret set DON_PETRY_BOT_GH_PAT --repo petry-projects/.github-private --body "<new-token>"`
4. Revoke the old token
Classic PATs expire on the date set when generated (1 year recommended).

1. Sign in as the bot.
2. Generate a new classic PAT with the same scopes.
3. Update the secret:
```bash
gh secret set DON_PETRY_BOT_GH_PAT \
--repo petry-projects/.github-private \
--body "<new-token>"
```
4. Revoke the old token from the bot's
**Settings → Developer settings → Tokens (classic)** page.

## Trade-offs vs GitHub App

| Aspect | Machine User (PAT) | GitHub App (previous) |
|--------|-------------------|----------------------|
| CODEOWNERS support | Yes | No |
| Token expiry | Manual (90-day fine-grained) | Automatic (1-hour JWT) |
| Permissions | PAT scopes | Fine-grained App manifest |
| Account overhead | Requires a GitHub seat | No human account |
| Setup complexity | Simple | Medium |
| Aspect | Machine User (classic PAT) | GitHub App |
|---------------------|-------------------------------------------|----------------------------------|
| CODEOWNERS support | Yes (team membership grants approval) | No (apps can't be in CODEOWNERS) |
| Token lifetime | Manual (1 year recommended) | Automatic (1-hour JWT) |
| Permission model | PAT scopes (org/repo coarse) | Fine-grained app manifest |
| Account overhead | Requires a free GitHub account + 2FA | No human account |
| Branch-protection / rulesets bypass | Honored when bot is on the bypass list | Sometimes blocked |
| Setup complexity | Low | Medium |

## Troubleshooting

### Issue: "Resource not accessible by integration"
- Verify the PAT has the required scopes
- Check the machine user has access to the target repos
- Ensure `DON_PETRY_BOT_GH_PAT` secret is set correctly
### `Resource not accessible by personal access token (addPullRequestReview)`

The PAT in `DON_PETRY_BOT_GH_PAT` is fine-grained, not classic. Generate a
classic PAT (Step 3) and update the secret (Step 4). The fine-grained version
is blocked by an org-policy gate that has no UI surface — there's no
configuration that makes it work.

### `Missing required token scopes: 'read:org'`

The classic PAT was generated without `read:org`. Edit the token at
**Settings → Developer settings → Tokens (classic) → Edit** and check the
`read:org` box. (No need to regenerate — editing scopes is sufficient.)

### `gh auth status` reports the wrong account

The PAT was created from your primary account by mistake, not the bot.
Sign out of your primary account, sign in as the bot, regenerate the token
(Step 3), and update the secret (Step 4).

### Approval doesn't satisfy code-owner requirement

- Confirm the bot is on the team listed in the repo's `CODEOWNERS`
(org admin → team → Members).
- Validate `CODEOWNERS` syntax:
`gh api repos/<owner>/<repo>/codeowners/errors`
- Confirm the repo's branch protection / ruleset has
`Require review from Code Owners` enabled.

### Issue: Approval doesn't satisfy code owner requirement
- Verify the machine user is in the team listed in CODEOWNERS
- Check the repo's CODEOWNERS file is valid (`gh api repos/OWNER/REPO/codeowners/errors`)
- Ensure branch protection has `require_code_owner_review: true`
### Token expired

### Issue: Token expired
- Generate a new fine-grained PAT and update the `DON_PETRY_BOT_GH_PAT` secret
Sign in as the bot, regenerate the classic PAT (Step 3), and update the
secret (Step 4).

## References

- [GitHub: Managing CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners)
- [GitHub: Fine-grained PATs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens)
- [Issue #27: Why GitHub Apps can't be in CODEOWNERS](https://github.com/petry-projects/.github-private/issues/27)
- [GitHub: Classic PATs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic)
- [`AGENT.md`](AGENT.md) — full agent design and runtime behavior
- [`BOT_SETUP.md`](BOT_SETUP.md) — abbreviated setup checklist
8 changes: 4 additions & 4 deletions scripts/engine.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ case "$REVIEW_ENGINE" in
ENGINE_SINGLE_MODEL="claude-opus-4-6"
ENGINE_LABEL="triage: haiku 4.5 → deep: sonnet 4.6 + duck: gpt-5.4 → audit: opus 4.6"
ENGINE_SINGLE_LABEL="single-reviewer mode: opus 4.6"
# Cross-engine rubber duck: always the opposite engine
# Cross-engine rubber duck: use Copilot when Claude is primary
DUCK_ENGINE="copilot"
DUCK_MODEL="gpt-5.4"
;;
Expand All @@ -61,7 +61,7 @@ case "$REVIEW_ENGINE" in
ENGINE_SINGLE_MODEL="gpt-5.4"
ENGINE_LABEL="triage: gpt-5-mini → deep: gpt-5.2 + duck: sonnet 4.6 → audit: gpt-5.4"
ENGINE_SINGLE_LABEL="single-reviewer mode: gpt-5.4"
# Cross-engine rubber duck: always the opposite engine
# Cross-engine rubber duck: use Claude when Copilot is primary
DUCK_ENGINE="claude"
DUCK_MODEL="claude-sonnet-4-6"
;;
Expand Down Expand Up @@ -227,8 +227,8 @@ sys.exit(1)

# run_duck <prompt_file> <model>
# Cross-engine adversarial "rubber duck" review.
# Always uses a different model family from REVIEW_ENGINE. Output to stdout.
# Strips the opposing engine's credentials to prevent cross-engine leakage.
# Uses Copilot when REVIEW_ENGINE=claude; otherwise uses Claude for Gemini/Copilot runs.
# Output to stdout. Strips non-selected engine credentials to prevent cross-engine leakage.
run_duck() {
local prompt_file="$1"
local model="$2"
Expand Down
4 changes: 2 additions & 2 deletions scripts/review-batch.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
# PRS_FILE — path to candidate list (default: prs.txt)
# MAX_PRS — stop after this many actual reviews (no-ops don't count)
# CANDIDATE_LIMIT — hard cap on candidates inspected (timeout backstop)
# REVIEW_ENGINE — primary engine (claude|copilot); may be flipped on
# rate-limit fallback
# REVIEW_ENGINE — primary engine (claude|gemini|copilot); may follow
# fallback chain claude -> gemini -> copilot
# GH_TOKEN — workflow auth (set at job level)
#
# Exit:
Expand Down
Loading