Skip to content

docs: explain GITHUB_TOKEN forwarding for eval containers - #3857

Merged
Sayt-0 merged 1 commit into
docker:mainfrom
priyanka25aug:docs/eval-github-token-forwarding
Jul 28, 2026
Merged

Sayt-0 merged 1 commit into
docker:mainfrom
priyanka25aug:docs/eval-github-token-forwarding

Conversation

@priyanka25aug

Copy link
Copy Markdown
Contributor

Fixes #3841

Eval containers deliberately do not receive GITHUB_TOKEN / GH_TOKEN, even though dedicated provider API keys are forwarded automatically (nonForwardableTokenEnvVars in pkg/config/auto.go excludes GITHUB_TOKEN). This is intentional but undocumented, and the asymmetry with other providers is confusing.

This PR adds a warning to the evaluation docs and the GitHub Copilot provider docs:

  • How to pass the token explicitly with -e GITHUB_TOKEN
  • That --env-from-file alone is not sufficient
  • That the LLM judge runs on the host, so judge scoring can succeed while the evaluated agent fails to authenticate

@priyanka25aug
priyanka25aug requested a review from a team as a code owner July 27, 2026 23:14
@aheritier aheritier added area/docs Documentation changes kind/docs Documentation-only changes labels Jul 27, 2026

@Sayt-0 Sayt-0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good thank you for the contribution !

@Sayt-0

Sayt-0 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

One more thing : can you sign your commits ? It's mandatory in this repo

@priyanka25aug
priyanka25aug force-pushed the docs/eval-github-token-forwarding branch from c17d62e to 76c1051 Compare July 28, 2026 13:48
GITHUB_TOKEN and GH_TOKEN are not forwarded into eval containers by design
(nonForwardableTokenEnvVars in pkg/config/auto.go). Add a WARNING callout
with the -e GITHUB_TOKEN workaround to both the evaluation docs and the
GitHub Copilot provider docs.

Fixes docker#3841

Signed-off-by: Priyanka Bajaj <priyanka.bajaja@gmail.com>
@priyanka25aug
priyanka25aug force-pushed the docs/eval-github-token-forwarding branch from 76c1051 to badc210 Compare July 28, 2026 13:58
@Sayt-0
Sayt-0 merged commit 877df71 into docker:main Jul 28, 2026
14 checks passed
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
docker-agent 1.119.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>This release introduces first-class plan management with a new CLI command group and TUI browser, adds YAML safety mode defaults, and includes several bug fixes for runtime stability and structured output handling.

## What's New

- Adds `docker agent plans` CLI command group and a host-facing plan service package for managing plans from outside the agent runtime
- Adds a `/plans` browser, detail view, and action dialogs to the TUI for interactive plan management
- Adds YAML safety mode defaults to team runtime, individual agents, user settings, and aliases, with declarative precedence rules
- Adds an `Active agents only` option in TUI settings to filter the sidebar to agents participating in the current session
- Logs a warning when Docker Desktop serves an expired token to the models gateway

## Bug Fixes

- Fixes runtime re-entry loop when a content-only turn ends with a bare EOF and no `finish_reason` from the provider
- Fixes structured output handling for Claude models served through OpenAI-compatible endpoints by reinforcing schema constraints in system instructions
- Fixes evaluation budget termination outcomes so `budget_exceeded` results are preserved correctly across transcripts, SQLite, and session JSON
- Hardens host plan management and stabilizes asynchronous TUI workflows for plans
- Fixes atomic file replacement on Windows to allow plan storage reads during open file operations

## Technical Changes

- Exports `ValidateName`, `CorruptPlanError`, `SharedStorage`, and `ChangeNotifier` from the plan package
- Adds `PlanChangedEvent` emission on shared plan mutations in the runtime
- Centralizes plan editor handling in the TUI
- Adds plan storage testing on Windows as a CI gate
- Adds documentation for the Provider Credentials section in eval containers, clarifying `GITHUB_TOKEN` forwarding behavior
---

## What's Changed
* docs: update CHANGELOG.md for v1.118.0 by @docker-read-write[bot] in docker/docker-agent#3861
* fix(openai): reinforce structured output for Claude proxies by @Sayt-0 in docker/docker-agent#3863
* feat(config): add YAML safety mode defaults by @Sayt-0 in docker/docker-agent#3860
* fix(eval): preserve budget termination outcomes by @Sayt-0 in docker/docker-agent#3862
* docs: explain GITHUB_TOKEN forwarding for eval containers by @priyanka25aug in docker/docker-agent#3857
* feat: add first-class host UX for plan management (#3844) by @Sayt-0 in docker/docker-agent#3853
* feat(tui): filter sidebar to active agents by @Sayt-0 in docker/docker-agent#3864
* fix(plans): address review follow-ups from #3853 by @Sayt-0 in docker/docker-agent#3865
* fix(runtime): stop content-only turns that end with a bare EOF (no `finish_reason`) by @awschmeder in docker/docker-agent#3669
* feat(desktop): log when Docker Desktop serves an expired token by @trungutt in docker/docker-agent#3867

## New Contributors
* @priyanka25aug made their first contribution in docker/docker-agent#3857
* @awschmeder made their first contribution in docker/docker-agent#3669

**Full Changelog**: docker/docker-agent@v1.118.0...v1.119.0
</pre>
  <p>View the full release notes at <a href="https://github.com/docker/docker-agent/releases/tag/v1.119.0">https://github.com/docker/docker-agent/releases/tag/v1.119.0</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!15464
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation changes kind/docs Documentation-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: Explain GITHUB_TOKEN forwarding for eval

4 participants