Support extensions hosting their own functional tests (public test-support module + source-scoped replace)

[vhvb1989]

Summary

Enable azd extensions to host their own functional tests inside their own extension folder (e.g. cli/azd/extensions/<id>/test/...) by exposing a public, internal-free test-support package in azd core that extensions can consume from source for build/test only.

This is a follow-up direction coming out of #8760 / PR #8754, which added PR-gate functional tests for the azure.ai.agents extension. Those tests currently live in cli/azd/test/functional/ai_agents/ (the azd core module) rather than in the extension folder, and the investigation below explains why and what core changes would unblock co-location.

Why extension tests can't live in the extension folder today

Each extension is a separate Go module (e.g. module azureaiagent) that consumes azd core as an external dependency pinned to a released version (e.g. github.com/azure/azure-dev/cli/azd v1.24.3), with no replace.

The functional-test helpers the tests rely on are cli/azd/test/azdcli and cli/azd/test/recording. Two problems block importing them from an extension module:

1. Internal-package barrier (small, easily fixable)

Because an extension module's import paths are azureaiagent/... — outside the github.com/azure/azure-dev/cli/azd/ tree — Go's internal-package rule forbids importing any cli/azd/internal/... package. Verified empirically:

use of internal package github.com/azure/azure-dev/cli/azd/internal/runcontext/agentdetect not allowed

The entire internal closure of test/azdcli + test/recording is just two packages:

• cli/azd/internal/runcontext/agentdetect
• cli/azd/internal (root) — pulled in only transitively through agentdetect

…and both trace back to a single line in test/azdcli/cli.go:

cmd.Env = append(cmd.Env, agentdetect.DisableAgentDetectEnvVar+"=1")

where DisableAgentDetectEnvVar = "AZD_DISABLE_AGENT_DETECT". Relocating that one constant to a public package removes both internal packages from the closure. No need to make the broad root internal package public.

2. Version-coupling barrier (the real one)

Even with the internal barrier removed, an extension consuming test/azdcli/test/recording from a pinned release of cli/azd would always lag the helper code by a release cycle. Test-harness changes (e.g. the cli.go token-format fix in PR #8754) aren't available until a cli/azd release + dependency bump. That's exactly why the tests currently live in core, sharing the module with the helpers.

Proposed core change

1. Carve out a public, internal-free test-support module for extension functional testing — e.g. cli/azd/test/extframework with its own go.mod — wrapping/relocating the reusable bits of azdcli, recording, cmdrecord, ostest. Prerequisite refactor: move AZD_DISABLE_AGENT_DETECT (and any other internal-only symbols the helpers touch) into a public package so the module is internal-free.

2. Let extensions pin it from source via a scoped replace. Important nuance: a replace in an extension's go.mod is global to the module. Replacing the whole cli/azd dependency (replace github.com/azure/azure-dev/cli/azd => ../../) would also redirect the production dependency to source — which is the thing currently forbidden to merge (see cli/azd/extensions/<id>/AGENTS.md), because release binaries must build against the pinned, tagged cli/azd.

   Making the test-support code a separate module lets the extension replace only that module from source:

   // extension go.mod — test-only support pinned from source; production cli/azd pin untouched
   require github.com/azure/azure-dev/cli/azd/test/extframework v0.0.0
   replace github.com/azure/azure-dev/cli/azd/test/extframework => ../../../test/extframework
   

   Because the extension's production code never imports the test-support module, this replace only affects go test builds, not the released binary. This is the key to the reporter's question — "always consuming from source is fine" holds only when the from-source replace is scoped to a test-only module that production never imports; it does not hold for a blanket replace of the whole cli/azd module.

Reusable interactive-CLI driver (for interactive Tier 1/Tier 2 tests)

Beyond the non-interactive runner, the module should ship a reusable interactive-CLI driver so extensions can test interactive flows (wizards, list-pickers) by defining tests only, not by hand-rolling a per-extension PTY harness. This replaces the tmux + Python driver prototyped in #8758 with a Go-native one.

It wraps the Go-native expect stack already in azd's dependency graph (cli/azd/go.sum):

• github.com/creack/pty — real PTY for the spawned azd binary
• github.com/Netflix/go-expect — expect-style send/expect API (the Go equivalent of Python pexpect)
• github.com/hinshun/vt10x — terminal emulator that renders ANSI/cursor sequences into a 2D screen (the equivalent of tmux capture-pane -p)

Capability parity with the tmux/Python driver was assessed: PTY, literal text, special/arrow keys (raw escape codes wrapped as named-key helpers), rendered-screen capture (vt10x.State.String()), substring/regex waiting, live streaming, timeouts/watchdog, and sequential-session flow are all covered — and it removes the detached-tmux-server cleanup problem (the PTY + child are owned by the test process). Only open item: a one-time spike confirming vt10x renders the real azd ai agent init wizard identically to a real terminal.

Sketch of the extension-facing API (driver lives in the shared module; tests are just send/expect/assert):

sess := extframework.StartInteractive(t, "ai", "agent", "init") // PTY + vt10x + go-expect, auto-cleanup via t.Cleanup
sess.ExpectString("Select a subscription")
sess.Send(extframework.KeyDown, extframework.KeyEnter)           // named-key helpers
sess.ExpectString("Enter a name")
sess.SendLine("my-agent")
sess.ExpectString("added to your azd project successfully")
require.Contains(t, sess.Screen(), "Done")                      // rendered snapshot (== capture-pane)

out, code := extframework.Run(t, "provision", "--no-prompt")    // non-interactive phases reuse the plain runner

The driver shares binary resolution (CLI_TEST_AZD_PATH) and the recording session with the non-interactive runner, so the same driver serves Tier 1 (recorded/playback) and Tier 2 (live). Any Go extension pins it from source via the test-only replace and writes interactive tests with no per-extension harness.

Acceptance criteria

• A public, internal-free test-support module exists (own go.mod), reusing the existing recording/playback proxy infrastructure.

• AZD_DISABLE_AGENT_DETECT (and any other internal symbols the helpers need) relocated to a public package; test/azdcli/recording no longer import cli/azd/internal/....

• An extension (pilot: azure.ai.agents) can place functional tests under its own folder, importing the test-support module via a source-scoped replace, with the production cli/azd dependency still pinned to a release.

• CI builds/runs the extension's in-folder functional tests (record/playback), with no replace of the production cli/azd module.

• Docs updated (cli/azd/AGENTS.md + extension AGENTS.md) to describe the sanctioned source-scoped test replace pattern vs. the forbidden production replace.

• The module exposes a reusable interactive-CLI driver (StartInteractive/send/expect/Screen() + named-key helpers) built on creack/pty + Netflix/go-expect + hinshun/vt10x, with auto-cleanup via t.Cleanup.

• Spike: confirm vt10x renders the real azd ai agent init wizard identically to a real terminal (parity with tmux capture-pane).

• The interactive driver works for both recorded (Tier 1) and live (Tier 2) runs, sharing CLI_TEST_AZD_PATH and the recording session with the non-interactive runner.

• Pilot: migrate Add live golden-path (Tier 2) pipeline for azd ai agent extension #8758's tmux/Python Tier 2 init step to the Go interactive driver to prove parity.

Notes / open questions

• Confirm the released-binary pipeline never picks up the test-only replace (it shouldn't, since production code doesn't import the module, but worth a guard/lint check).
• Decide whether to publish/tag the test-support module or keep it source-only (source-only replace is sufficient for build/test and avoids a release cadence dependency).
• Consider a lint/CI check that fails if an extension go.mod contains a replace of github.com/azure/azure-dev/cli/azd itself (the forbidden case), while allowing a replace scoped to the test-support module.

Related: #8760, #8754