test: add HTTP request regression snapshots by penalosa · Pull Request #290 · cloudflare/cf

penalosa · 2026-10-13T10:11:57Z

Adds a dedicated Vitest + MSW regression suite that runs the real cf CLI for every generated API command and compares every recorded HTTP request with persisted snapshots. It is intended to catch request changes during large CLI refactors.

The suite covers 2,954 generated commands and 1,243 command groups. Each command runs with no arguments and with a large set of argument combinations: required values, individual flags, enum and boolean values, pairwise flag presence, positional alternates, aliases, global options, local mode, body inputs, and file uploads. Snapshots record the method, full URL, headers, JSON or binary body, and ordered multipart parts. Failures name the command, snapshot, changed case, and exact arguments alongside the request diff. Every command must also reach its declared method and endpoint.

Every command listed in hand-written-commands.json is excluded, along with groups that only contain hand-written commands. Those commands orchestrate local projects, external tools such as Docker, SSH and cloudflared, and multi-request workflows, which would need fixtures and fakes outside the scope of this suite. Mock responses are empty success envelopes apart from the zone-name lookup used by --zone <domain>. Local mode keeps request rewriting real and routes Miniflare's requests through MSW.

The command cases are split into 12 files balanced by case count. The new HTTP regression workflow runs them as 12 Linux GitHub Actions matrix jobs, one file per job, using Vitest --shard. The small harness tests run with the normal CLI suite on Linux, macOS, and Windows.

Generated snapshots make up almost all of the diff and are marked as generated in .gitattributes.

Validation:

pnpm check
pnpm --filter cf test:regression — 4,197 tests passed in 96.17s locally
pnpm --filter cf test src/__tests__/regression/harness.test.ts

pkg-pr-new · 2026-10-13T10:12:47Z

npm i https://pkg.pr.new/cf@290

commit: 26d71fc

github-actions · 2026-10-13T10:13:34Z

`cf --help` startup

	mean	stddev	min	max
PR	260.6 ms	4.4 ms	255.9 ms	272.2 ms
base	262.3 ms	4.1 ms	257.0 ms	270.1 ms

Δ (PR vs base): ↓ -1.8 ms (-0.7%)

Measured with hyperfine on ubuntu-latest (3 warmup runs, 20 timed runs). CI numbers carry ±a few ms of runner jitter; treat small deltas as noise.

ask-bonk · 2026-10-13T10:17:43Z

🤖 I've done a thorough review of your PR.

Adds a parallel MSW/Vitest snapshot suite that records request behavior for the generated and hand-written CLI command surface.

Warning: cf tunnels run is hand-written and bypasses the generic generated-command endpoint assertion. It performs /cfd_tunnel lookup requests, but is absent from workflowRequestPaths, so that request can silently disappear when its snapshot is updated.

"cf tunnels run": "/cfd_tunnel",
"cf tunnels tail": "/management",

Warning: The workflow assertion only checks a path substring, so it accepts prerequisite reads in place of the primary mutation. Preview deploy's GET .../deployments/latest passes in place of its POST .../deployments, and image deletion's HEAD passes in place of DELETE.

const workflowPath = workflowRequestPaths[command.command];
const workflowMethod =
  command.command === "cf previews deploy"
    ? "POST"
    : command.command === "cf containers images delete"
      ? "DELETE"
      : undefined;
if (workflowPath) {
  expect(
    recordings.some((recording) =>
      recording.requests.some(
        (request) =>
          (!workflowMethod || request.method === workflowMethod) &&
          new URL(request.url).pathname.includes(workflowPath)
      )
    ),
    `${command.command} never reached ${workflowPath}. Fix its fixture before updating the request baseline.`
  ).toBe(true);
}

ask-bonk · 2026-10-13T10:17:48Z

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots across the CLI surface.

Warning: cf tunnels run lacks workflow endpoint coverage.
Warning: Path-only workflow checks can accept preflight requests instead of mutations.

github run

ask-bonk · 2026-10-13T10:26:20Z

The workflow request guard has two gaps that let required hand-written API calls disappear when baselines are updated.

cf tunnels run is hand-written and bypasses the generated-command endpoint assertion. When no token is supplied it resolves the tunnel through GET /cfd_tunnel, but workflowRequestPaths does not require that request. Store an expected method with each path and include this flow:

export const workflowRequestPaths: Record<
	string,
	{ path: string; method: string }
> = {
	"cf ai run": { path: "/ai/run", method: "POST" },
	"cf docs search": { path: "/api/ai-search/search", method: "POST" },
	"cf registrar registrations create": {
		path: "/registrar/registrations",
		method: "POST",
	},
	"cf workers versions create": { path: "/versions", method: "POST" },
	"cf workers triggers deploy": { path: "/schedules", method: "PUT" },
	"cf previews deploy": { path: "/deployments", method: "POST" },
	"cf containers images list": { path: "/_catalog", method: "GET" },
	"cf containers images delete": { path: "/manifests/", method: "DELETE" },
	"cf d1 migrations list": { path: "/query", method: "POST" },
	"cf d1 migrations apply": { path: "/query", method: "POST" },
	"cf tunnels run": { path: "/cfd_tunnel", method: "GET" },
	"cf tunnels tail": { path: "/management", method: "POST" },
};

The suite checks only a pathname. Preview deploy records GET .../deployments/latest before its required POST .../deployments, and image deletion similarly sends HEAD before DELETE; dropping either mutation still satisfies the check. Require the mapped HTTP verb:

				const workflow = workflowRequestPaths[command.command];
				if (workflow) {
					expect(
						recordings.some((recording) =>
							recording.requests.some(
								(request) =>
									request.method === workflow.method &&
									new URL(request.url).pathname.includes(workflow.path)
							)
						),
						`${command.command} never reached ${workflow.method} ${workflow.path}. Fix its fixture before updating the request baseline.`
					).toBe(true);
				}

ask-bonk · 2026-10-13T10:26:48Z

Review: 2 findings.

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots and CI coverage.

Warning: cf tunnels run lacks a required API endpoint guard.
Warning: workflow guards check paths but not HTTP methods, allowing prerequisite requests to mask missing mutations.

Posted both fixes as GitHub suggestion comments.

github run

ask-bonk · 2026-10-13T10:30:59Z

The workflow guard needs to check the expected HTTP method as well as the path. The preview snapshot records GET .../deployments/latest before the required POST .../deployments, and image deletion records HEAD .../manifests/... before DELETE; removing a mutation and updating its snapshot would still pass.

export const workflowRequestPaths: Record<
	string,
	{ path: string; method: string }
> = {
	"cf ai run": { path: "/ai/run", method: "POST" },
	"cf docs search": { path: "/api/ai-search/search", method: "POST" },
	"cf registrar registrations create": {
		path: "/registrar/registrations",
		method: "POST",
	},
	"cf workers versions create": { path: "/versions", method: "POST" },
	"cf workers triggers deploy": { path: "/schedules", method: "PUT" },
	"cf previews deploy": { path: "/deployments", method: "POST" },
	"cf containers images list": { path: "/_catalog", method: "GET" },
	"cf containers images delete": { path: "/manifests/", method: "DELETE" },
	"cf d1 migrations list": { path: "/query", method: "POST" },
	"cf d1 migrations apply": { path: "/query", method: "POST" },
};

const workflow = workflowRequestPaths[command.command];
if (workflow) {
	expect(
		recordings.some((recording) =>
			recording.requests.some(
				(request) =>
					request.method === workflow.method &&
					new URL(request.url).pathname.includes(workflow.path)
			)
		),
		`${command.command} never reached ${workflow.method} ${workflow.path}. Fix its fixture before updating the request baseline.`
	).toBe(true);
}

ask-bonk · 2026-10-13T10:31:04Z

🤖 I've done a thorough review of your PR.

Adds parallel MSW request snapshots and Linux CI coverage.

Warning: Workflow guards accept prerequisite reads instead of required mutations. Posted a suggestion comment: test: add HTTP request regression snapshots #290 (comment)

github run

ask-bonk · 2026-10-13T10:35:00Z

No new concerns since the last review

github run

ask-bonk

The suite claims alias coverage, but every included invocation uses canonical option names. In particular, cf containers images delete is exercised with --force but never its public -f alias, so a regression in that yargs alias or telemetry alias mapping can be accepted by updating this snapshot. Generate one equivalent case for each OptionMeta.alias (all alias forms are represented, and the current included surface has -f).

ask-bonk · 2026-10-13T11:23:19Z

+	const canonicalCases = [...cases];
+	for (const alias of command.aliases ?? []) {
+		for (const scenario of canonicalCases) {
+			cases.push({
+				name: `alias ${alias}: ${scenario.name}`,
+				args: scenario.args,
+				commandPath: [...command.fullPath.slice(0, -1), alias],
+			});
+		}
+	}


Suggested change

const canonicalCases = [...cases];

for (const alias of command.aliases ?? []) {

for (const scenario of canonicalCases) {

cases.push({

name: `alias ${alias}: ${scenario.name}`,

args: scenario.args,

commandPath: [...command.fullPath.slice(0, -1), alias],

});

}

}

const canonicalCases = [...cases];

for (const option of options) {

const aliases =

typeof option.alias === "string"

? [option.alias]

: (option.alias ?? []);

for (const alias of aliases) {

const args = withOptions(matrixBase, [option], source);

const optionIndex = args.findIndex(

(arg) =>

arg === `--${option.name}` ||

arg.startsWith(`--${option.name}=`)

);

if (optionIndex >= 0) {

const flag = alias.length === 1 ? `-${alias}` : `--${alias}`;

args[optionIndex] = flag;

cases.push({ name: `alias ${flag}`, args });

}

}

}

for (const alias of command.aliases ?? []) {

for (const scenario of canonicalCases) {

cases.push({

name: `alias ${alias}: ${scenario.name}`,

args: scenario.args,

commandPath: [...command.fullPath.slice(0, -1), alias],

});

}

}

ask-bonk · 2026-10-13T11:23:32Z

🤖 I've done a thorough review of your PR.

Adds HTTP request regression snapshots and CI coverage.

Warning: option aliases are not exercised. Posted a suggestion for alias cases, including cf containers images delete -f.

github run

ask-bonk · 2026-10-13T11:37:58Z

No new concerns since the last review

github run

ask-bonk · 2026-10-13T12:03:13Z

LGTM

github run

penalosa added 3 commits October 13, 2026 10:44

test: add parallel HTTP regression snapshots for every cf command

ee43d2b

test: exclude workflow commands from HTTP regression snapshots

e0fb8ba

ci: run HTTP regression snapshots on Linux only

425ddf1

chore: mark regression snapshots as generated

543cc24

penalosa added 2 commits October 13, 2026 11:22

test: simplify HTTP regression boundaries

8be6b74

ci: run HTTP regression snapshots in a separate job

f40ae69

test: exclude cloudflared commands from HTTP regression snapshots

251c417

ci: run HTTP regression snapshots in sharded workflow

b571f1e

test: use Vitest default workers for regression suite

002f0e6

ask-bonk Bot reviewed Oct 13, 2026

View reviewed changes

test: limit HTTP regression snapshots to generated commands

a226444

test: omit SDK runtime headers from regression snapshots

26d71fc

petebacondarwin requested review from a team and NuroDev and removed request for a team October 23, 2026 12:59

-	const canonicalCases = [...cases];
-	for (const alias of command.aliases ?? []) {
-		for (const scenario of canonicalCases) {
-			cases.push({
-				name: `alias ${alias}: ${scenario.name}`,
-				args: scenario.args,
-				commandPath: [...command.fullPath.slice(0, -1), alias],
-			});
-		}
-	}
+	const canonicalCases = [...cases];
+	for (const option of options) {
+		const aliases =
+			typeof option.alias === "string"
+				? [option.alias]
+				: (option.alias ?? []);
+		for (const alias of aliases) {
+			const args = withOptions(matrixBase, [option], source);
+			const optionIndex = args.findIndex(
+				(arg) =>
+					arg === `--${option.name}` ||
+					arg.startsWith(`--${option.name}=`)
+			);
+			if (optionIndex >= 0) {
+				const flag = alias.length === 1 ? `-${alias}` : `--${alias}`;
+				args[optionIndex] = flag;
+				cases.push({ name: `alias ${flag}`, args });
+			}
+		}
+	}
+	for (const alias of command.aliases ?? []) {
+		for (const scenario of canonicalCases) {
+			cases.push({
+				name: `alias ${alias}: ${scenario.name}`,
+				args: scenario.args,
+				commandPath: [...command.fullPath.slice(0, -1), alias],
+			});
+		}
+	}

Conversation

penalosa commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cf --help startup

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot left a comment

Choose a reason for hiding this comment

Uh oh!

ask-bonk Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

ask-bonk Bot commented Oct 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

penalosa commented Oct 13, 2026 •

edited

Loading

pkg-pr-new Bot commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading

`cf --help` startup