Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
packages/cli/src/commands/_generated/** linguist-generated
packages/cli/src/__tests__/regression/** text eol=lf
packages/cli/src/__tests__/regression/snapshots/** linguist-generated
44 changes: 44 additions & 0 deletions .github/workflows/regression.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: HTTP regression

# Snapshots every HTTP request made by the cf command tree. The suite has 12
# balanced files; each GitHub Actions shard runs one file.

on:
pull_request:
merge_group:
types: [checks_requested]
branches:
- main
push:
branches:
- main

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
regression:
if: ${{ github.repository_owner == 'cloudflare' }}
name: regression (${{ matrix.shard }}/12)
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 1
persist-credentials: false

- name: Install dependencies
uses: ./.github/actions/install-dependencies

- name: Snapshot cf commands
run: pnpm --filter cf test:regression --shard=${{ matrix.shard }}/12
1 change: 1 addition & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"build:test": "vp run task:build:test",
"check:type": "vp run task:check:type",
"test": "vp run task:test",
"test:regression": "vp test run --config regression.config.ts",
"test:watch": "vp run task:test:watch",
"test:imports": "vp run task:test:imports"
},
Expand Down
17 changes: 17 additions & 0 deletions packages/cli/regression.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
import { defineConfig } from "vite-plus";
import base from "./vite.config.js";

export default defineConfig({
resolve: base.resolve,
test: {
...base.test,
env: { TZ: "UTC", CI: "1", CF_SEND_TELEMETRY: "0" },
include: ["src/__tests__/regression/shards/*.test.ts"],
exclude: [],
server: {
deps: {
inline: ["@cloudflare/workers-utils", "@cloudflare/workers-auth"],
},
},
},
});
53 changes: 53 additions & 0 deletions packages/cli/src/__tests__/regression/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# HTTP request regression suite

From the repository root:

```sh
pnpm --filter cf test:regression
pnpm --filter cf test:regression -t 'cf zones list$'
pnpm --filter cf test:regression -t 'cf zones list$' -u
pnpm --filter cf test:regression --shard=1/12
```

Install against the frozen lockfile and run `pnpm generate` first when preparing
a fresh checkout. The suite has twelve command files balanced by case count.
Locally, Vitest runs them in parallel worker forks. CI runs the separate
`.github/workflows/regression.yml` workflow as twelve GitHub Actions shards,
each running one file. Each fork owns its temporary working directory, MSW
interceptor, credentials and state, and each command owns its snapshot.
Arguments after `test:regression` go directly to Vitest. The harness checks in
`harness.test.ts` run with the normal CLI test suite.

`cases.ts` covers generated API commands from the tracked command catalogue,
plus every command group that contains one. It omits every command listed in
`hand-written-commands.json`, because those commands orchestrate local
projects, external tools and multi-request workflows. Every remaining leaf and
group runs without arguments. Leaves also run
with required arguments, individual optional flags, all enum choices, both
boolean values, pairwise flag presence/absence, alternate positional values,
aliases, global flags, local mode, raw/file/array bodies and derived body fields.
Equivalent argument vectors are deduplicated. Pairwise coverage is deliberately
bounded rather than an exponential Cartesian product; it covers flag presence,
not every combination of values. Invalid combinations remain in the baseline
with their exit code and error, alongside successful requests.

The real CLI parser, command modules, SDK and HTTP clients run against MSW.
Invocations use deterministic dummy API-token credentials in the environment.
Mock responses are empty success envelopes, apart from the zone-name lookup
used by `--zone <domain>`. Every command must reach its declared method and
endpoint. Local mode keeps request rewriting real and routes Miniflare's
requests through MSW.

Snapshots under `snapshots/<command/path>.snap` contain named cases, complete
arguments, exit outcomes and ordered requests. Requests preserve method, full
URL/query encoding, headers, JSON structure, binary bytes and ordered multipart
parts (including repeated names, file names and MIME types). Only CLI/Node
versions, temporary/fixture paths, random multipart boundaries and multipart
content lengths (which depend on those paths/boundaries) are normalized. Part
contents remain recorded. Fixtures use LF on every OS. No live account is needed.

On failure, Vitest names the command and snapshot file and shows the changed
request beside its case and arguments. Inspect the diff before using `-u`;
review method, URL, authentication/other headers and serialized body. If a new
command cannot reach HTTP, fix its fixture arguments or workflow responses
before accepting a baseline. Commit the reviewed snapshots with the change.
Loading
Loading