From d81ab6c38153a841139345cd3d2250d5a260eafc Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:27:54 -0400 Subject: [PATCH 01/13] docs: add CI testing guidelines (nested lifecycle, ID capture, compose separation) Canonical design reference for the CI test suite: four nested lifecycle scopes (session/suite/test/step) with guaranteed teardown, dynamic ID capture rules, layered test flow, unique-names convention, and the rationale for keeping docker-compose.ci.yml separate from the dev docker-compose.yml. Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/TESTING_GUIDELINES.md | 248 +++++++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 cicd/TESTING_GUIDELINES.md diff --git a/cicd/TESTING_GUIDELINES.md b/cicd/TESTING_GUIDELINES.md new file mode 100644 index 0000000000..4f7f164746 --- /dev/null +++ b/cicd/TESTING_GUIDELINES.md @@ -0,0 +1,248 @@ +# CI Testing Guidelines + +Design rules for the TestLink CI test suite under `cicd/`. Framework-agnostic — applies whether the runner is YAML-based, Vitest, or plain shell. + +--- + +## 1. Purpose + +TestLink is a test management system. Its value is the correctness of its API and data operations. These guidelines define **how we test it** so that: + +- CI runs are reproducible and deterministic +- Test failures point to the real cause, not infrastructure drift +- Tests can be run in isolation, reordered, or rerun without manual cleanup +- Teardown is guaranteed, even on failure + +--- + +## 2. Design Principles + +1. **Tests talk through the public API.** XML-RPC and REST are the contracts under test. Go through them like a real client would. +2. **Direct DB access is reserved for three jobs only:** seeding the admin API key, asserting side effects that are not exposed via API, and emergency cleanup. Never use SQL to shortcut a test setup that could go through the API. +3. **Every test owns its data.** A test creates what it needs, asserts against it, and deletes it. Tests do not rely on residue from previous tests. +4. **IDs flow through capture, never hardcoded.** The ID of a created entity comes from the creation response, not from assuming "it'll be id=1". +5. **Teardown is guaranteed.** Every scope's teardown runs in a `finally` / `trap EXIT` so that a failure does not leak state into the next run. +6. **Idempotency.** Running the same suite twice in a row against a fresh environment produces identical results. + +--- + +## 3. Nested Lifecycle Scopes + +Four scopes nest like Russian dolls. Each has its own `setup → run children → teardown`, with teardown always guaranteed. + +``` +SESSION (once per CI run) + └── SUITE (once per suite: smoke, auth, crud, workflow, …) + └── TEST CASE (once per test) + └── STEP (one action) +``` + +### 3.1 Session scope + +Runs once per CI invocation. Owns the infrastructure. + +**Setup:** +- `docker compose -f cicd/docker-compose.ci.yml up -d` +- Wait for healthchecks +- Load schema: `testlink_create_tables.sql` + UDFs +- Load default data: `testlink_create_default_data.sql` +- Seed the admin API key (`UPDATE users SET script_key='…' WHERE id=1`) + +**Teardown (always):** +- `docker compose -f cicd/docker-compose.ci.yml down -v --remove-orphans` + +**Rule:** session setup owns infrastructure and baseline (admin + API key) **only**. No test-specific data. + +### 3.2 Suite scope + +One per feature area (auth, crud, workflow, …). Owns shared fixtures for that area. + +**Setup:** +- Create a dedicated test project with a unique name (`suite--`) +- Capture the project ID for all tests in the suite +- Create any users, roles, or custom fields this suite needs + +**Teardown (always):** +- Delete the suite's test project (cascades to everything under it) + +**Rule:** no suite may depend on another suite's residue. Any suite must be runnable alone and in any order. + +### 3.3 Test case scope + +One per test. Owns the minimum entities the test needs. + +**Setup:** +- Create the entities the test acts on (test suite, test case, test plan, build) +- Use unique names tied to the test ID: `-` +- Capture every created ID + +**Teardown (always):** +- Delete the entities this test created, in reverse order + +**Rule:** a test must be runnable twice in a row against the same suite fixtures without manual cleanup between runs. + +### 3.4 Step scope + +One API call with one assertion. No hidden side effects. If a step creates state that outlives itself, the creation must be explicit and captured. + +--- + +## 4. Test Flow + +Test categories layer on top of each other. A failure at a lower layer should short-circuit the higher layers — no point testing CRUD if the server isn't responding. + +| Layer | Tests | Purpose | +|---|---|---| +| **Smoke** | `tl.ping`, login page loads | Is the system alive? | +| **Auth** | valid key, invalid key, missing key | Can we talk to it? | +| **CRUD (leaf-up)** | create/read/update/delete for each entity in dependency order: **project → testsuite → testcase → testplan → build → execution** | Do basic operations work? | +| **Workflow** | assign case to plan → create build → record execution → query results | Do operations compose end-to-end? | +| **Negative** | bad input, missing required fields, permission denied, FK violations | Does it fail correctly? | +| **Regression** | specific bug repros (one test per closed bug) | Does a past bug stay fixed? | + +**Short-circuit rule:** if Smoke fails, skip everything below. Implementation: make Auth depend on Smoke, CRUD on Auth, and so on. The runner skips dependents when a dep fails. + +--- + +## 5. Dynamic ID Capture + +IDs are never hardcoded. They flow from creation responses into subsequent steps via captured variables. + +### 5.1 Rule + +> **No numeric ID literal (`1`, `2`, `42`) may appear in a test step except in assertions about the response shape itself.** + +### 5.2 Pattern + +```yaml +- name: Create test project + command: | + curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ + -H "Content-Type: text/xml" \ + -d '' + capture: + projectId: "$[name=id].value" # resolve from response + +- name: Create test suite under it + command: | + curl -sf … testprojectid={{projectId}} testsuitename=suite-{{runId}} … + capture: + suiteId: "$[name=id].value" + +- name: Delete test suite + command: curl -sf … testsuiteid={{suiteId}} … + +- name: Delete test project + command: curl -sf … prjid={{projectId}} … +``` + +### 5.3 Rationale + +- Runs are reorderable: no dependency on seed counters. +- Runs are re-runnable: no "id=1 already exists" collisions. +- Parallel safety: different runs pick different IDs without coordination. +- Failures are traceable: logs show which concrete IDs were in play. + +--- + +## 6. Test Data Rules + +1. **Unique names.** Every created entity gets a name that includes the run ID or a timestamp: `proj-20260419-143022`. Never `proj`, never `CI Test Project`. +2. **Prefix by test ID** where practical (`TC-CRUD-003-project-…`) so orphaned data is traceable. +3. **Capture every ID** from creation responses. Never assume the next auto-increment value. +4. **Delete in reverse creation order.** Children before parents. +5. **Idempotency.** A second run against the same fixtures must behave identically. Verify by running the suite twice locally before committing. +6. **No implicit ordering between test cases** beyond declared `dependencies`. If test B silently needs test A's data, declare the dependency or duplicate the setup. + +--- + +## 7. Teardown Guarantees + +Each scope's teardown must run **even when a test or step fails**. Implementation depends on scope: + +| Scope | Guarantee mechanism | +|---|---| +| Session | Wrapper script with `trap 'ci-down.sh' EXIT` | +| Suite | Runner executes suite-teardown in a `finally` block after all tests, regardless of results | +| Test case | Runner executes test-teardown in a `finally` block after all steps, regardless of step outcomes | +| Step | Step is atomic; no sub-teardown | + +A teardown failure must surface loudly but must **not** prevent outer teardowns from running. + +--- + +## 8. Why `docker-compose.ci.yml` Is Separate from `docker-compose.yml` + +The repo has two compose files on purpose. They serve different audiences and have incompatible lifecycle assumptions. This section is grounded in the actual current files (`docker-compose.yml` at repo root and `cicd/docker-compose.ci.yml`). + +| Aspect | `docker-compose.yml` (dev) | `cicd/docker-compose.ci.yml` (CI) | +|---|---|---| +| Audience | Developers working locally | Automated CI and test runs | +| Lifecycle | `up` once, leave running for days | `up → test → down -v` every run | +| DB state | Named volume `postgres` — persistent across restarts | No named DB volume — anonymous, destroyed by `down -v` | +| Services | `db` + `maildev` + `app` + `restore` (profile=tools, opt-in) | `db` + `app` only | +| DB credentials | `teste` / `teste` | `testlink` / `testlink` | +| Config injection | None — relies on defaults baked into the image | Mounts `ci-config_db.inc.php` (CI DB creds) + `ci-custom_config.inc.php` (API on, SMTP off) | +| Seed data | Optional: `restore` service loads `docs/db_sample` sample data | `init-db.sh` loads schema + default data + hardcoded admin API key | +| Image build | `build: .` from repo root Dockerfile | `build: ..` from the same Dockerfile | +| Image pinning | `postgres:9.6` pinned; `maildev:latest` floats | Fully pinned (`postgres:9.6`) | +| Host port | `0.0.0.0:8090:80` | `8090:80` | + +### Important: port 8090 collides + +Both compose files publish the app on host port **8090**. They cannot run at the same time. If the dev stack is up, CI setup will fail to bind. This is a known constraint — tests and the dev environment are serialized on a single developer machine. Long-term, the CI compose should either take a different port or use ephemeral host port assignment to remove this footgun. + +### Why the separation is mandatory + +1. **Teardown safety.** CI teardown runs `down -v`, which destroys volumes. The dev compose's `postgres` named volume holds a developer's in-progress database. Because the two compose files live in different directories (`./` and `cicd/`), Docker Compose gives them different project namespaces (`testlink-code` vs `cicd`) — so their volumes, containers, and networks are isolated. `down -v` run against the CI compose cannot touch the dev compose's `postgres` volume. Merging them would break this guarantee. +2. **Reproducibility of seed and config.** CI must start from a known state: fresh schema, default data, known admin API key `a1b2…`, API enabled, SMTP off. The dev compose intentionally does not inject any of that — it relies on whatever the developer has set up. Merging the two would either corrupt dev state on every CI run or require runtime conditionals that obscure both flows. +3. **Credentials and data isolation.** Different DB users (`teste` vs `testlink`), different injected configs, different (or no) seed data. A single file trying to serve both audiences would need env-var gymnastics that make each flow harder to reason about. +4. **Service minimalism.** CI starts only `db` + `app`. Dev adds `maildev` for catching outbound mail and `restore` (profile-gated) for loading sample data. CI doesn't need either — faster startup, narrower failure surface, fewer moving parts when diagnosing a test failure. + +### Naming convention + +- Dev compose: **`docker-compose.yml`** at the repo root. The file `docker compose up` picks up by default — optimize for developer ergonomics. +- CI compose: **`cicd/docker-compose.ci.yml`**, explicit path, never the default. Living under `cicd/` also changes Docker Compose's default project name to `cicd`, which is what keeps volumes and containers isolated from the dev stack. + +**Rule:** any new compose file for a specific purpose (e2e-only, perf testing, …) gets its own name under a dedicated directory (not the repo root). Never overload `docker-compose.yml`. + +--- + +## 9. Directory Conventions + +``` +cicd/ +├── docker-compose.ci.yml # CI environment definition +├── scripts/ # Session-scope lifecycle +│ ├── ci-up.sh # Session setup (build + up + seed) +│ ├── ci-down.sh # Session teardown (down -v) +│ ├── init-db.sh # Schema + default data + admin API key +│ ├── ci-config_db.inc.php # DB config injected into app +│ └── ci-custom_config.inc.php # App config (API on, SMTP off) +├── tests/ +│ ├── src/ # Test runner (TypeScript) +│ └── testcases/ # Test definitions +│ ├── smoke/ +│ ├── auth/ +│ ├── crud/ +│ ├── workflow/ +│ ├── negative/ +│ └── regression/ +└── results/ # Per-run output (gitignored) +``` + +**Rule:** infrastructure code (scripts, compose) lives in `cicd/`. Test definitions live in `cicd/tests/testcases/`. Nothing test-related lives outside `cicd/`. + +--- + +## 10. Checklist for Writing a New Test + +- [ ] Suite category is correct (smoke / auth / crud / workflow / negative / regression) +- [ ] `dependencies` is declared if this test needs another test's entities to already pass +- [ ] All created entities use unique names that include the run ID or timestamp +- [ ] All IDs used in later steps are `capture:`d from earlier responses — no hardcoded integers +- [ ] Test creates exactly what it needs and deletes it in reverse order in teardown +- [ ] Test passes when run alone: `npm run test -- --id TC-XXX-NNN` +- [ ] Test passes when run twice in a row against the same session +- [ ] Assertions check response shape and content, not just exit code +- [ ] Teardown runs even when a prior step fails (verified by injecting a failure locally) From 485e6d78da8a4a0af01556c06a3f8599a1f81518 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:30:05 -0400 Subject: [PATCH 02/13] fix(ci): point log collector at docker-compose.ci.yml The log collector was pointed at `/docker/`, a directory that contains no compose file. `docker compose logs` in that cwd failed to start and the collector silently disabled itself, so per-test log extraction never produced output. Replace `RunConfig.dockerComposePath` (directory) with `composeFile` (absolute path to the compose file). Invoke `docker compose -f logs --follow --timestamps` with cwd set to the project root. Default target is the CI compose at `cicd/docker-compose.ci.yml`. Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/tests/src/cli.ts | 6 +++--- cicd/tests/src/executor.ts | 3 ++- cicd/tests/src/log-collector.ts | 12 +++++++----- cicd/tests/src/types.ts | 4 ++-- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/cicd/tests/src/cli.ts b/cicd/tests/src/cli.ts index b9a04c351d..cd88e11336 100644 --- a/cicd/tests/src/cli.ts +++ b/cicd/tests/src/cli.ts @@ -45,7 +45,7 @@ program const testsDir = path.dirname(new URL(import.meta.url).pathname); const projectRoot = path.resolve(testsDir, '..', '..', '..'); const testcasesDir = path.join(testsDir, '..', 'testcases'); - const dockerDir = path.join(projectRoot, 'docker'); + const composeFile = path.join(projectRoot, 'cicd', 'docker-compose.ci.yml'); // Generate output directory with timestamp const timestamp = startTime.toISOString().replace(/[:.]/g, '-').substring(0, 19); @@ -66,11 +66,11 @@ program outputDir, outputFormat: options.format as RunConfig['outputFormat'], workingDir: projectRoot, - dockerComposePath: dockerDir, + composeFile, }; process.stderr.write(`\n[CONFIG] Project root: ${projectRoot}\n`); - process.stderr.write(`[CONFIG] Docker compose: ${dockerDir}\n`); + process.stderr.write(`[CONFIG] Compose file: ${composeFile}\n`); process.stderr.write(`[CONFIG] Testcases: ${testcasesDir}\n`); process.stderr.write(`[CONFIG] Output: ${outputDir}\n`); process.stderr.write(`[CONFIG] LLM Judge: ${config.noLlm ? 'disabled' : config.judgeUrl}\n`); diff --git a/cicd/tests/src/executor.ts b/cicd/tests/src/executor.ts index 9e6a7f0c3e..301fa41674 100644 --- a/cicd/tests/src/executor.ts +++ b/cicd/tests/src/executor.ts @@ -325,7 +325,8 @@ ${r.stderr || '(empty)'} if (needsLogCollector) { this.logCollector = new LogCollector( - this.config.dockerComposePath, + this.config.composeFile, + this.config.workingDir, this.config.outputDir ); try { diff --git a/cicd/tests/src/log-collector.ts b/cicd/tests/src/log-collector.ts index 4167ff1e06..9c35f6fc77 100644 --- a/cicd/tests/src/log-collector.ts +++ b/cicd/tests/src/log-collector.ts @@ -28,15 +28,17 @@ export class LogCollector { private process: ChildProcess | null = null; private sessionFile: string; private logFileStream: WriteStream | null = null; - private dockerComposeDir: string; + private composeFile: string; + private workingDir: string; private isRunning: boolean = false; private testMarkers: Map = new Map(); private writeQueue: Promise = Promise.resolve(); private lineBuffer: string = ''; private outputDir: string; - constructor(dockerComposeDir: string, outputDir: string) { - this.dockerComposeDir = dockerComposeDir; + constructor(composeFile: string, workingDir: string, outputDir: string) { + this.composeFile = composeFile; + this.workingDir = workingDir; this.outputDir = outputDir; this.sessionFile = `/tmp/${CONFIG.sessionPrefix}-${Date.now()}.log`; } @@ -56,9 +58,9 @@ export class LogCollector { this.process = spawn( 'docker', - ['compose', 'logs', '--follow', '--timestamps'], + ['compose', '-f', this.composeFile, 'logs', '--follow', '--timestamps'], { - cwd: this.dockerComposeDir, + cwd: this.workingDir, stdio: ['ignore', 'pipe', 'pipe'], } ); diff --git a/cicd/tests/src/types.ts b/cicd/tests/src/types.ts index f91d2242c3..a808cfc840 100644 --- a/cicd/tests/src/types.ts +++ b/cicd/tests/src/types.ts @@ -231,8 +231,8 @@ export interface RunConfig { outputFormat: 'console' | 'json'; /** Working directory (project root) */ workingDir: string; - /** Path to docker-compose directory for log collection */ - dockerComposePath: string; + /** Absolute path to the compose file used for log collection */ + composeFile: string; } /** From 213f3a8710ed4f057f23f2390f9abc389edd661b Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:33:55 -0400 Subject: [PATCH 03/13] refactor(ci): move session lifecycle into run-tests.sh wrapper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Session setup (ci-up.sh) and teardown (ci-down.sh) used to live inside test-case steps — TC-INTEGRATION-001 started the stack, TC-E2E-001's last step tore it down. That made teardown conditional on the e2e test reaching its final step: any earlier failure left containers and volumes behind. Running only one suite was inconsistent too (build skipped lifecycle entirely, integration started but never stopped). Introduce cicd/scripts/run-tests.sh as the single entry point. It: - Runs ci-up.sh before tests (except for --suite build, which only exercises the image artifact). - Traps EXIT/INT/TERM to invoke ci-down.sh regardless of outcome — pass, fail, Ctrl-C, or crash all produce a clean teardown. - Passes remaining args through to the tsx CLI. Call sites updated: - TC-INTEGRATION-001 and TC-E2E-001 no longer invoke ci-up/ci-down. - TC-INTEGRATION-001's spurious dependency on TC-BUILD-002 removed; the compose image is built by ci-up.sh, and TC-BUILD-002 builds a separate testlink-ci-test tag only used by TC-BUILD-001. - package.json npm scripts route through the wrapper. - .github/workflows/test-suite.yml calls the wrapper from repo root. Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/test-suite.yml | 7 ++- cicd/scripts/run-tests.sh | 53 +++++++++++++++++++ cicd/tests/package.json | 8 +-- cicd/tests/testcases/e2e/TC-E2E-001.yml | 7 +-- .../integration/TC-INTEGRATION-001.yml | 18 +++---- 5 files changed, 68 insertions(+), 25 deletions(-) create mode 100755 cicd/scripts/run-tests.sh diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml index 71f4268e91..22157c01d7 100644 --- a/.github/workflows/test-suite.yml +++ b/.github/workflows/test-suite.yml @@ -68,8 +68,6 @@ jobs: - name: Run tests id: run-tests run: | - cd cicd/tests - # Build judge flags based on input JUDGE_FLAGS="" if [ "${{ inputs.judge_mode }}" = "simple" ] || [ -z "${{ inputs.judge_mode }}" ]; then @@ -88,8 +86,9 @@ jobs: echo "Judge mode: ${{ inputs.judge_mode || 'simple' }}" echo "Judge flags: $JUDGE_FLAGS" - # Run tests with JSON output - npx tsx src/cli.ts run $SUITE_FLAG $JUDGE_FLAGS --format json > /tmp/test-results.json || true + # Run through the wrapper: handles ci-up before and ci-down after + # (trap EXIT) so teardown is guaranteed even on test failure. + bash cicd/scripts/run-tests.sh $SUITE_FLAG $JUDGE_FLAGS --format json > /tmp/test-results.json || true echo "--- JSON Results ---" cat /tmp/test-results.json diff --git a/cicd/scripts/run-tests.sh b/cicd/scripts/run-tests.sh new file mode 100755 index 0000000000..afb89fe0fc --- /dev/null +++ b/cicd/scripts/run-tests.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Session-scope test runner with guaranteed teardown. +# +# Brings up the CI stack (for suites that need it), runs the test CLI, +# and tears the stack down on any exit path — pass, fail, Ctrl-C, or +# crash. The build suite skips infrastructure since it only exercises +# the image artifact. +# +# Usage: +# cicd/scripts/run-tests.sh [--suite ] [--id ] [tsx cli flags] +# +# Examples: +# cicd/scripts/run-tests.sh # all suites +# cicd/scripts/run-tests.sh --suite integration +# cicd/scripts/run-tests.sh --suite build # no ci-up/ci-down +# cicd/scripts/run-tests.sh --id TC-E2E-001 --no-llm + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + +ARGS=("$@") + +# The build suite lints PHP and builds the image via `docker build`. +# It does not talk to a running stack, so skip the heavy lifecycle. +NEEDS_STACK=true +for ((i=0; i<${#ARGS[@]}; i++)); do + if [ "${ARGS[i]}" = "--suite" ] && [ "${ARGS[i+1]:-}" = "build" ]; then + NEEDS_STACK=false + break + fi +done + +cleanup() { + local exit_code=$? + if [ "$NEEDS_STACK" = "true" ]; then + echo "" + echo "=== Session teardown (exit code: $exit_code) ===" + "$SCRIPT_DIR/ci-down.sh" || echo "[WARN] Teardown failed — inspect containers manually" + fi + exit "$exit_code" +} + +if [ "$NEEDS_STACK" = "true" ]; then + # Arm the trap before ci-up so a failed startup still triggers teardown. + trap cleanup EXIT INT TERM + echo "=== Session setup ===" + "$SCRIPT_DIR/ci-up.sh" +fi + +cd "$REPO_ROOT/cicd/tests" +npx tsx src/cli.ts run "${ARGS[@]}" diff --git a/cicd/tests/package.json b/cicd/tests/package.json index f685a28ca7..1ded7a90e8 100644 --- a/cicd/tests/package.json +++ b/cicd/tests/package.json @@ -6,10 +6,10 @@ "main": "dist/index.js", "scripts": { "build": "tsc", - "test": "tsx src/cli.ts run", - "test:build": "tsx src/cli.ts run --suite build", - "test:integration": "tsx src/cli.ts run --suite integration", - "test:e2e": "tsx src/cli.ts run --suite e2e", + "test": "bash ../scripts/run-tests.sh", + "test:build": "bash ../scripts/run-tests.sh --suite build", + "test:integration": "bash ../scripts/run-tests.sh --suite integration", + "test:e2e": "bash ../scripts/run-tests.sh --suite e2e", "list": "tsx src/cli.ts list", "dev": "tsx src/cli.ts" }, diff --git a/cicd/tests/testcases/e2e/TC-E2E-001.yml b/cicd/tests/testcases/e2e/TC-E2E-001.yml index c58e783842..e31100ac60 100644 --- a/cicd/tests/testcases/e2e/TC-E2E-001.yml +++ b/cicd/tests/testcases/e2e/TC-E2E-001.yml @@ -87,10 +87,6 @@ steps: expectPatterns: - "Success|status.*1|true" - - name: Tear down CI environment - command: bash ./cicd/scripts/ci-down.sh - timeout: 30000 - criteria: | Verify full CRUD workflow through XML-RPC API: - Create a test project with prefix CIT @@ -99,4 +95,5 @@ criteria: | - Retrieve and verify the test case - Delete the test case via tl.deleteTestCase - Delete the test suite via tl.deleteTestSuite - - Clean up the environment + Session teardown (docker compose down -v) is handled by + cicd/scripts/run-tests.sh after the run completes. diff --git a/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml b/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml index db71a152aa..56e2aa9365 100644 --- a/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml +++ b/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml @@ -1,16 +1,11 @@ id: TC-INTEGRATION-001 -name: TestLink Container Startup +name: TestLink Login Page Responds suite: integration priority: 1 -timeout: 180000 -dependencies: - - TC-BUILD-002 +timeout: 30000 +dependencies: [] steps: - - name: Start CI environment - command: bash ./cicd/scripts/ci-up.sh - timeout: 150000 - - name: Verify app is responding command: | curl -sf http://localhost:8090/login.php | grep -o "Login" | head -1 @@ -18,7 +13,6 @@ steps: - "Login" criteria: | - Verify TestLink starts and serves the login page: - - Docker containers start (app + postgres) - - Database is initialized with schema and default data - - Login page is accessible on port 8090 + Verify the login page is served on port 8090. + Session setup (docker compose up, DB init) is handled by + cicd/scripts/run-tests.sh before any test runs. From e6b3be4fad80ecf947c708e1b39cbfecfcc62c6e Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:41:30 -0400 Subject: [PATCH 04/13] feat(ci): add runId/testId auto-vars, XML-RPC helper, new suite names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pieces of framework scaffolding for the test restructure: 1. Executor auto-populates `{{runId}}` (Date.now()) and `{{testId}}` (test case ID) as captured variables at the start of each test. Tests can use these to generate unique entity names without bespoke bash. 2. Add cicd/scripts/xmlrpc-capture.sh — reads an XML-RPC methodCall document from stdin, POSTs to the TestLink API, mirrors the response to stderr for expectPatterns, and emits structured JSON on stdout for the framework's capture: mechanism. Replaces ~6 lines of inline bash extraction in every CRUD step. 3. SUITES list updated to match the guidelines: build, smoke, auth, crud, workflow, negative, regression. Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/scripts/xmlrpc-capture.sh | 51 ++++++++++++++++++++++++++++++++++ cicd/tests/src/config.ts | 10 ++++++- cicd/tests/src/executor.ts | 5 +++- 3 files changed, 64 insertions(+), 2 deletions(-) create mode 100755 cicd/scripts/xmlrpc-capture.sh diff --git a/cicd/scripts/xmlrpc-capture.sh b/cicd/scripts/xmlrpc-capture.sh new file mode 100755 index 0000000000..2c08337de9 --- /dev/null +++ b/cicd/scripts/xmlrpc-capture.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# XML-RPC helper for test cases. +# +# Reads an XML-RPC document from stdin, POSTs it to the +# TestLink API, mirrors the raw response to stderr (so expectPatterns +# can match), and emits structured JSON on stdout for the test +# framework's capture: mechanism. +# +# Emitted JSON shape: +# {"ok": true, "id": N} — response contained an +# {"ok": true} — response had no (no-value call) +# {"ok": false, "faultCode": N} — server returned a +# +# Usage in a YAML step: +# command: | +# bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' +# ... +# XMLDOC +# capture: +# projectId: "id" +# +# Env: +# TL_URL — base URL of the TestLink instance (default http://localhost:8090) + +set -eo pipefail + +URL="${TL_URL:-http://localhost:8090}/lib/api/xmlrpc/v1/xmlrpc.php" +REQUEST=$(cat) + +RESPONSE=$(curl -sS -X POST "$URL" \ + -H "Content-Type: text/xml" \ + --data-binary "$REQUEST") + +# Mirror the response to stderr for expectPatterns / debugging. +printf '%s\n' "$RESPONSE" >&2 + +# First in the body — typically the created entity id. +ID=$(printf '%s' "$RESPONSE" | grep -oP '(?<=)\d+' | head -1 || true) + +# Fault code (if the server returned a ). +FAULT=$(printf '%s' "$RESPONSE" \ + | grep -oP '(?<=faultCode)\d+' \ + | head -1 || true) + +if [ -n "${FAULT:-}" ]; then + printf '{"ok": false, "faultCode": %s}\n' "$FAULT" +elif [ -n "${ID:-}" ]; then + printf '{"ok": true, "id": %s}\n' "$ID" +else + printf '{"ok": true}\n' +fi diff --git a/cicd/tests/src/config.ts b/cicd/tests/src/config.ts index 771457eda2..5c7c0955f8 100644 --- a/cicd/tests/src/config.ts +++ b/cicd/tests/src/config.ts @@ -7,7 +7,15 @@ /** * Available test suites - extend this for your project. */ -export const SUITES = ['build', 'integration', 'e2e'] as const; +export const SUITES = [ + 'build', + 'smoke', + 'auth', + 'crud', + 'workflow', + 'negative', + 'regression', +] as const; export type Suite = typeof SUITES[number]; /** diff --git a/cicd/tests/src/executor.ts b/cicd/tests/src/executor.ts index 301fa41674..45bb973b53 100644 --- a/cicd/tests/src/executor.ts +++ b/cicd/tests/src/executor.ts @@ -203,7 +203,10 @@ export class TestExecutor { this.currentTest++; this.currentTestId = testCase.id; - this.variables = {}; + this.variables = { + runId: Date.now().toString(), + testId: testCase.id, + }; this.progress( `[${timestamp}] [${this.currentTest}/${this.totalTests}] ${testCase.id}: ${testCase.name}` ); From c54559f5da94bbfcfd9cb3807fdb114608f015b5 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:45:45 -0400 Subject: [PATCH 05/13] refactor(ci): restructure tests into flow layers with dynamic IDs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorganize cicd/tests/testcases/ per the guidelines: drop the integration/e2e bucket names, organize by the test-flow pyramid (smoke → auth → crud → workflow). Rewrite each test to follow the three core rules: - Unique names — every created entity embeds {{testId}} and {{runId}} (auto-populated by the executor) so rerunning against the same DB never collides with residue. - Dynamic ID capture — the xmlrpc-capture.sh helper extracts the created entity's id from the XML response and emits JSON; the framework's capture: mechanism pipes it into {{projectId}}, {{suiteId}}, {{caseId}} for subsequent steps. - Per-test data ownership — every test creates its parent entities in setup steps and deletes them in reverse order as teardown steps. No test leaks data that another test depends on. Layout: smoke/ TC-SMOKE-001 login page responds TC-SMOKE-002 XML-RPC tl.ping auth/ TC-AUTH-001 valid API key accepted TC-AUTH-002 invalid API key rejected crud/ TC-CRUD-001 project CRUD TC-CRUD-002 test suite CRUD TC-CRUD-003 test case CRUD workflow/ TC-WORKFLOW-001 full entity-graph round-trip The old e2e/TC-E2E-001 is superseded by TC-WORKFLOW-001, which exercises the same graph but with captured IDs (the original hardcoded testprojectid=1, testsuiteid=2, external-id CIT-1). test-pipeline.yml updated to run the new suites in dependency order: build → smoke → auth → crud → workflow. Lower-layer failure short-circuits the higher layers. Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/test-pipeline.yml | 30 +++-- .github/workflows/test-suite.yml | 8 +- cicd/tests/testcases/auth/TC-AUTH-001.yml | 21 ++++ cicd/tests/testcases/auth/TC-AUTH-002.yml | 22 ++++ cicd/tests/testcases/crud/TC-CRUD-001.yml | 52 ++++++++ cicd/tests/testcases/crud/TC-CRUD-002.yml | 79 ++++++++++++ cicd/tests/testcases/crud/TC-CRUD-003.yml | 112 ++++++++++++++++++ cicd/tests/testcases/e2e/TC-E2E-001.yml | 99 ---------------- .../integration/TC-INTEGRATION-001.yml | 18 --- .../integration/TC-INTEGRATION-002.yml | 21 ---- .../integration/TC-INTEGRATION-003.yml | 29 ----- cicd/tests/testcases/smoke/TC-SMOKE-001.yml | 17 +++ cicd/tests/testcases/smoke/TC-SMOKE-002.yml | 19 +++ .../testcases/workflow/TC-WORKFLOW-001.yml | 111 +++++++++++++++++ 14 files changed, 462 insertions(+), 176 deletions(-) create mode 100644 cicd/tests/testcases/auth/TC-AUTH-001.yml create mode 100644 cicd/tests/testcases/auth/TC-AUTH-002.yml create mode 100644 cicd/tests/testcases/crud/TC-CRUD-001.yml create mode 100644 cicd/tests/testcases/crud/TC-CRUD-002.yml create mode 100644 cicd/tests/testcases/crud/TC-CRUD-003.yml delete mode 100644 cicd/tests/testcases/e2e/TC-E2E-001.yml delete mode 100644 cicd/tests/testcases/integration/TC-INTEGRATION-001.yml delete mode 100644 cicd/tests/testcases/integration/TC-INTEGRATION-002.yml delete mode 100644 cicd/tests/testcases/integration/TC-INTEGRATION-003.yml create mode 100644 cicd/tests/testcases/smoke/TC-SMOKE-001.yml create mode 100644 cicd/tests/testcases/smoke/TC-SMOKE-002.yml create mode 100644 cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml diff --git a/.github/workflows/test-pipeline.yml b/.github/workflows/test-pipeline.yml index 0a5dc972a1..2297a44090 100644 --- a/.github/workflows/test-pipeline.yml +++ b/.github/workflows/test-pipeline.yml @@ -24,18 +24,34 @@ jobs: suite: build judge_mode: ${{ inputs.judge_mode || 'simple' }} - integration: - name: Integration Tests + smoke: + name: Smoke Tests needs: build uses: ./.github/workflows/test-suite.yml with: - suite: integration + suite: smoke judge_mode: ${{ inputs.judge_mode || 'simple' }} - e2e: - name: E2E Tests - needs: integration + auth: + name: Auth Tests + needs: smoke uses: ./.github/workflows/test-suite.yml with: - suite: e2e + suite: auth + judge_mode: ${{ inputs.judge_mode || 'simple' }} + + crud: + name: CRUD Tests + needs: auth + uses: ./.github/workflows/test-suite.yml + with: + suite: crud + judge_mode: ${{ inputs.judge_mode || 'simple' }} + + workflow: + name: Workflow Tests + needs: crud + uses: ./.github/workflows/test-suite.yml + with: + suite: workflow judge_mode: ${{ inputs.judge_mode || 'simple' }} diff --git a/.github/workflows/test-suite.yml b/.github/workflows/test-suite.yml index 22157c01d7..d36a2eac93 100644 --- a/.github/workflows/test-suite.yml +++ b/.github/workflows/test-suite.yml @@ -9,8 +9,12 @@ on: type: choice options: - "build" - - "integration" - - "e2e" + - "smoke" + - "auth" + - "crud" + - "workflow" + - "negative" + - "regression" - "all" judge_mode: description: "Test judge mode" diff --git a/cicd/tests/testcases/auth/TC-AUTH-001.yml b/cicd/tests/testcases/auth/TC-AUTH-001.yml new file mode 100644 index 0000000000..34af039354 --- /dev/null +++ b/cicd/tests/testcases/auth/TC-AUTH-001.yml @@ -0,0 +1,21 @@ +id: TC-AUTH-001 +name: checkDevKey accepts the admin API key +suite: auth +priority: 1 +timeout: 30000 +dependencies: + - TC-SMOKE-002 + +steps: + - name: tl.checkDevKey with valid key + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.checkDevKey + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + + XMLDOC + expectPatterns: + - "1|true" + +criteria: | + The admin API key seeded by init-db.sh is accepted by tl.checkDevKey. diff --git a/cicd/tests/testcases/auth/TC-AUTH-002.yml b/cicd/tests/testcases/auth/TC-AUTH-002.yml new file mode 100644 index 0000000000..3b31e1dab1 --- /dev/null +++ b/cicd/tests/testcases/auth/TC-AUTH-002.yml @@ -0,0 +1,22 @@ +id: TC-AUTH-002 +name: checkDevKey rejects an invalid API key +suite: auth +priority: 2 +timeout: 30000 +dependencies: + - TC-SMOKE-002 + +steps: + - name: tl.checkDevKey with bogus key + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.checkDevKey + devKeynot-a-real-key-{{runId}} + + XMLDOC + expectPatterns: + - "faultCode|invalid|2000" + +criteria: | + An unknown API key is rejected with a fault response — the server + does not silently return a success envelope for bad credentials. diff --git a/cicd/tests/testcases/crud/TC-CRUD-001.yml b/cicd/tests/testcases/crud/TC-CRUD-001.yml new file mode 100644 index 0000000000..f3b7e4854f --- /dev/null +++ b/cicd/tests/testcases/crud/TC-CRUD-001.yml @@ -0,0 +1,52 @@ +id: TC-CRUD-001 +name: Test project CRUD +suite: crud +priority: 1 +timeout: 60000 +dependencies: + - TC-AUTH-001 + +steps: + - name: Create test project + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectnameproj-{{testId}}-{{runId}} + testcaseprefixP{{runId}} + notes{{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + projectId: "id" + + - name: Read project back by name + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.getTestProjectByName + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectnameproj-{{testId}}-{{runId}} + + XMLDOC + expectPatterns: + - "proj-{{testId}}-{{runId}}" + + - name: Delete project (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + prefixP{{runId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + +criteria: | + Test project CRUD via XML-RPC: + - Create a project with a unique name and prefix derived from runId. + - Read it back by name and verify the name round-trips. + - Delete it by prefix so no residue remains in the DB. + IDs are captured from the create response; no hardcoded integers. diff --git a/cicd/tests/testcases/crud/TC-CRUD-002.yml b/cicd/tests/testcases/crud/TC-CRUD-002.yml new file mode 100644 index 0000000000..ffd345cb98 --- /dev/null +++ b/cicd/tests/testcases/crud/TC-CRUD-002.yml @@ -0,0 +1,79 @@ +id: TC-CRUD-002 +name: Test suite CRUD +suite: crud +priority: 2 +timeout: 60000 +dependencies: + - TC-AUTH-001 + +steps: + - name: Create parent project (setup) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectnameproj-{{testId}}-{{runId}} + testcaseprefixP{{runId}} + notes{{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + projectId: "id" + + - name: Create test suite under the project + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectid{{projectId}} + testsuitenamesuite-{{testId}}-{{runId}} + detailsCreated by {{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + suiteId: "id" + + - name: Read test suite back + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.getTestSuiteByID + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testsuiteid{{suiteId}} + + XMLDOC + expectPatterns: + - "suite-{{testId}}-{{runId}}" + + - name: Delete test suite (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testsuiteid{{suiteId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + + - name: Delete parent project (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + prefixP{{runId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + +criteria: | + Test suite CRUD via XML-RPC: + - Create a parent project (setup), capturing its ID. + - Create a suite under the project, capturing its ID from the response. + - Read it back by ID and confirm the name round-trips. + - Delete the suite, then the project, in reverse creation order. + All IDs flow through capture; no numeric literals used as parameters. diff --git a/cicd/tests/testcases/crud/TC-CRUD-003.yml b/cicd/tests/testcases/crud/TC-CRUD-003.yml new file mode 100644 index 0000000000..a3c5161381 --- /dev/null +++ b/cicd/tests/testcases/crud/TC-CRUD-003.yml @@ -0,0 +1,112 @@ +id: TC-CRUD-003 +name: Test case CRUD +suite: crud +priority: 3 +timeout: 60000 +dependencies: + - TC-AUTH-001 + +steps: + - name: Create parent project (setup) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectnameproj-{{testId}}-{{runId}} + testcaseprefixP{{runId}} + notes{{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + projectId: "id" + + - name: Create parent test suite (setup) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectid{{projectId}} + testsuitenamesuite-{{testId}}-{{runId}} + detailsCreated by {{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + suiteId: "id" + + - name: Create test case + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcasenamecase-{{testId}}-{{runId}} + testsuiteid{{suiteId}} + testprojectid{{projectId}} + authorloginadmin + summaryCreated by {{testId}} + steps + step_number1 + actionsnoop + expected_resultsok + + + XMLDOC + expectPatterns: + - "" + capture: + caseId: "id" + + - name: Read test case back + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.getTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcaseid{{caseId}} + + XMLDOC + expectPatterns: + - "case-{{testId}}-{{runId}}" + + - name: Delete test case (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcaseid{{caseId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + + - name: Delete parent suite (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testsuiteid{{suiteId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + + - name: Delete parent project (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + prefixP{{runId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + +criteria: | + Test case CRUD via XML-RPC: + - Create parent project and suite with unique names. + - Create a test case in the suite, capturing its internal ID. + - Read it back by ID and verify the name round-trips. + - Delete in reverse order (case → suite → project). + All IDs flow through capture; no numeric literals used as parameters. diff --git a/cicd/tests/testcases/e2e/TC-E2E-001.yml b/cicd/tests/testcases/e2e/TC-E2E-001.yml deleted file mode 100644 index e31100ac60..0000000000 --- a/cicd/tests/testcases/e2e/TC-E2E-001.yml +++ /dev/null @@ -1,99 +0,0 @@ -id: TC-E2E-001 -name: Full CRUD Workflow via XML-RPC -suite: e2e -priority: 1 -timeout: 120000 -dependencies: - - TC-INTEGRATION-003 - -steps: - - name: Create test project - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.createTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testprojectnameCI Test Project - testcaseprefixCIT - notesCreated by CI pipeline - ' - expectPatterns: - - "Success|id" - - - name: Create test suite - command: | - # Extract project ID from previous response — get the first integer id value - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.createTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testprojectid1 - testsuitenameCI Suite - detailsSuite created by CI - ' - expectPatterns: - - "id|name_already_exists" - - - name: Create test case - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.createTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testcasenameCI Smoke Test - testsuiteid2 - testprojectid1 - authorloginadmin - summaryAutomated CI smoke test - steps - step_number1 - actionsVerify system is operational - expected_resultsSystem responds correctly - - ' - expectPatterns: - - "id|Success" - - - name: Verify test case was created - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.getTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testcaseexternalidCIT-1 - ' - expectPatterns: - - "CI Smoke Test" - - - name: Delete test case via API - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.deleteTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testcaseexternalidCIT-1 - ' - expectPatterns: - - "Success|status.*1|true" - - - name: Delete test suite via API - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.deleteTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 - testsuiteid2 - ' - expectPatterns: - - "Success|status.*1|true" - -criteria: | - Verify full CRUD workflow through XML-RPC API: - - Create a test project with prefix CIT - - Create a test suite under the project - - Create a test case in the suite - - Retrieve and verify the test case - - Delete the test case via tl.deleteTestCase - - Delete the test suite via tl.deleteTestSuite - Session teardown (docker compose down -v) is handled by - cicd/scripts/run-tests.sh after the run completes. diff --git a/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml b/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml deleted file mode 100644 index 56e2aa9365..0000000000 --- a/cicd/tests/testcases/integration/TC-INTEGRATION-001.yml +++ /dev/null @@ -1,18 +0,0 @@ -id: TC-INTEGRATION-001 -name: TestLink Login Page Responds -suite: integration -priority: 1 -timeout: 30000 -dependencies: [] - -steps: - - name: Verify app is responding - command: | - curl -sf http://localhost:8090/login.php | grep -o "Login" | head -1 - expectPatterns: - - "Login" - -criteria: | - Verify the login page is served on port 8090. - Session setup (docker compose up, DB init) is handled by - cicd/scripts/run-tests.sh before any test runs. diff --git a/cicd/tests/testcases/integration/TC-INTEGRATION-002.yml b/cicd/tests/testcases/integration/TC-INTEGRATION-002.yml deleted file mode 100644 index e777407d22..0000000000 --- a/cicd/tests/testcases/integration/TC-INTEGRATION-002.yml +++ /dev/null @@ -1,21 +0,0 @@ -id: TC-INTEGRATION-002 -name: XML-RPC API Ping -suite: integration -priority: 2 -timeout: 30000 -dependencies: - - TC-INTEGRATION-001 - -steps: - - name: Call tl.ping via XML-RPC - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.ping' - expectPatterns: - - "Hello!" - -criteria: | - Verify the XML-RPC API is accessible: - - tl.ping endpoint responds - - Response contains "Hello!" greeting diff --git a/cicd/tests/testcases/integration/TC-INTEGRATION-003.yml b/cicd/tests/testcases/integration/TC-INTEGRATION-003.yml deleted file mode 100644 index 4a7b222040..0000000000 --- a/cicd/tests/testcases/integration/TC-INTEGRATION-003.yml +++ /dev/null @@ -1,29 +0,0 @@ -id: TC-INTEGRATION-003 -name: XML-RPC API Authentication -suite: integration -priority: 3 -timeout: 30000 -dependencies: - - TC-INTEGRATION-002 - -steps: - - name: Call tl.checkDevKey with valid API key - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.checkDevKeydevKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4' - expectPatterns: - - "1|true" - - - name: Call tl.checkDevKey with invalid API key - command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ - -H "Content-Type: text/xml" \ - -d 'tl.checkDevKeydevKeyinvalid_key_12345' - expectPatterns: - - "faultCode|invalid|5000" - -criteria: | - Verify API authentication works: - - Valid API key returns true/success - - Invalid API key returns error/fault diff --git a/cicd/tests/testcases/smoke/TC-SMOKE-001.yml b/cicd/tests/testcases/smoke/TC-SMOKE-001.yml new file mode 100644 index 0000000000..fe5d2d6b2e --- /dev/null +++ b/cicd/tests/testcases/smoke/TC-SMOKE-001.yml @@ -0,0 +1,17 @@ +id: TC-SMOKE-001 +name: Login page responds +suite: smoke +priority: 1 +timeout: 30000 +dependencies: [] + +steps: + - name: GET /login.php + command: curl -sf http://localhost:8090/login.php | grep -o "Login" | head -1 + expectPatterns: + - "Login" + +criteria: | + Baseline check: the HTTP server is up and serving the login page. + Session setup (docker compose up + DB init) is handled by + cicd/scripts/run-tests.sh before any test runs. diff --git a/cicd/tests/testcases/smoke/TC-SMOKE-002.yml b/cicd/tests/testcases/smoke/TC-SMOKE-002.yml new file mode 100644 index 0000000000..a30ac786eb --- /dev/null +++ b/cicd/tests/testcases/smoke/TC-SMOKE-002.yml @@ -0,0 +1,19 @@ +id: TC-SMOKE-002 +name: XML-RPC tl.ping +suite: smoke +priority: 2 +timeout: 30000 +dependencies: + - TC-SMOKE-001 + +steps: + - name: Call tl.ping + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.ping + XMLDOC + expectPatterns: + - "Hello!" + +criteria: | + The XML-RPC endpoint is reachable and returns the canonical Hello! greeting. diff --git a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml new file mode 100644 index 0000000000..264e544356 --- /dev/null +++ b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml @@ -0,0 +1,111 @@ +id: TC-WORKFLOW-001 +name: Full entity-graph workflow +suite: workflow +priority: 1 +timeout: 120000 +dependencies: + - TC-CRUD-003 + +steps: + - name: Create project + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectnameproj-{{testId}}-{{runId}} + testcaseprefixW{{runId}} + notes{{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + projectId: "id" + + - name: Create test suite + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testprojectid{{projectId}} + testsuitenamesuite-{{testId}}-{{runId}} + details{{testId}} + + XMLDOC + expectPatterns: + - "" + capture: + suiteId: "id" + + - name: Create test case + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.createTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcasenamecase-{{testId}}-{{runId}} + testsuiteid{{suiteId}} + testprojectid{{projectId}} + authorloginadmin + summary{{testId}} + steps + step_number1 + actionsVerify system is operational + expected_resultsSystem responds correctly + + + XMLDOC + expectPatterns: + - "" + capture: + caseId: "id" + + - name: Read case back and verify name + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.getTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcaseid{{caseId}} + + XMLDOC + expectPatterns: + - "case-{{testId}}-{{runId}}" + + - name: Delete case (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestCase + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testcaseid{{caseId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + + - name: Delete suite (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestSuite + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + testsuiteid{{suiteId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + + - name: Delete project (teardown) + command: | + bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' + tl.deleteTestProject + devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + prefixW{{runId}} + + XMLDOC + expectPatterns: + - "1|status.*true" + +criteria: | + End-to-end workflow exercising the full entity graph: + project → testsuite → testcase → read → delete (reverse order). + Each entity's ID is captured from its creation response and used + in the next step. Delete uses the captured IDs, not external IDs, + so the test works regardless of how many entities exist in the DB. From dc8b88f5f5ed2683723e04cfbc7064b4ee05a122 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 10:52:01 -0400 Subject: [PATCH 06/13] fix(ci): handle ids and substitute vars in expectPatterns Two bugs surfaced when running the CRUD and workflow suites against a live TestLink: 1. xmlrpc-capture.sh pulled the "first in the response" as the created id. But TestLink's XML-RPC emits the `id` field as N for several entity types (createTestProject among them), so the capture emitted `{"ok": true}` with no id and every downstream step referenced an empty {{projectId}}. Match the `id` member explicitly and accept both and wrappers; do the same for faultCode. 2. The executor substituted {{runId}} / {{testId}} / captured vars into step.command but not into step.expectPatterns. "Read back" steps used expectPatterns like "case-{{testId}}-{{runId}}" to verify the round-trip, which never matched because the regex was the literal template string. Run the same substitution over expect/reject patterns before checking. Test expect-patterns for creation steps updated from "" (never going to match ) to "id", which is stable across both id representations. Verified against a fresh CI stack: - smoke: 2/2 pass - auth: 2/2 pass - crud: 3/3 pass - workflow: 1/1 pass - workflow re-run in same session: 1/1 pass (idempotent) Refs #5 Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/scripts/xmlrpc-capture.sh | 16 ++++++++++++---- cicd/tests/src/executor.ts | 10 ++++++++-- cicd/tests/testcases/crud/TC-CRUD-001.yml | 2 +- cicd/tests/testcases/crud/TC-CRUD-002.yml | 4 ++-- cicd/tests/testcases/crud/TC-CRUD-003.yml | 6 +++--- .../tests/testcases/workflow/TC-WORKFLOW-001.yml | 6 +++--- 6 files changed, 29 insertions(+), 15 deletions(-) diff --git a/cicd/scripts/xmlrpc-capture.sh b/cicd/scripts/xmlrpc-capture.sh index 2c08337de9..52734103f4 100755 --- a/cicd/scripts/xmlrpc-capture.sh +++ b/cicd/scripts/xmlrpc-capture.sh @@ -34,12 +34,20 @@ RESPONSE=$(curl -sS -X POST "$URL" \ # Mirror the response to stderr for expectPatterns / debugging. printf '%s\n' "$RESPONSE" >&2 -# First in the body — typically the created entity id. -ID=$(printf '%s' "$RESPONSE" | grep -oP '(?<=)\d+' | head -1 || true) +# The `id` member of the response — typically the created entity id. +# TestLink's XML-RPC is inconsistent about typing and whitespace: the +# same field is sometimes emitted as and sometimes as , +# and the wrapper may or may not have surrounding whitespace. +# Collapse the response to one line and match the `id` member directly +# to cope with both shapes. +FLAT=$(printf '%s' "$RESPONSE" | tr -d '\n' | tr -s ' ') +ID=$(printf '%s' "$FLAT" \ + | grep -oP 'id\s*\s*<(?:int|string)>\K[0-9]+' \ + | head -1 || true) # Fault code (if the server returned a ). -FAULT=$(printf '%s' "$RESPONSE" \ - | grep -oP '(?<=faultCode)\d+' \ +FAULT=$(printf '%s' "$FLAT" \ + | grep -oP 'faultCode\s*\s*<(?:int|string)>\K[0-9]+' \ | head -1 || true) if [ -n "${FAULT:-}" ]; then diff --git a/cicd/tests/src/executor.ts b/cicd/tests/src/executor.ts index 45bb973b53..b9a27858bc 100644 --- a/cicd/tests/src/executor.ts +++ b/cicd/tests/src/executor.ts @@ -233,10 +233,16 @@ export class TestExecutor { const resolvedStep = { ...step, command: resolvedCommand }; const result = await this.executeStep(resolvedStep, testCase.timeout); + const resolvedExpect = step.expectPatterns?.map((p) => + this.substituteVariables(p) + ); + const resolvedReject = step.rejectPatterns?.map((p) => + this.substituteVariables(p) + ); result.patternMatches = this.checkPatterns( result, - step.expectPatterns, - step.rejectPatterns + resolvedExpect, + resolvedReject ); this.captureVariables(step, result); diff --git a/cicd/tests/testcases/crud/TC-CRUD-001.yml b/cicd/tests/testcases/crud/TC-CRUD-001.yml index f3b7e4854f..380e0d7652 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-001.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-001.yml @@ -18,7 +18,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: projectId: "id" diff --git a/cicd/tests/testcases/crud/TC-CRUD-002.yml b/cicd/tests/testcases/crud/TC-CRUD-002.yml index ffd345cb98..4cd918bf35 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-002.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-002.yml @@ -18,7 +18,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: projectId: "id" @@ -33,7 +33,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: suiteId: "id" diff --git a/cicd/tests/testcases/crud/TC-CRUD-003.yml b/cicd/tests/testcases/crud/TC-CRUD-003.yml index a3c5161381..040533253e 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-003.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-003.yml @@ -18,7 +18,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: projectId: "id" @@ -33,7 +33,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: suiteId: "id" @@ -55,7 +55,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: caseId: "id" diff --git a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml index 264e544356..e6e365eb8e 100644 --- a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml +++ b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml @@ -18,7 +18,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: projectId: "id" @@ -33,7 +33,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: suiteId: "id" @@ -55,7 +55,7 @@ steps: XMLDOC expectPatterns: - - "" + - "id" capture: caseId: "id" From fbc579f3a7fd09ef5f44ce2dd5973f66ab763746 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:03:33 -0400 Subject: [PATCH 07/13] feat(ci): make CI host port configurable via TL_PORT/TL_URL The CI compose file used to publish the app on host port 8090, the same port as the dev compose, so the two stacks could not coexist. Move the port behind ${TL_PORT:-8091} and have all CI scripts and test helpers reference $TL_URL. run-tests.sh sources cicd/tests/.env (gitignored) so a single override changes every consumer. Why: a developer who left the dev stack running would see CI fail to bind on every test invocation. Splitting the port lets both run side by side. Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/TESTING_GUIDELINES.md | 8 ++++---- cicd/docker-compose.ci.yml | 2 +- cicd/scripts/ci-up.sh | 15 +++++++++++++-- cicd/scripts/run-tests.sh | 10 ++++++++++ cicd/scripts/xmlrpc-capture.sh | 6 ++++-- 5 files changed, 32 insertions(+), 9 deletions(-) diff --git a/cicd/TESTING_GUIDELINES.md b/cicd/TESTING_GUIDELINES.md index 4f7f164746..df93717f24 100644 --- a/cicd/TESTING_GUIDELINES.md +++ b/cicd/TESTING_GUIDELINES.md @@ -117,7 +117,7 @@ IDs are never hardcoded. They flow from creation responses into subsequent steps ```yaml - name: Create test project command: | - curl -sf -X POST http://localhost:8090/lib/api/xmlrpc/v1/xmlrpc.php \ + curl -sf -X POST "$TL_URL/lib/api/xmlrpc/v1/xmlrpc.php" \ -H "Content-Type: text/xml" \ -d '' capture: @@ -186,11 +186,11 @@ The repo has two compose files on purpose. They serve different audiences and ha | Seed data | Optional: `restore` service loads `docs/db_sample` sample data | `init-db.sh` loads schema + default data + hardcoded admin API key | | Image build | `build: .` from repo root Dockerfile | `build: ..` from the same Dockerfile | | Image pinning | `postgres:9.6` pinned; `maildev:latest` floats | Fully pinned (`postgres:9.6`) | -| Host port | `0.0.0.0:8090:80` | `8090:80` | +| Host port | `0.0.0.0:8090:80` | `${TL_PORT:-8091}:80` (default 8091) | -### Important: port 8090 collides +### Host port: 8090 (dev) vs 8091 (CI) -Both compose files publish the app on host port **8090**. They cannot run at the same time. If the dev stack is up, CI setup will fail to bind. This is a known constraint — tests and the dev environment are serialized on a single developer machine. Long-term, the CI compose should either take a different port or use ephemeral host port assignment to remove this footgun. +The two stacks bind different host ports so they can run simultaneously on a developer machine. CI uses `8091` by default, sourced from `TL_PORT` in `cicd/tests/.env`. All CI scripts and YAML test steps reference `$TL_URL` (default `http://localhost:8091`) rather than hardcoding the port; change the port in one place and everything follows. ### Why the separation is mandatory diff --git a/cicd/docker-compose.ci.yml b/cicd/docker-compose.ci.yml index 868ec583fa..ee9c79631c 100644 --- a/cicd/docker-compose.ci.yml +++ b/cicd/docker-compose.ci.yml @@ -32,7 +32,7 @@ services: db: condition: service_healthy ports: - - "8090:80" + - "${TL_PORT:-8091}:80" volumes: - ../logs:/var/testlink/logs:Z - ../upload_area:/var/testlink/upload_area:Z diff --git a/cicd/scripts/ci-up.sh b/cicd/scripts/ci-up.sh index 6fb7c0cf79..48460e7042 100755 --- a/cicd/scripts/ci-up.sh +++ b/cicd/scripts/ci-up.sh @@ -5,6 +5,17 @@ set -e SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" COMPOSE_FILE="$SCRIPT_DIR/../docker-compose.ci.yml" +# Allow direct invocation (outside run-tests.sh) to still pick up .env values. +ENV_FILE="$SCRIPT_DIR/../tests/.env" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck disable=SC1090 + . "$ENV_FILE" + set +a +fi + +TL_URL="${TL_URL:-http://localhost:${TL_PORT:-8091}}" + echo "=== Starting TestLink CI Environment ===" echo "Building Docker image..." @@ -15,7 +26,7 @@ docker compose -f "$COMPOSE_FILE" up -d echo "Waiting for app to be healthy..." for i in $(seq 1 60); do - if curl -sf http://localhost:8090/login.php > /dev/null 2>&1; then + if curl -sf "$TL_URL/login.php" > /dev/null 2>&1; then echo "App is responding." break fi @@ -31,6 +42,6 @@ echo "Initializing database..." docker compose -f "$COMPOSE_FILE" exec -T app bash /var/www/html/cicd/scripts/init-db.sh echo "=== CI Environment Ready ===" -echo "TestLink URL: http://localhost:8090" +echo "TestLink URL: $TL_URL" echo "Admin user: admin / admin" echo "API key: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" diff --git a/cicd/scripts/run-tests.sh b/cicd/scripts/run-tests.sh index afb89fe0fc..7f06d7dbf9 100755 --- a/cicd/scripts/run-tests.sh +++ b/cicd/scripts/run-tests.sh @@ -20,6 +20,16 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +# Load cicd/tests/.env if present so LLM_JUDGE_URL / LLM_JUDGE_MODEL etc. +# reach the tsx CLI. Keep it optional — absent .env means rely on defaults. +ENV_FILE="$REPO_ROOT/cicd/tests/.env" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck disable=SC1090 + . "$ENV_FILE" + set +a +fi + ARGS=("$@") # The build suite lints PHP and builds the image via `docker build`. diff --git a/cicd/scripts/xmlrpc-capture.sh b/cicd/scripts/xmlrpc-capture.sh index 52734103f4..8c39a72824 100755 --- a/cicd/scripts/xmlrpc-capture.sh +++ b/cicd/scripts/xmlrpc-capture.sh @@ -20,11 +20,13 @@ # projectId: "id" # # Env: -# TL_URL — base URL of the TestLink instance (default http://localhost:8090) +# TL_URL — base URL of the TestLink instance (default http://localhost:8091, +# matching the CI compose. run-tests.sh sources cicd/tests/.env which +# sets this; override per-call by exporting TL_URL beforehand.) set -eo pipefail -URL="${TL_URL:-http://localhost:8090}/lib/api/xmlrpc/v1/xmlrpc.php" +URL="${TL_URL:-http://localhost:8091}/lib/api/xmlrpc/v1/xmlrpc.php" REQUEST=$(cat) RESPONSE=$(curl -sS -X POST "$URL" \ From 8a6afdb3db867b35dfef6911da6ad317c97e7182 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:03:50 -0400 Subject: [PATCH 08/13] =?UTF-8?q?refactor(ci):=20redesign=20LLM=20judge=20?= =?UTF-8?q?=E2=80=94=20chat=20API,=20per-test=20framing,=20ctx=20tuning?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes to the judge: 1. Switch from Ollama /api/generate to /api/chat with a system + user message split. Make LLM_JUDGE_URL and LLM_JUDGE_MODEL env-driven via cicd/tests/.env so the host running tests can target any Ollama instance and model without touching code. 2. Redesign the prompt along role/task/behavior/output lines. Drop the hardcoded heuristics ("exit code 0 = pass", etc.) that were fighting negative tests. Add two optional YAML fields the test author owns: - objective — what the test proves and why it matters - judgeContext — what evidence each step produces and what silent failures look like in this domain The judge reads OBJECTIVE → CONTEXT → CRITERIA → OBSERVATIONS in that order, so the per-test situational framing reaches the model before the raw evidence. 3. Tune Ollama options for small (~4B) models: - num_ctx 8192 — prompts can hit ~5-7k chars; 4096 default was silently truncating OBSERVATIONS for multi-step tests, producing empty/garbage JSON - num_predict 512 — enough headroom for FAIL evidence quotes while still catching runaway generation Why: gemma3:4b was returning empty {} or off-schema JSON on roughly 20-30% of tests with the old prompt and default options. The combo of per-test framing and proper context size brings 11/11 testcases through the judge cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/tests/src/config.ts | 8 +- cicd/tests/src/judge/llm-judge.ts | 157 ++++++++++++++++++++---------- cicd/tests/src/loader.ts | 2 + cicd/tests/src/types.ts | 11 ++- 4 files changed, 120 insertions(+), 58 deletions(-) diff --git a/cicd/tests/src/config.ts b/cicd/tests/src/config.ts index 5c7c0955f8..aa75f403ee 100644 --- a/cicd/tests/src/config.ts +++ b/cicd/tests/src/config.ts @@ -32,10 +32,12 @@ export const CONFIG = { defaultTimeout: 120000, defaultStepTimeout: 60000, - // LLM Judge defaults + // LLM Judge defaults — overridable via LLM_JUDGE_URL / LLM_JUDGE_MODEL + // in cicd/tests/.env (loaded by run-tests.sh) or the --judge-url / + // --judge-model CLI flags. llm: { - defaultUrl: 'http://localhost:11434', - defaultModel: 'llama3:8b', + defaultUrl: process.env.LLM_JUDGE_URL || 'http://localhost:11434', + defaultModel: process.env.LLM_JUDGE_MODEL || 'llama3:8b', timeout: 300000, stdoutLimit: 1000, stderrLimit: 500, diff --git a/cicd/tests/src/judge/llm-judge.ts b/cicd/tests/src/judge/llm-judge.ts index 2bf84d8a93..0ea79021a2 100644 --- a/cicd/tests/src/judge/llm-judge.ts +++ b/cicd/tests/src/judge/llm-judge.ts @@ -44,83 +44,131 @@ export class LLMJudge { } /** - * Build structured JSON prompt for LLM evaluation of a single test. + * Build the system + user messages for LLM evaluation of a single test. + * + * Per-test framing comes from the YAML, not from this file: + * - objective — what the test proves and why it matters + * - judgeContext — what evidence the steps produce and what silent + * failures look like in this domain + * - criteria — pass/fail conditions + * The system message defines the role, the mission ("hunt silent failures"), + * and the output schema. It does not encode mechanical rules like "exit + * code 0 = pass" — those are the test author's call, expressed in the + * judgeContext / criteria. */ - private buildPrompt(result: TestResult): string { + private buildPrompt(result: TestResult): { system: string; user: string } { const r = result; - const steps = r.steps.map((step, j) => { + const system = [ + `# ROLE`, + `You are a senior QA engineer reviewing one automated test run for ${CONFIG.projectName}.`, + ``, + `# MISSION`, + `Decide whether the test PASSED or FAILED. The author of the test has provided:`, + `- OBJECTIVE: what the test is trying to prove and why it matters.`, + `- CONTEXT: what each step is supposed to produce, and what silent failures look like in this domain.`, + `- CRITERIA: the pass/fail conditions for this specific test.`, + `- OBSERVATIONS: what actually happened (commands, exit codes, stdout, stderr, container logs).`, + `Read the OBJECTIVE and CONTEXT first to understand what you are looking at.`, + `Then read the CRITERIA to know what counts as success.`, + `Then read the OBSERVATIONS and decide whether the criteria were met.`, + `Apply judgment, not pattern matching: a step that exits 0 can still be a silent failure if the output shows it skipped its real work; a step that exits non-zero can be a pass if the criteria expected an error.`, + ``, + `# OUTPUT`, + `Respond with exactly one JSON object — no prose, no code fences, no thinking tokens. Schema:`, + `{"testId": string, "pass": boolean, "reason": string, "evidence": string}`, + `- testId : echo the test id from the OBJECTIVE block verbatim.`, + `- pass : true if criteria met, false otherwise.`, + `- reason : one short sentence (<= 25 words) grounded in the observations.`, + `- evidence : the exact line(s) from OBSERVATIONS that drove the verdict; required when pass is false, otherwise "".`, + ``, + `Example pass : {"testId":"TC-X-001","pass":true,"reason":"Login page returned the expected 'Login' marker.","evidence":""}`, + `Example fail : {"testId":"TC-X-002","pass":false,"reason":"Build appeared to succeed but skipped composer install.","evidence":"CACHED [stage 3/5]"}`, + ].join('\n'); + + const stepsBlock = r.steps.map((step, j) => { const stepDef = r.testCase.steps[j]; - return { - name: step.name, - command: step.command.trim(), - exit_code: step.exitCode, - duration_ms: step.duration, - timeout_ms: stepDef?.timeout || r.testCase.timeout, - stdout: this.truncate(step.stdout, CONFIG.llm.stdoutLimit), - stderr: this.truncate(step.stderr, CONFIG.llm.stderrLimit), - }; - }); - - const promptData = { - role: `You are a test result evaluator for ${CONFIG.projectName}. Analyze the test execution data and determine if the test passed or failed.`, - rules: [ - 'Check step stdout for error responses (e.g. {"error":"..."} means FAIL)', - 'Errors with exit code 0 are still FAIL', - 'For AI-generated text, accept reasonable variations', - 'Long durations within timeout are acceptable', - 'Focus on semantic correctness, not formatting differences', - ], - test: { - id: r.testCase.id, - name: r.testCase.name, - suite: r.testCase.suite, - goal: r.testCase.goal || r.testCase.name, - criteria: r.testCase.criteria, - timeout_ms: r.testCase.timeout, - duration_ms: r.totalDuration, - }, - steps, - container_logs: this.truncate(r.logs, CONFIG.llm.logsLimit), - respond: { - format: 'Respond with a single JSON object', - fields: { - testId: r.testCase.id, - pass: 'true if test meets all criteria, false otherwise', - reason: 'Brief explanation of your verdict', - evidence: 'Required if pass is false — the exact stdout content or log line that caused failure', - }, - }, - }; + const limit = stepDef?.timeout || r.testCase.timeout; + const stdout = this.truncate(step.stdout, CONFIG.llm.stdoutLimit).trim() || '(empty)'; + const stderr = this.truncate(step.stderr, CONFIG.llm.stderrLimit).trim() || '(empty)'; + return [ + `Step ${j + 1}: ${step.name}`, + ` command : ${step.command.trim().split('\n').join(' \\n ')}`, + ` exit_code : ${step.exitCode}`, + ` duration : ${step.duration}ms (timeout ${limit}ms)`, + ` stdout : ${stdout}`, + ` stderr : ${stderr}`, + ].join('\n'); + }).join('\n\n'); + + const logs = this.truncate(r.logs, CONFIG.llm.logsLimit).trim() || '(none)'; + const objective = (r.testCase.objective || r.testCase.goal || r.testCase.name).trim(); + const judgeContext = (r.testCase.judgeContext || '').trim() || + `(none provided — read the criteria carefully and watch for silent failures: steps that exited 0 but did not actually achieve the test's objective.)`; + const criteria = (r.testCase.criteria || '(none provided)').trim(); - const prompt = JSON.stringify(promptData, null, 2); + const user = [ + `# OBJECTIVE`, + `Test id : ${r.testCase.id}`, + `Suite : ${r.testCase.suite}`, + `Name : ${r.testCase.name}`, + ``, + objective, + ``, + `# CONTEXT`, + judgeContext, + ``, + `# CRITERIA`, + criteria, + ``, + `# OBSERVATIONS`, + `Total duration: ${r.totalDuration}ms (test timeout ${r.testCase.timeout}ms)`, + ``, + stepsBlock, + ``, + `## Container logs`, + logs, + ``, + `# VERDICT`, + `Return the JSON object specified in the system message. Use testId="${r.testCase.id}".`, + ].join('\n'); - // Log prompt stats const totalStdout = r.steps.reduce((sum, s) => sum + s.stdout.length, 0); const totalStderr = r.steps.reduce((sum, s) => sum + s.stderr.length, 0); process.stderr.write(` [LLM] Prompt for ${r.testCase.id}: logs ${r.logs.length} chars, stdout ${totalStdout} chars, stderr ${totalStderr} chars\n`); - process.stderr.write(` [LLM] Prompt size: ${prompt.length} chars\n`); + process.stderr.write(` [LLM] Prompt size: system ${system.length} + user ${user.length} = ${system.length + user.length} chars\n`); - return prompt; + return { system, user }; } /** * Judge a single test result. */ private async judgeOne(result: TestResult): Promise { - const prompt = this.buildPrompt(result); + const { system, user } = this.buildPrompt(result); const testId = result.testCase.id; const response = await axios.post( - `${this.ollamaUrl}/api/generate`, + `${this.ollamaUrl}/api/chat`, { model: this.model, - prompt, + messages: [ + { role: 'system', content: system }, + { role: 'user', content: user }, + ], stream: false, format: 'json', options: { temperature: 0.1, - num_predict: 1024, + // Default context is 4096 tokens — our multi-step prompts can + // hit ~5000-7000 and Ollama silently truncates the END of the + // prompt (which is where OBSERVATIONS live). 8192 fits every + // current test with headroom; raise if logsLimit/stdoutLimit grow. + num_ctx: 8192, + // Output cap: a typical verdict is 50-200 tokens, but FAIL + // verdicts may quote 5-10 lines of evidence. 512 covers honest + // verbosity with headroom; well below "the model went rogue". + num_predict: 512, }, }, { @@ -128,7 +176,7 @@ export class LLMJudge { } ); - const responseText = response.data.response; + const responseText = response.data.message?.content ?? ''; const promptTokens = response.data.prompt_eval_count ?? '?'; const responseTokens = response.data.eval_count ?? '?'; process.stderr.write(` [LLM] Tokens for ${testId}: prompt=${promptTokens}, response=${responseTokens}\n`); @@ -218,9 +266,10 @@ export class LLMJudge { try { process.stderr.write(` [LLM] Unloading judge model ${this.model}...\n`); await axios.post( - `${this.ollamaUrl}/api/generate`, + `${this.ollamaUrl}/api/chat`, { model: this.model, + messages: [], keep_alive: 0, }, { diff --git a/cicd/tests/src/loader.ts b/cicd/tests/src/loader.ts index 6beadc967d..ed536f8f2c 100644 --- a/cicd/tests/src/loader.ts +++ b/cicd/tests/src/loader.ts @@ -215,6 +215,8 @@ export class TestLoader { dependencies: Array.isArray(raw.dependencies) ? raw.dependencies : [], steps, goal: typeof raw.goal === 'string' ? raw.goal : undefined, + objective: typeof raw.objective === 'string' ? raw.objective : undefined, + judgeContext: typeof raw.judgeContext === 'string' ? raw.judgeContext : undefined, criteria: typeof raw.criteria === 'string' ? raw.criteria : '', }; } diff --git a/cicd/tests/src/types.ts b/cicd/tests/src/types.ts index a808cfc840..4de55cd59a 100644 --- a/cicd/tests/src/types.ts +++ b/cicd/tests/src/types.ts @@ -44,8 +44,17 @@ export interface TestCase { steps: TestStep[]; /** Human-readable criteria for LLM judge evaluation */ criteria: string; - /** Short goal statement for LLM judge context (optional) */ + /** Short goal statement for LLM judge context (optional, legacy — prefer objective) */ goal?: string; + /** One-paragraph plain-English objective: what this test proves and why it matters */ + objective?: string; + /** + * Situational framing for the LLM judge: what kind of evidence each step + * produces, what success looks like in the output, and what silent failures + * to watch for. Lets the judge interpret logs in context instead of + * mechanically checking exit codes. + */ + judgeContext?: string; } // ============================================ From 655fd20b573eaacc5d40e62c42ae298aff85677d Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:04:01 -0400 Subject: [PATCH 09/13] docs(ci): backfill objective + judgeContext for all testcases; fix TC-BUILD-003 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each YAML now declares: - objective — the plain-English purpose of the test - judgeContext — what each step does, what evidence it emits, and what silent failures look like in that test's domain This is the per-test framing the redesigned LLM judge consumes. Negative tests (TC-AUTH-002) explicitly call out that they EXPECT a fault response, which fixes the prior misjudgment where the judge labeled the expected fault as a failure. TC-BUILD-003 is also rewritten to validate composer.json from inside the testlink-ci-test image (depending on TC-BUILD-002 to build it) instead of shelling out to a host python3 that may not exist. TC-SMOKE-001 picks up the $TL_URL change from the prior port-config commit. Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/tests/testcases/auth/TC-AUTH-001.yml | 27 +++++++++- cicd/tests/testcases/auth/TC-AUTH-002.yml | 40 +++++++++++++- cicd/tests/testcases/build/TC-BUILD-001.yml | 39 ++++++++++++-- cicd/tests/testcases/build/TC-BUILD-002.yml | 44 ++++++++++++++-- cicd/tests/testcases/build/TC-BUILD-003.yml | 52 ++++++++++++++++--- cicd/tests/testcases/crud/TC-CRUD-001.yml | 45 ++++++++++++++-- cicd/tests/testcases/crud/TC-CRUD-002.yml | 46 +++++++++++++--- cicd/tests/testcases/crud/TC-CRUD-003.yml | 47 ++++++++++++++--- cicd/tests/testcases/smoke/TC-SMOKE-001.yml | 25 ++++++++- cicd/tests/testcases/smoke/TC-SMOKE-002.yml | 26 +++++++++- .../testcases/workflow/TC-WORKFLOW-001.yml | 42 +++++++++++++-- 11 files changed, 393 insertions(+), 40 deletions(-) diff --git a/cicd/tests/testcases/auth/TC-AUTH-001.yml b/cicd/tests/testcases/auth/TC-AUTH-001.yml index 34af039354..d85dedcdf9 100644 --- a/cicd/tests/testcases/auth/TC-AUTH-001.yml +++ b/cicd/tests/testcases/auth/TC-AUTH-001.yml @@ -17,5 +17,30 @@ steps: expectPatterns: - "1|true" +objective: | + Confirm the admin API key seeded by init-db.sh is accepted by + tl.checkDevKey, proving the auth path works for the canonical admin + identity. Every CRUD and workflow test depends on this key being usable. + +judgeContext: | + Step 1 sends `tl.checkDevKey` with the seeded admin key + (a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4) via cicd/scripts/xmlrpc-capture.sh. + The helper mirrors the raw XML response to stderr; on success the + stdout JSON envelope is `{"ok": true}`. + + A healthy response is an XML-RPC methodResponse containing + `1` — the canonical XML-RPC representation of true. + + Silent failures to watch for: + - Server returns `0` — admin key is being rejected + (likely a DB seed problem); this would fail the expectPattern but + looks like a "valid response" superficially. + - Server returns a fault response with text that happens to include + "true" somewhere. + - Auth bypassed entirely — every key returns true (would also pass + here but should fail TC-AUTH-002, which is the negative pair). + criteria: | - The admin API key seeded by init-db.sh is accepted by tl.checkDevKey. + PASS when the XML response contains 1 (or the literal + word "true" in the response). + FAIL on 0, a fault envelope, or no response. diff --git a/cicd/tests/testcases/auth/TC-AUTH-002.yml b/cicd/tests/testcases/auth/TC-AUTH-002.yml index 3b31e1dab1..275e75536d 100644 --- a/cicd/tests/testcases/auth/TC-AUTH-002.yml +++ b/cicd/tests/testcases/auth/TC-AUTH-002.yml @@ -17,6 +17,42 @@ steps: expectPatterns: - "faultCode|invalid|2000" +objective: | + Confirm the auth layer explicitly rejects unknown API keys with a fault + response. The point is to detect a security regression where a bad key + silently passes — that would defeat every other auth-protected test. + +judgeContext: | + THIS IS A NEGATIVE TEST — the EXPECTED outcome is a fault response, not + a success envelope. An error here is the desired result, not a failure. + + Step 1 sends `tl.checkDevKey` with a deliberately bogus key + (`not-a-real-key-`). The server should return an XML-RPC + envelope with faultCode 2000 and a message containing "invalid". + + The xmlrpc-capture.sh helper produces: + - stdout: `{"ok": false, "faultCode": 2000}` on the expected fault path + - stderr: the raw ...faultCode 2000... + envelope, which is what expectPatterns matches against + + A pass for this test means the rejection happened with a real fault + envelope. Compare the OBSERVATIONS to the criteria below — do not let + the words "ok": true in the stdout JSON mislead you; that JSON wraps + the HTTP request, not the auth verdict. + + Silent failures to watch for: + - Server returns `1` for a bogus key — this is the + nightmare case (silent acceptance of bad credentials). FAIL hard. + - Server returns `0` (rejection but no fault) — this + is a partial regression; the rejection is too quiet. FAIL. + - HTTP 200 with empty body or HTML error page — auth layer crashed, + can't tell what it would have decided. FAIL. + criteria: | - An unknown API key is rejected with a fault response — the server - does not silently return a success envelope for bad credentials. + PASS when the response (stderr) contains a faultCode (typically 2000) + with a message indicating the key is invalid. The fault envelope is + what we want to see. + + FAIL when the bogus key is silently accepted (1), + silently rejected without a fault (0), or the + response is missing/malformed. diff --git a/cicd/tests/testcases/build/TC-BUILD-001.yml b/cicd/tests/testcases/build/TC-BUILD-001.yml index 989e49c269..d9561bf348 100644 --- a/cicd/tests/testcases/build/TC-BUILD-001.yml +++ b/cicd/tests/testcases/build/TC-BUILD-001.yml @@ -1,7 +1,6 @@ id: TC-BUILD-001 name: PHP Syntax Check suite: build -goal: Verify PHP files have no syntax errors using the Docker image priority: 1 timeout: 180000 dependencies: @@ -28,7 +27,39 @@ steps: - "Parse error" - "Fatal error" +objective: | + Confirm every PHP source file in lib/ and cfg/ passes `php -l` syntax + check against the PHP version baked into the Docker image. Catches + syntax breakage before it reaches runtime. + +judgeContext: | + Both steps run `docker run --rm testlink-ci-test find ... -exec php -l` + against directories baked into the image (`/var/www/html/lib` and + `/var/www/html/cfg`). The find/-exec pattern lints every .php file in + one process tree. + + We pipe through `tail -5` so only the last few lines reach stdout — + per-file "No syntax errors detected" lines are expected on every file, + so the tail catches the final files plus any error that ended the run. + + A clean output is several "No syntax errors detected" lines. + A failure prints "Parse error: ..." or "PHP Fatal error: ..." with + file path and line number. + + Silent failures to watch for: + - find found no .php files (e.g. directory typo or empty mount). Stdout + would be empty — no "No syntax errors detected" but also no error. + expectPatterns catches this as a fail (missing the expected string), + but worth noting in your verdict reason if it's the cause. + - tail truncated the actual error and only kept clean tail lines from + later files. Less likely with -exec because php -l prints + deterministic per-file output. + - php binary missing in the image — would surface as "/bin/sh: php: + not found" in stderr, very obvious. + criteria: | - Verify PHP source files have no syntax errors: - - php -l runs inside the Docker container on lib/ and cfg/ files - - No parse errors or fatal errors detected + PASS when both steps' stdout contains "No syntax errors detected" and + neither contains "Parse error" or "Fatal error". + + FAIL on any parse/fatal error, or on empty output (suggesting find + matched nothing). diff --git a/cicd/tests/testcases/build/TC-BUILD-002.yml b/cicd/tests/testcases/build/TC-BUILD-002.yml index d88f178d4b..9737d65423 100644 --- a/cicd/tests/testcases/build/TC-BUILD-002.yml +++ b/cicd/tests/testcases/build/TC-BUILD-002.yml @@ -17,7 +17,45 @@ steps: expectPatterns: - "sha256:" +objective: | + The Dockerfile at the repo root must build cleanly into a TestLink image + that downstream build-suite tests can run against. This is the foundation + test for the build suite — every other TC-BUILD-* depends on this image + existing as `testlink-ci-test`. + +judgeContext: | + Step 1 runs `docker build`. Its stdout is BuildKit progress output: + layer-by-layer "[N/M] STEP" lines, "CACHED" markers when a layer is + reused from cache, and a final "writing image sha256:..." or + "Successfully built " line. Stderr is mostly the same stream + duplicated by docker; an "ERROR:" line at the start of a line means a + build failure. + + Step 2 runs `docker image inspect`. Its stdout should be a single line + containing `sha256:<64 hex chars>` if the image was registered by step 1. + + Silent failures to watch for: + - Step 1 exits 0 but the output ends mid-stage (truncated build, often + an OOM or daemon disconnect — no explicit ERROR but no final "writing + image" either). + - All layers report CACHED including layers that should have been + rebuilt because the Dockerfile or COPY context changed (suggests the + test ran against a stale cache and didn't actually exercise current + sources). Some CACHED layers are normal; suspicious only if every + layer is cached and the test's purpose was to validate a Dockerfile + change. + - Step 2 returns a sha256 from a stale prior image (this test alone + can't detect that — flag only if the build output suggested failure + but the inspect still returned a SHA). + criteria: | - Verify the Docker image builds successfully: - - docker build completes without errors - - The resulting image is available locally + PASS when: + - The build progresses through stages and produces a final image + identifier (a "writing image sha256:..." or equivalent end-of-build + marker in step 1 output). + - Step 2 returns a sha256:... line confirming the image is registered. + + FAIL when: + - Step 1 emits an ERROR line or aborts mid-build. + - Step 2 cannot find the image. + - The build output suggests a silent failure per the context above. diff --git a/cicd/tests/testcases/build/TC-BUILD-003.yml b/cicd/tests/testcases/build/TC-BUILD-003.yml index ae9499e1f9..8dbbad9c71 100644 --- a/cicd/tests/testcases/build/TC-BUILD-003.yml +++ b/cicd/tests/testcases/build/TC-BUILD-003.yml @@ -3,22 +3,62 @@ name: Composer Dependencies suite: build priority: 3 timeout: 120000 -dependencies: [] +dependencies: + - TC-BUILD-002 steps: - name: Validate composer.json command: | - cd . && python3 -c "import json; json.load(open('composer.json')); print('valid')" + docker run --rm testlink-ci-test php -r ' + $j = json_decode(file_get_contents("/var/www/html/composer.json"), true); + if (!is_array($j)) { fwrite(STDERR, "invalid json\n"); exit(1); } + echo "valid\n"; + ' expectPatterns: - "valid" - name: Check vendor directory exists command: | - test -f ./vendor/autoload.php && echo "vendor present" || echo "vendor missing" + docker run --rm testlink-ci-test sh -c ' + test -f /var/www/html/vendor/autoload.php && echo "vendor present" || echo "vendor missing" + ' expectPatterns: - "vendor present|vendor missing" +objective: | + Confirm the composer.json baked into the Docker image is valid JSON, and + report whether vendor/ is present. Validates the dependency manifest + shipped in the image; vendor presence depends on the build pipeline. + +judgeContext: | + Step 1 runs `docker run --rm testlink-ci-test php -r ''`. + The PHP one-liner reads /var/www/html/composer.json from inside the + image, parses it, and prints "valid" on success or writes "invalid json" + to stderr and exits 1 on failure. + + Step 2 runs `docker run --rm testlink-ci-test sh -c 'test -f + /var/www/html/vendor/autoload.php && echo "vendor present" || echo + "vendor missing"'`. Either output is acceptable — the current + Dockerfile does not run `composer install`, so vendor is only present + if the host had a vendor/ directory when the image was built (and got + COPY'd in). The point of step 2 is to *report* the state, not to + enforce a specific one. + + Silent failures to watch for: + - composer.json is valid JSON but has missing required fields (e.g. no + "require" key). Beyond this test's scope — json_decode happily + parses any valid JSON. Flag in your verdict if you notice the + objective wants stricter validation (it doesn't, currently). + - "vendor missing" output is not itself a failure; only fail if BOTH + "vendor present" AND "vendor missing" are absent (suggests step 2 + crashed before the test/echo). + criteria: | - Verify composer configuration is valid: - - composer.json is valid JSON - - Vendor directory state is reported + PASS when: + - Step 1 stdout contains "valid". + - Step 2 stdout contains either "vendor present" or "vendor missing" + (both are acceptable per the objective). + + FAIL on: + - Step 1 missing "valid" or stderr containing "invalid json". + - Step 2 stdout missing both vendor markers. diff --git a/cicd/tests/testcases/crud/TC-CRUD-001.yml b/cicd/tests/testcases/crud/TC-CRUD-001.yml index 380e0d7652..2cd667e9f7 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-001.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-001.yml @@ -44,9 +44,44 @@ steps: expectPatterns: - "1|status.*true" +objective: | + Confirm the Test Project entity supports the full create-read-delete + lifecycle via XML-RPC. Project is the root entity in the TestLink + hierarchy, so this CRUD must work before suites or cases can be tested. + +judgeContext: | + Three steps, each via xmlrpc-capture.sh: + 1. Create — calls tl.createTestProject with a unique name derived from + {{testId}}-{{runId}}. Successful response contains id + followed by the new project ID; the helper captures it as projectId. + 2. Read — calls tl.getTestProjectByName with the same unique name. + Successful response contains the project name string verbatim, + proving the entity round-trips. + 3. Delete — calls tl.deleteTestProject by prefix (P). Successful + response contains 1 or status:true. + + Each helper invocation puts the raw XML response in stderr (where + expectPatterns matches) and a JSON envelope in stdout. A `{"ok": true, + "id": N}` stdout means the XML-RPC call returned a value; a `{"ok": + false, "faultCode": N}` means a fault was returned. + + Silent failures to watch for: + - Read step matches the expected name pattern but the response actually + contains a *different* project that happens to share part of the + name. The expected pattern is the full proj-- string, + so an exact substring match here is meaningful. + - Delete returns true but a stale project with the same prefix from a + prior aborted run is what got deleted (the new one persists). With + runId in the prefix this is unlikely but worth noting. + - Create reports success with no id pair (envelope + structure changed) — would fail expectPattern but read carefully. + criteria: | - Test project CRUD via XML-RPC: - - Create a project with a unique name and prefix derived from runId. - - Read it back by name and verify the name round-trips. - - Delete it by prefix so no residue remains in the DB. - IDs are captured from the create response; no hardcoded integers. + PASS when: + - Step 1 stdout shows {"ok": true, "id": } and stderr contains + id. + - Step 2 stderr contains the literal string "proj-{{testId}}-{{runId}}". + - Step 3 stderr contains 1. + + FAIL on any fault envelope, missing entity ID, name mismatch, or + failed deletion. diff --git a/cicd/tests/testcases/crud/TC-CRUD-002.yml b/cicd/tests/testcases/crud/TC-CRUD-002.yml index 4cd918bf35..2b7e9ec9c1 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-002.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-002.yml @@ -70,10 +70,44 @@ steps: expectPatterns: - "1|status.*true" +objective: | + Confirm the Test Suite entity supports CRUD under a parent project, with + parent-child IDs flowing through capture (no hardcoded numeric literals). + This validates the project→suite hierarchy that case-level tests build on. + +judgeContext: | + Five steps form a setup → exercise → teardown sequence: + 1. Create parent project — needed because suites can't exist without one. + 2. Create suite under {{projectId}} — captures suiteId from response. + 3. Read suite by {{suiteId}} — proves the suite's name round-trips. + 4. Delete suite by {{suiteId}}. + 5. Delete project by prefix. + + Steps 4 and 5 are reverse-creation-order teardown; both must succeed + even if step 3 fails (the framework runs them as part of the case). + + All XML-RPC calls go through xmlrpc-capture.sh (raw response in stderr, + JSON envelope in stdout). Successful create responses contain + id; successful reads contain the literal entity name; + successful deletes contain 1. + + Silent failures to watch for: + - Suite created but assigned to a different parent project (read still + returns the right name, but the parent_id in the response is wrong — + not directly checked here, but suspicious if entity counts in the + container logs disagree). + - Teardown delete returns true but only deleted a stale entity from a + previous run, leaving the new entity orphaned. Run-ID-suffixed names + make this unlikely, but flag if container logs show double-delete + or constraint violations. + - Read returns a fault envelope but the suite name string happens to + appear in the fault message — false positive. + criteria: | - Test suite CRUD via XML-RPC: - - Create a parent project (setup), capturing its ID. - - Create a suite under the project, capturing its ID from the response. - - Read it back by ID and confirm the name round-trips. - - Delete the suite, then the project, in reverse creation order. - All IDs flow through capture; no numeric literals used as parameters. + PASS when: + - Steps 1, 2 each return a new id (visible in id). + - Step 3 stderr contains "suite-{{testId}}-{{runId}}". + - Steps 4, 5 each return 1. + + FAIL on any fault envelope, missing entity, name mismatch, or + incomplete teardown (residue would block re-runs). diff --git a/cicd/tests/testcases/crud/TC-CRUD-003.yml b/cicd/tests/testcases/crud/TC-CRUD-003.yml index 040533253e..f1b657856d 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-003.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-003.yml @@ -103,10 +103,45 @@ steps: expectPatterns: - "1|status.*true" +objective: | + Confirm the Test Case entity supports CRUD under the suite/project + hierarchy, including the steps[] sub-structure. Test Case is the leaf + entity TestLink users actually author against, so its CRUD path is the + most user-visible. + +judgeContext: | + Seven steps in setup → create-leaf → read → reverse-teardown order: + 1. Create parent project (P prefix) + 2. Create parent suite under that project + 3. Create test case under that suite, with one inline step entry + 4. Read the case back by ID, verify its name + 5-7. Delete case → suite → project (reverse creation order) + + The test exercises: + - tl.createTestCase with embedded steps[] array (deeper XML-RPC payload + than the simpler create operations) + - getTestCase round-trip (case names round-trip via the suite hierarchy) + - cascade-aware teardown that doesn't rely on cascading deletes + + All calls go through xmlrpc-capture.sh (raw XML in stderr, JSON envelope + in stdout). Each create captures its new ID into a variable used by + subsequent steps; no numeric literals are baked into params. + + Silent failures to watch for: + - Case created but with empty steps[] (the embedded steps array got + serialized incorrectly). Read might still return the name, masking + the data-loss. Container logs may show steps_get returning 0 rows. + - Read returns the right name but a different testcaseid (rare; would + indicate a routing bug). + - Teardown delete-case returns true but the case row stays in the DB + because of an FK constraint — would block re-runs and surface as + "duplicate name" on the next attempt. + criteria: | - Test case CRUD via XML-RPC: - - Create parent project and suite with unique names. - - Create a test case in the suite, capturing its internal ID. - - Read it back by ID and verify the name round-trips. - - Delete in reverse order (case → suite → project). - All IDs flow through capture; no numeric literals used as parameters. + PASS when: + - Steps 1-3 each return a new id (visible in id). + - Step 4 stderr contains "case-{{testId}}-{{runId}}". + - Steps 5-7 each return 1. + + FAIL on any fault envelope, missing entity, name mismatch, or + incomplete teardown (would prevent idempotent re-run). diff --git a/cicd/tests/testcases/smoke/TC-SMOKE-001.yml b/cicd/tests/testcases/smoke/TC-SMOKE-001.yml index fe5d2d6b2e..708f893460 100644 --- a/cicd/tests/testcases/smoke/TC-SMOKE-001.yml +++ b/cicd/tests/testcases/smoke/TC-SMOKE-001.yml @@ -7,11 +7,34 @@ dependencies: [] steps: - name: GET /login.php - command: curl -sf http://localhost:8090/login.php | grep -o "Login" | head -1 + command: curl -sf "$TL_URL/login.php" | grep -o "Login" | head -1 expectPatterns: - "Login" +objective: | + Confirm the TestLink HTTP server is up and serving the login page on the + CI host port. This is the first gate of the smoke suite; if it fails, + every downstream test is meaningless. + +judgeContext: | + Step 1 runs `curl -sf` against $TL_URL/login.php (default + http://localhost:8091/login.php) and pipes through `grep -o "Login" | + head -1`. A healthy run prints the single word "Login" on stdout. + + Silent failures to watch for: + - HTTP 200 but the page content is "Service unavailable" or a generic + PHP fatal — grep finds no "Login" and stdout is empty, but exit code + could still be 0 because of the pipe. The expected "Login" pattern + not appearing is the real signal. + - Container running but Apache misconfigured: response is a directory + listing or default Apache page (no "Login" string). + - Curl follows a redirect to a different page (3xx) — grep silently + misses. + criteria: | Baseline check: the HTTP server is up and serving the login page. Session setup (docker compose up + DB init) is handled by cicd/scripts/run-tests.sh before any test runs. + + PASS when stdout contains "Login". + FAIL when stdout is empty or grep returned nothing. diff --git a/cicd/tests/testcases/smoke/TC-SMOKE-002.yml b/cicd/tests/testcases/smoke/TC-SMOKE-002.yml index a30ac786eb..4e54a44a00 100644 --- a/cicd/tests/testcases/smoke/TC-SMOKE-002.yml +++ b/cicd/tests/testcases/smoke/TC-SMOKE-002.yml @@ -15,5 +15,29 @@ steps: expectPatterns: - "Hello!" +objective: | + Confirm the XML-RPC API endpoint is reachable and the server responds to + the canonical no-auth ping. This proves the API layer is wired up before + any auth or CRUD test attempts a real call. + +judgeContext: | + Step 1 sends an XML-RPC `tl.ping` request via cicd/scripts/xmlrpc-capture.sh. + The helper: + - POSTs the request to $TL_URL/lib/api/xmlrpc/v1/xmlrpc.php + - mirrors the raw XML response to stderr (where expectPatterns can match) + - prints structured JSON to stdout: `{"ok": true, "id": N}` or `{"ok": true}` + on success, or `{"ok": false, "faultCode": N}` on a fault response + + A healthy ping returns the literal string "Hello!" inside an XML-RPC + response envelope (visible in stderr). + + Silent failures to watch for: + - HTTP 200 but the body is a PHP error page (no XML, no "Hello!"). + - XML-RPC fault returned where the faultString happens to mention + "Hello" (very unlikely, but pattern matchers can't tell). + - API endpoint disabled by config but the server returns an empty + valid-looking envelope. + criteria: | - The XML-RPC endpoint is reachable and returns the canonical Hello! greeting. + PASS when the response (visible in stderr) contains the string "Hello!". + FAIL on a fault response, an HTML error page, or empty output. diff --git a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml index e6e365eb8e..64a729c3ab 100644 --- a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml +++ b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml @@ -103,9 +103,41 @@ steps: expectPatterns: - "1|status.*true" +objective: | + Exercise the full entity-creation graph (project → suite → case → read → + reverse-delete) end-to-end with IDs flowing through capture. This is the + workflow-layer integration anchor — if the individual CRUD tests pass + but this fails, the bug is in cross-entity coordination, not in any + single CRUD path. + +judgeContext: | + Seven steps that mirror TC-CRUD-003 in shape but use the W + prefix to avoid colliding with the CRUD suite's P entities + (so both can run back-to-back without DB residue). + + The read-back step (4) verifies the leaf case round-trips. The fact + that all three creates and the read succeed in sequence is the real + signal — it proves the project_id → suite_id → case_id chain is + consistent, not just that each create works in isolation. + + All calls go through xmlrpc-capture.sh; raw XML in stderr, + {"ok": true, "id": N} JSON envelope in stdout. + + Silent failures to watch for: + - Same as TC-CRUD-003 plus: + - The W prefix collides with a stale entity from a prior + workflow run that wasn't cleaned up — create returns a fault about + duplicate prefix. Container logs would show this clearly. + - Suite is created with the wrong project_id (perhaps capture + extracted the wrong field) — case creation would still succeed but + the entity-graph integrity is broken; downstream read returns the + case but its parent path is wrong. + criteria: | - End-to-end workflow exercising the full entity graph: - project → testsuite → testcase → read → delete (reverse order). - Each entity's ID is captured from its creation response and used - in the next step. Delete uses the captured IDs, not external IDs, - so the test works regardless of how many entities exist in the DB. + PASS when all 7 steps complete successfully: + - Steps 1-3: each create returns a new id. + - Step 4: read-back stderr contains "case-{{testId}}-{{runId}}". + - Steps 5-7: each delete returns 1. + + FAIL on any fault envelope, name mismatch on read, or incomplete + teardown that leaves residue in the DB. From d0da4fdcc83410b0bbd8a0f88dc155cb410c276f Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:12:02 -0400 Subject: [PATCH 10/13] fix(ci): refresh run-tests.sh examples; revert judge unload to /api/generate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - run-tests.sh header still listed --suite integration and TC-E2E-001 in the usage examples, both deleted in the suite restructure. Update to --suite crud and --id TC-WORKFLOW-001. - LLMJudge.unloadModel was POSTing to /api/chat with messages: [], which is awkward — the unload doesn't need conversational semantics, and an empty messages array isn't well-defined for the chat endpoint. Use /api/generate with keep_alive: 0 (Ollama's documented unload pattern). /api/chat stays in place for the actual judging. Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/scripts/run-tests.sh | 8 ++++---- cicd/tests/src/judge/llm-judge.ts | 6 ++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/cicd/scripts/run-tests.sh b/cicd/scripts/run-tests.sh index 7f06d7dbf9..12a7448729 100755 --- a/cicd/scripts/run-tests.sh +++ b/cicd/scripts/run-tests.sh @@ -10,10 +10,10 @@ # cicd/scripts/run-tests.sh [--suite ] [--id ] [tsx cli flags] # # Examples: -# cicd/scripts/run-tests.sh # all suites -# cicd/scripts/run-tests.sh --suite integration -# cicd/scripts/run-tests.sh --suite build # no ci-up/ci-down -# cicd/scripts/run-tests.sh --id TC-E2E-001 --no-llm +# cicd/scripts/run-tests.sh # all suites +# cicd/scripts/run-tests.sh --suite crud +# cicd/scripts/run-tests.sh --suite build # no ci-up/ci-down +# cicd/scripts/run-tests.sh --id TC-WORKFLOW-001 --no-llm set -euo pipefail diff --git a/cicd/tests/src/judge/llm-judge.ts b/cicd/tests/src/judge/llm-judge.ts index 0ea79021a2..1c8517050f 100644 --- a/cicd/tests/src/judge/llm-judge.ts +++ b/cicd/tests/src/judge/llm-judge.ts @@ -265,11 +265,13 @@ export class LLMJudge { async unloadModel(): Promise { try { process.stderr.write(` [LLM] Unloading judge model ${this.model}...\n`); + // Use /api/generate (not /api/chat) for the unload — it doesn't need + // conversational semantics and an empty messages[] is awkward. + // keep_alive=0 is the documented Ollama unload pattern. await axios.post( - `${this.ollamaUrl}/api/chat`, + `${this.ollamaUrl}/api/generate`, { model: this.model, - messages: [], keep_alive: 0, }, { From 5500e6e2ea0140c632736f157341f070e22386e7 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:12:12 -0400 Subject: [PATCH 11/13] refactor(ci): plumb admin API key through TL_DEV_KEY env var Replace 24 hardcoded literals of the admin API key in YAML test steps with the framework's {{devKey}} substitution. The key now flows from a single source: cicd/tests/.env (TL_DEV_KEY) -> run-tests.sh / ci-up.sh source it -> executor.ts injects it into TestExecutor.variables as {{devKey}} -> ci-up.sh forwards it via `docker compose exec -e TL_DEV_KEY` into init-db.sh, which seeds the matching value into the users table Each consumer falls back to the previous hardcoded default (a1b2c3d4...) when TL_DEV_KEY is unset, so existing setups keep working without an .env file. Why: rotating the seeded admin key used to require editing init-db.sh, ci-up.sh, and 5 YAML files. Now it's one .env line. Co-Authored-By: Claude Opus 4.7 (1M context) --- cicd/scripts/ci-up.sh | 7 +++++-- cicd/scripts/init-db.sh | 5 +++-- cicd/tests/src/executor.ts | 4 ++++ cicd/tests/testcases/auth/TC-AUTH-001.yml | 5 +++-- cicd/tests/testcases/crud/TC-CRUD-001.yml | 6 +++--- cicd/tests/testcases/crud/TC-CRUD-002.yml | 10 +++++----- cicd/tests/testcases/crud/TC-CRUD-003.yml | 14 +++++++------- cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml | 14 +++++++------- 8 files changed, 37 insertions(+), 28 deletions(-) diff --git a/cicd/scripts/ci-up.sh b/cicd/scripts/ci-up.sh index 48460e7042..d599ebd8de 100755 --- a/cicd/scripts/ci-up.sh +++ b/cicd/scripts/ci-up.sh @@ -39,9 +39,12 @@ for i in $(seq 1 60); do done echo "Initializing database..." -docker compose -f "$COMPOSE_FILE" exec -T app bash /var/www/html/cicd/scripts/init-db.sh +# Forward TL_DEV_KEY into the container so init-db.sh seeds the matching +# admin key. Falls back to the script's own default if the env var is unset. +docker compose -f "$COMPOSE_FILE" exec -T -e "TL_DEV_KEY=${TL_DEV_KEY:-}" app \ + bash /var/www/html/cicd/scripts/init-db.sh echo "=== CI Environment Ready ===" echo "TestLink URL: $TL_URL" echo "Admin user: admin / admin" -echo "API key: a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" +echo "API key: ${TL_DEV_KEY:-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4}" diff --git a/cicd/scripts/init-db.sh b/cicd/scripts/init-db.sh index f0a959163e..9c8b40aa5d 100755 --- a/cicd/scripts/init-db.sh +++ b/cicd/scripts/init-db.sh @@ -34,8 +34,9 @@ sed 's|/\*prefix\*/||g' "$SQL_DIR/testlink_create_default_data.sql" | \ psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -q # Generate an API key for the admin user (user_id=1) -# The key is a known value so test cases can use it -API_KEY="a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" +# The key is a known value so test cases can use it. Override via TL_DEV_KEY +# (see cicd/tests/.env); ci-up.sh forwards the env var into this exec. +API_KEY="${TL_DEV_KEY:-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4}" echo "Setting admin API key..." psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -q -c \ "UPDATE users SET script_key='$API_KEY' WHERE id=1;" diff --git a/cicd/tests/src/executor.ts b/cicd/tests/src/executor.ts index b9a27858bc..1929cfaea9 100644 --- a/cicd/tests/src/executor.ts +++ b/cicd/tests/src/executor.ts @@ -206,6 +206,10 @@ export class TestExecutor { this.variables = { runId: Date.now().toString(), testId: testCase.id, + // Admin API key, sourced from cicd/tests/.env (TL_DEV_KEY) and matching + // what init-db.sh seeded. Default mirrors the seed default so tests + // keep working if the env var is unset. + devKey: process.env.TL_DEV_KEY || 'a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4', }; this.progress( `[${timestamp}] [${this.currentTest}/${this.totalTests}] ${testCase.id}: ${testCase.name}` diff --git a/cicd/tests/testcases/auth/TC-AUTH-001.yml b/cicd/tests/testcases/auth/TC-AUTH-001.yml index d85dedcdf9..35809e58c4 100644 --- a/cicd/tests/testcases/auth/TC-AUTH-001.yml +++ b/cicd/tests/testcases/auth/TC-AUTH-001.yml @@ -11,7 +11,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.checkDevKey - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} XMLDOC expectPatterns: @@ -24,7 +24,8 @@ objective: | judgeContext: | Step 1 sends `tl.checkDevKey` with the seeded admin key - (a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4) via cicd/scripts/xmlrpc-capture.sh. + (passed in as `{{devKey}}` from TL_DEV_KEY in cicd/tests/.env, matching + what init-db.sh seeded into the users table) via cicd/scripts/xmlrpc-capture.sh. The helper mirrors the raw XML response to stderr; on success the stdout JSON envelope is `{"ok": true}`. diff --git a/cicd/tests/testcases/crud/TC-CRUD-001.yml b/cicd/tests/testcases/crud/TC-CRUD-001.yml index 2cd667e9f7..449aa8e82d 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-001.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-001.yml @@ -11,7 +11,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectnameproj-{{testId}}-{{runId}} testcaseprefixP{{runId}} notes{{testId}} @@ -26,7 +26,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.getTestProjectByName - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectnameproj-{{testId}}-{{runId}} XMLDOC @@ -37,7 +37,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} prefixP{{runId}} XMLDOC diff --git a/cicd/tests/testcases/crud/TC-CRUD-002.yml b/cicd/tests/testcases/crud/TC-CRUD-002.yml index 2b7e9ec9c1..44f28889ce 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-002.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-002.yml @@ -11,7 +11,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectnameproj-{{testId}}-{{runId}} testcaseprefixP{{runId}} notes{{testId}} @@ -26,7 +26,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectid{{projectId}} testsuitenamesuite-{{testId}}-{{runId}} detailsCreated by {{testId}} @@ -41,7 +41,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.getTestSuiteByID - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testsuiteid{{suiteId}} XMLDOC @@ -52,7 +52,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testsuiteid{{suiteId}} XMLDOC @@ -63,7 +63,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} prefixP{{runId}} XMLDOC diff --git a/cicd/tests/testcases/crud/TC-CRUD-003.yml b/cicd/tests/testcases/crud/TC-CRUD-003.yml index f1b657856d..1bee4c176b 100644 --- a/cicd/tests/testcases/crud/TC-CRUD-003.yml +++ b/cicd/tests/testcases/crud/TC-CRUD-003.yml @@ -11,7 +11,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectnameproj-{{testId}}-{{runId}} testcaseprefixP{{runId}} notes{{testId}} @@ -26,7 +26,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectid{{projectId}} testsuitenamesuite-{{testId}}-{{runId}} detailsCreated by {{testId}} @@ -41,7 +41,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcasenamecase-{{testId}}-{{runId}} testsuiteid{{suiteId}} testprojectid{{projectId}} @@ -63,7 +63,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.getTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcaseid{{caseId}} XMLDOC @@ -74,7 +74,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcaseid{{caseId}} XMLDOC @@ -85,7 +85,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testsuiteid{{suiteId}} XMLDOC @@ -96,7 +96,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} prefixP{{runId}} XMLDOC diff --git a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml index 64a729c3ab..3aa4f7e609 100644 --- a/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml +++ b/cicd/tests/testcases/workflow/TC-WORKFLOW-001.yml @@ -11,7 +11,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectnameproj-{{testId}}-{{runId}} testcaseprefixW{{runId}} notes{{testId}} @@ -26,7 +26,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testprojectid{{projectId}} testsuitenamesuite-{{testId}}-{{runId}} details{{testId}} @@ -41,7 +41,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.createTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcasenamecase-{{testId}}-{{runId}} testsuiteid{{suiteId}} testprojectid{{projectId}} @@ -63,7 +63,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.getTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcaseid{{caseId}} XMLDOC @@ -74,7 +74,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestCase - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testcaseid{{caseId}} XMLDOC @@ -85,7 +85,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestSuite - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} testsuiteid{{suiteId}} XMLDOC @@ -96,7 +96,7 @@ steps: command: | bash cicd/scripts/xmlrpc-capture.sh <<'XMLDOC' tl.deleteTestProject - devKeya1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 + devKey{{devKey}} prefixW{{runId}} XMLDOC From 8ca3ee8a045c0b9eec2f37a885a9348bfb193a68 Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:30:26 -0400 Subject: [PATCH 12/13] chore(ci): scope automated workflows to main branch only - docker-publish.yml: switch to workflow_dispatch only with a job-level guard restricting it to refs/heads/testlink_1_9_20_fixed, so accidental manual dispatches from a feature branch can't publish. Forks without GHCR write auth no longer fail on every branch push. - test-pipeline.yml: drop the pull_request trigger; PRs no longer fire CI automatically. Use workflow_dispatch from the Actions tab to run against a feature branch when needed. Push to main still runs it. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/docker-publish.yml | 11 ++++++++--- .github/workflows/test-pipeline.yml | 5 +++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index d164875220..92561b1ae3 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -5,9 +5,13 @@ name: Docker # separate terms of service, privacy policy, and support # documentation. +# Manual-only: this workflow publishes to GHCR and requires registry write +# auth that's not configured on every fork. Trigger from the Actions tab +# when you actually want to release. Restricted to the main branch via the +# job's `if:` guard below so accidental dispatches from feature branches +# can't publish. on: - push: - branches: [ "*" ] + workflow_dispatch: env: # Use docker.io for Docker Hub if empty @@ -18,7 +22,8 @@ env: jobs: build: - + # Guard: only ever publish from the main branch, even when dispatched. + if: github.ref == 'refs/heads/testlink_1_9_20_fixed' runs-on: ubuntu-latest permissions: contents: read diff --git a/.github/workflows/test-pipeline.yml b/.github/workflows/test-pipeline.yml index 2297a44090..cf64791955 100644 --- a/.github/workflows/test-pipeline.yml +++ b/.github/workflows/test-pipeline.yml @@ -1,10 +1,11 @@ name: CI Pipeline on: + # Only fire automatically on push to the main branch. PRs do not trigger; + # use workflow_dispatch from the Actions tab to run against a feature + # branch on demand. push: branches: ["testlink_1_9_20_fixed"] - pull_request: - branches: ["testlink_1_9_20_fixed"] workflow_dispatch: inputs: judge_mode: From 0defe293b3779cae65b03eb351a87e38fbebb1dd Mon Sep 17 00:00:00 2001 From: Shang Chieh Tseng Date: Sun, 19 Apr 2026 12:31:02 -0400 Subject: [PATCH 13/13] chore(ci): test-pipeline is manual-only too Drop the push-to-main trigger. Both workflows (CI Pipeline and Docker) now only fire via workflow_dispatch. Run from the Actions tab against whichever ref you want to validate or release. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/test-pipeline.yml | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/.github/workflows/test-pipeline.yml b/.github/workflows/test-pipeline.yml index cf64791955..fecf2dd8c5 100644 --- a/.github/workflows/test-pipeline.yml +++ b/.github/workflows/test-pipeline.yml @@ -1,11 +1,9 @@ name: CI Pipeline on: - # Only fire automatically on push to the main branch. PRs do not trigger; - # use workflow_dispatch from the Actions tab to run against a feature - # branch on demand. - push: - branches: ["testlink_1_9_20_fixed"] + # Manual-only: trigger from the Actions tab. No automatic firing on push + # or PR — keeps CI noise off feature branches and lets you run the suite + # against any ref on demand. workflow_dispatch: inputs: judge_mode: