feat: support TESTSPRITE_PROJECT_ID default#144
Conversation
The Lint & Format job runs prettier --check over workflow YAML; the aligned trailing comment failed it on every push.
Updated the README with a new link and added a video description.
- prettier-clean the README video block (fixes CI format:check on main) - bump version to 0.1.1 - CHANGELOG: add [0.1.1] docs-only entry Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
NPM_TOKEN secret was never configured, so tag-triggered releases failed with ENEEDAUTH. Auth now uses npm trusted publishing (configured on npmjs.com for this repo + release.yaml): - drop NODE_AUTH_TOKEN env (empty token would shadow OIDC auth) - upgrade npm to >= 11.5.1 (trusted publishing requirement; Node 22 bundles 10.x) - bump checkout/setup-node to v5 ahead of the June 16 Node 20 runner cutoff
ci(release): switch npm publish to OIDC trusted publishing
- Onboarding consolidated into `testsprite setup` (formerly `init`); the granular auth commands remain as hidden, deprecated aliases. - CLI reports its version in the User-Agent header. - README: the launch video no longer renders as a bare URL on npm.
create-batch --run launched its first concurrencyLimit triggers in parallel, but the steady-state loop awaited each subsequent job to fully finish (trigger + full --wait poll) before launching the next one. Effective concurrency dropped to 1 after the initial wave regardless of --max-concurrency. Switch to the launch-then-race pattern already used by the other three fan-outs in this file: launch up to the limit, relaunch on each completion via startNext(), never await a whole job inline. Add a regression test using equal-delay trigger responses so the first wave settles in the same microtask batch, which is the exact condition that exposed the bug.
…urrency fix(test): prevent batch-run scheduler from serializing after first wave
…estSprite#7) test code get --out <path> opened (truncated) the destination file before the network request. If the GET then failed, or hit the "no code generated yet" branch which writes nothing, the user's pre-existing --out file was left emptied with no way to recover it. Write to a sibling temp file instead and rename it onto the real path only after a successful, complete write. Mirrors the atomic rename contract bundle.ts already uses for multi-file bundles. Add regression tests for both failure modes: a failing fetch and the no-code-yet branch. Both reproduce the truncation on the old code and pass on the fix.
…ite#9) Batch-rerun chunks (>50 testIds) were dispatched concurrently via Promise.all. The backend's producer/teardown closure dedup happens per-request, not across requests, so two concurrent chunks sharing a project's producer could each independently decide to trigger it, double-running the producer or teardown. Dispatch chunks sequentially in both the initial and deferred-retry paths, closing the race at the source. Also dedupe the aggregated accepted[] by testId and merge closure.byProject across chunks as a defensive second layer, warning on stderr if a duplicate trigger is detected. Fixed a pre-existing test whose fixture relied on the old double-counting behavior (same accepted entry returned from every retry call).
…RROR (TestSprite#19) A malformed API endpoint produced an opaque or misleading failure instead of a clear config error: --endpoint-url "not a url" -> `Error: Invalid URL` (exit 1, a raw `new URL()` throw with no guidance) --endpoint-url "localhost:3000" (missing scheme, parses as scheme "localhost:") and "ftp://x" (wrong scheme) -> `fetch failed` / Service unavailable, emitted only after a full retry+backoff cycle — looks like a network outage, not a config typo. Add `assertValidEndpointUrl` in client-factory.ts and run it in both the real and dry-run paths of `makeHttpClient`, on the resolved endpoint (so it covers --endpoint-url, TESTSPRITE_API_URL, and the credentials file). A malformed value now throws a typed VALIDATION_ERROR (exit 5) with an actionable message. Crucially, and unlike the `--target-url` SSRF guard, this does NOT reject localhost or private hosts — the API endpoint legitimately points at a self-hosted, local-dev, or mock backend. Only syntactically invalid values (unparseable, or a non-http(s) scheme) are rejected, so existing self-hosted/CI configs and the test suite's localhost mock backend are unaffected. Adds unit coverage for assertValidEndpointUrl and the two makeHttpClient paths, plus subprocess regressions.
runFailureGet now resolves and validates --out via resolveBundleDir and assertOutDirParentExists before calling GET /tests/{id}/failure, matching runArtifactGet and runCodeGet fast-fail behavior.
Adds regression tests asserting zero fetch calls on empty --out and missing parent dir paths.
…TestSprite#23) Co-authored-by: zeshi-du <zeshi@testsprite.com>
…on (TestSprite#21) A profile name (`--profile` / `TESTSPRITE_PROFILE`) is written verbatim as an INI section header (`[name]`) in `~/.testsprite/credentials`, but was never validated. A name containing the characters that break that grammar silently corrupted the file: --profile "prod]" -> serialises to `[prod]]`, which the section regex cannot match, so the api_key/api_url lines that follow are DROPPED on read. `setup` reports success while the credential never persists. --profile $'a\nb' -> the newline splits the header across two lines. --profile " x " -> does not round-trip (the parser trims section names, so it reads back as `x`). Add `assertValidProfileName` in credentials.ts (a conservative allowlist: letters, digits, dot, underscore, hyphen — covering `default`, `prod`, `ci-staging`, `team.qa`) and call it from every profile-keyed entry point (`readProfile`, `writeProfile`, `deleteProfile`). A malformed name now throws a typed VALIDATION_ERROR (exit 5) before any file write, instead of silently corrupting or failing to persist credentials. Adds unit coverage for the guard and the three entry points, plus a subprocess regression. Co-authored-by: Zeshi Du <duke.zeshi@gmail.com>
…t json (TestSprite#22) When a subcommand fired a parse error (unknown command, missing required argument, invalid option), Commander's outputError callback wrote plain text to stderr immediately and the catch block exited 5 with no further output. A machine consumer that always parses stderr as JSON received an unexpected plain-text string and crashed its JSON.parse. Root cause: configureOutput was only applied to the root program, not to subcommands. Each subcommand retained the default outputError that calls write(str) directly. applyExitOverrideDeep now also propagates configureOutput to every leaf so the message is buffered instead of written. In the CommanderError catch block, a resolved output mode is used to either write a VALIDATION_ERROR JSON envelope or the buffered plain-text message. An argv scan fallback handles the edge case where --output json appears after the bad argument and was not yet parsed when the error fired. The renderCommanderError helper is extracted to src/lib/render-error.ts (alongside the existing rephraseUnknownOption helper) so it is unit-testable without a subprocess. Eight unit tests cover JSON/text output, null fallback, message trimming, and rephrased global-flag embedding. Four subprocess regression tests in the [fix-5] block cover missing-arg, unknown subcommand, argv-fallback, and text-mode no-regression paths. Co-authored-by: zeshi-du <zeshi@testsprite.com>
New issue form for hackathon submissions — auto-applies the `hackathon` label and captures the submitter's Discord identity for reward payout coordination.
…rd (TestSprite#37) assertNotLocal lowercased the hostname but did not strip a trailing dot, so http://localhost. (the FQDN form of localhost, RFC 6761) and http://localhost%2e bypassed the host === 'localhost' loopback check. IP literals are already dot-normalized by the WHATWG URL parser, so only named hosts were affected. Strips one trailing dot before the comparison. Adds 4 regression tests (3 blocked variants + 1 public-FQDN no-false-positive).
* fix(skill-nudge): require complete Codex managed section * docs(skill-nudge): document helper contracts --------- Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com>
TestSprite#36) project create/update validated --name with the action handler's if (!name) check, which a whitespace-only string passes (a non-empty string is truthy). The blank name was then sent verbatim, creating a junk-named project. The sibling est create already rejects this via the requireString whitespace guard (dogfood P1 fix TestSprite#1); this aligns project create/update with that behavior. Adds 2 regression tests.
…Sprite#14) Only `test` and `project` validated the global `--output` flag. The `auth`, `usage`, `agent`, and `init` command groups resolved it with `globals.output ?? 'text'`, so an unrecognised value (e.g. a typo like `--output josn`) was silently coerced to text mode instead of being rejected. A coding agent that asked for `--output json` then received a human-readable text payload and failed to parse it as JSON, with no signal as to why. Extract the validation into a shared `resolveOutputMode` helper in `lib/output.js` and route every command group's `resolveCommonOptions` through it. Invalid values now throw a typed VALIDATION_ERROR (exit 5) with an actionable message everywhere. This also unifies the error wording, which previously differed between `test` ("Flag `--output` is invalid: must be one of: json, text.") and `project` ("--output must be one of: json, text").
…prite#34) * fix(test): reject empty code get --out and strip code-file BOM When test code get --out receives an empty inline body, closeOutputFile left a zero-byte file with exit 0 — scripts and agents treated that as a successful download. Abort the sink, unlink any artifact, and surface VALIDATION_ERROR instead. Plan/steps JSON already strip a leading UTF-8 BOM from PowerShell 5.1 files; apply the same strip to --code-file reads so uploaded test source is not corrupted by an invisible U+FEFF prefix. * fix(test): rebase onto v0.2.0 and harden empty-code --out cleanup - Resolve rebase conflicts: keep atomic temp-file --out writes from main while rejecting empty inline code with VALIDATION_ERROR (exit 5) - abortOutputFile: wait for stream close before unlinking tmpPath - BOM test: use .py code-file (assertPythonCodeFile from v0.2.0) - CI: build before test + fileParallelism false (dist/ race flake)
Co-authored-by: cmdr-chara <249489759+cmdr-chara@users.noreply.github.com>
…run and run --all (TestSprite#128) test run, test create, create-batch, plan put, code put, update, and delete all print the auto-minted idempotency key to stderr under --output json (as well as --verbose / --debug) so JSON-mode automation can capture the key and replay a retry safely. test rerun and test run --all minted a key but only echoed it under --debug / --verbose, so CI flows using --output json silently lost it. Align both paths with the shared guard used by every other minting site, and cover the JSON-mode emission (and the text-mode silence) with regression tests. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…alls (TestSprite#129) The codex managed-section branch already runs inspectTargetPath during --dry-run ([P2]) so a planted symlink is refused the same way the real install refuses it. The own-file targets (claude, cursor, cline, antigravity) skipped that guard in dry-run and reported the write as successful, so 'agent install --dry-run' could claim success for an install that would actually exit 5. Run the same guard in the own-file dry-run branch and cover both the symlinked-target and symlinked-parent cases with regression tests. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ared deadline (TestSprite#130) runTestRunAll computes each member's poll budget as Math.max(1, ceil(batchDeadlineMs - now)), so a run whose turn arrives after the shared --timeout deadline still gets a fresh >=1s poll. With --max-concurrency bounding the fan-out, a batch could overshoot the documented shared deadline and report a late 'passed' for a member that should have been reported as 'timeout' (exit 7). Guard the poll helper the same way the create-batch --run --wait path already does: if the shared deadline is exhausted before a member's poll starts, return the existing timeout-shaped member result without calling pollRunUntilTerminal. Regression test drives the clock past the deadline during the first member's poll and asserts the second member is never polled and reports 'timeout'. Co-authored-by: ahndohun <19940813+ahndohun@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…rd (TestSprite#37) assertNotLocal lowercased the hostname but did not strip a trailing dot, so http://localhost. (the FQDN form of localhost, RFC 6761) and http://localhost%2e bypassed the host === 'localhost' loopback check. IP literals are already dot-normalized by the WHATWG URL parser, so only named hosts were affected. Strips one trailing dot before the comparison. Adds 4 regression tests (3 blocked variants + 1 public-FQDN no-false-positive).
…TestSprite#17) `parseRequestTimeoutFlag` was copy-pasted byte-for-byte into five command files (auth, project, usage, init, test). Every copy silently returned `undefined` for an invalid value, so an explicit `--request-timeout 30s` (a natural "30 seconds" typo) resolved to undefined and the command ran with the default 120s deadline — the operator believed they had set a timeout but had not, with no signal. Hoist a single definition into client-factory.ts (next to resolveRequestTimeoutMs and the REQUEST_TIMEOUT_* constants) and make the flag strict: a non-numeric, zero, or negative value now throws a typed VALIDATION_ERROR (exit 5), consistent with every other validated flag (--page-size, --output, --type). Positive out-of-range values are still accepted and clamped by resolveRequestTimeoutMs, and the TESTSPRITE_REQUEST_TIMEOUT_MS env-var path stays lenient by design (a stray global env var should not hard-fail every command). Adds unit coverage for parseRequestTimeoutFlag and a subprocess regression that `--request-timeout 30s` exits 5 instead of falling back to 120s.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThis PR adds ChangesTESTSPRITE_PROJECT_ID default project support
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant Command
participant resolveProjectId
participant API
CLI->>Command: invoke without --project
Command->>resolveProjectId: resolveProjectId(projectId, deps)
resolveProjectId-->>Command: TESTSPRITE_PROJECT_ID fallback
Command->>Command: requireProjectId(projectId)
Command->>API: request using resolved projectId
Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (3)
src/commands/test.ts (3)
7785-7789: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winError message doesn't mention the new
TESTSPRITE_PROJECT_IDfallback.
requireProjectId's message ("is required") is used bytest listandtest createwhen neither--projectnorTESTSPRITE_PROJECT_IDis set, but it gives no hint about the env var. Compare with the bespoke check fortest run --all(Line 7475) which explicitly sayspass --project <id> or set TESTSPRITE_PROJECT_ID. Since all three commands now support the same fallback, the generic message should mention it too for a consistent, actionable error across commands.♻️ Suggested consistent message
function requireProjectId(projectId: string | undefined): asserts projectId is string { if (typeof projectId !== 'string' || projectId.length === 0) { - throw localValidationError('project', 'is required'); + throw localValidationError('project', 'is required; pass --project <id> or set TESTSPRITE_PROJECT_ID'); } }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/test.ts` around lines 7785 - 7789, Update requireProjectId in test.ts so its validation error mentions the TESTSPRITE_PROJECT_ID fallback instead of only saying “is required,” matching the messaging used by testRunAll and the shared behavior of test list, test create, and test run. Adjust the localValidationError text to instruct users to pass --project <id> or set TESTSPRITE_PROJECT_ID, and keep the check centralized in requireProjectId so all callers get the same actionable message.
7779-7784: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
resolveProjectIdcorrectly implements env-var fallback with trimming.One edge case to note: an explicitly passed empty string (
--project "") is treated as "set" (since the check is!== undefined), bypassing theTESTSPRITE_PROJECT_IDfallback entirely and failing with "is required" instead of falling back to the env var. Also, the env var value is trimmed but an explicit--projectvalue is not — a minor asymmetry.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/test.ts` around lines 7779 - 7784, The `resolveProjectId` helper treats an explicitly passed empty string as a real value, which prevents the `TESTSPRITE_PROJECT_ID` fallback and can lead to a missing-project error. Update `resolveProjectId` to treat blank CLI input the same as unset by trimming the `projectId` argument before deciding whether to return it or fall back to `deps.env`/`process.env`, keeping behavior consistent with the env-var handling.
7469-7477: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate validation logic vs.
requireProjectId.This manual
if (!projectId) throw localValidationError(...)duplicates the same conditionrequireProjectIdalready encodes (andrunTestRunAllre-checks it again internally viaresolveProjectId/requireProjectIdon the now-resolved value). Consider havingrequireProjectIdaccept an optional custom message/hint so both call sites share one implementation instead of maintaining two divergent error strings for the same validation rule.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/test.ts` around lines 7469 - 7477, The `isAll` branch in `runTestRunAll` duplicates the same project-id validation already handled by `requireProjectId`, so consolidate the check into a single implementation. Update `requireProjectId` to accept an optional custom error message or hint, then use it from the `--all` path instead of the manual `if (!projectId) throw localValidationError(...)` block. Keep the existing `resolveProjectId`, `requireProjectId`, and `runTestRunAll` flow consistent so both call sites share the same validation rule and error wording.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/commands/test.ts`:
- Around line 7785-7789: Update requireProjectId in test.ts so its validation
error mentions the TESTSPRITE_PROJECT_ID fallback instead of only saying “is
required,” matching the messaging used by testRunAll and the shared behavior of
test list, test create, and test run. Adjust the localValidationError text to
instruct users to pass --project <id> or set TESTSPRITE_PROJECT_ID, and keep the
check centralized in requireProjectId so all callers get the same actionable
message.
- Around line 7779-7784: The `resolveProjectId` helper treats an explicitly
passed empty string as a real value, which prevents the `TESTSPRITE_PROJECT_ID`
fallback and can lead to a missing-project error. Update `resolveProjectId` to
treat blank CLI input the same as unset by trimming the `projectId` argument
before deciding whether to return it or fall back to `deps.env`/`process.env`,
keeping behavior consistent with the env-var handling.
- Around line 7469-7477: The `isAll` branch in `runTestRunAll` duplicates the
same project-id validation already handled by `requireProjectId`, so consolidate
the check into a single implementation. Update `requireProjectId` to accept an
optional custom error message or hint, then use it from the `--all` path instead
of the manual `if (!projectId) throw localValidationError(...)` block. Keep the
existing `resolveProjectId`, `requireProjectId`, and `runTestRunAll` flow
consistent so both call sites share the same validation rule and error wording.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: bd688675-74da-4adc-a5f1-e51a303832b1
📒 Files selected for processing (4)
DOCUMENTATION.mdsrc/commands/test.run.spec.tssrc/commands/test.test.tssrc/commands/test.ts
|
Approved — |
|
Hi @naufalfx805-source — so sorry for the disruption here. 🙏 While we were publishing a new CLI release, a hiccup in our process unintentionally closed this PR. To be clear: your contribution wasn't rejected, and nothing on your end caused this. Your work is completely safe — your branch and every commit are intact and untouched, and Because of how the closure happened, the cleanest path is a fresh PR from your existing branch (rather than reopening this one) — and opening it yourself keeps it under your name, with full credit for your work: 👉 Open a new PR from your branch It should merge cleanly. So your work doesn't get lost, if we don't hear back in the next ~3 days we'll go ahead and open it for you — but opening it yourself keeps it under your name, and either way we're always happy to hand it back to you. Thank you for contributing to TestSprite, and again, we're sorry for the hassle. We really appreciate this work and want to see it merged. 🙌 |
Refs #76
Discord: npall_805
Summary:
Validation:
Summary by CodeRabbit
New Features
TESTSPRITE_PROJECT_IDfortest list,test create, andtest run --allwhen--projectis omitted.Bug Fixes
TESTSPRITE_PROJECT_ID.Documentation
DOCUMENTATION.mdto includeTESTSPRITE_PROJECT_IDas the default project and refined table formatting.