test: validate adapter authoring and override parity in local and hosted modes #311

Summary

Manually validate the WebCMD 0.7.1 authoring and override workflows in both local and hosted modes after:

• Prepare WebCMD 0.7.1 cloud/plugin parity #309
• agentrhq/webcmd-cloud#34

This is deliberately a manual end-to-end pass. The affected flows cross CLI mode selection, local storage, Cloud APIs, browser infrastructure, generated adapters, file transfer, and tenant scoping; unit/contract coverage alone cannot establish the complete user experience.

Goals

Validate and record the behavior of:

1. plugin overrides
2. adapter source discovery and editing
3. adapter authoring through browser init and browser verify
4. site memory, fixtures, and response samples
5. mutable input-output file arguments
6. hosted marketplace availability reporting
7. local/hosted isolation and regression boundaries

If local and hosted behavior is intentionally different, verify that the difference is explicit and produces actionable CLI guidance.

Prerequisites

• WebCMD built from 0.7.1 or the head of Prepare WebCMD 0.7.1 cloud/plugin parity #309
• WebCMD Cloud deployed from Fix sitemap author skill frontmatter #34
• two clean CLI homes, one local and one hosted
• a working local Cloak/daemon browser
• working hosted Browser Use infrastructure
• one hosted API key for workspace testing
• preferably a second user/API key for cross-user isolation testing
• an authenticated Twitter/X profile, or a private test adapter with an input-output argument

Record before testing:

webcmd --version
node --version
which webcmd
curl -sS "$WEBCMD_CLOUD_URL/version"

Suggested placeholders:

export LOCAL_TEST_HOME="/tmp/webcmd-local-parity"
export HOSTED_TEST_HOME="/tmp/webcmd-hosted-parity"
export TEST_SITE="example-parity"
export TEST_COMMAND="search"
export TEST_SITE_COMMAND="$TEST_SITE/$TEST_COMMAND"
export WEBCMD_WORKSPACE_A="parity-ws-a"
export WEBCMD_WORKSPACE_B="parity-ws-b"

Run webcmd setup separately with each test home and select the appropriate mode. Do not reuse one home between modes.

Test matrix

1. Mode boundaries

Local

Run basic commands without a Cloud API key:

HOME="$LOCAL_TEST_HOME" webcmd list -f json
HOME="$LOCAL_TEST_HOME" webcmd plugin list -f json

Expected:

• no hosted authentication is required
• commands and adapters come from local installation/overrides
• no Cloud request is made

Hosted

After hosted setup:

HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A" webcmd profile list -f json
HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A" webcmd plugin search github -f json

Expected:

• Cloud authentication succeeds
• workspace selection reaches Cloud
• hosted plugin rows include availability and excludedCommands
• hosted mode does not scan local ~/.webcmd/clis

Create a fake adapter inside the hosted test home and confirm it never appears in hosted webcmd list.

2. Plugin override and adapter source behavior

Use a real installed command such as linkedin/search for local testing.

Local override

HOME="$LOCAL_TEST_HOME" webcmd adapter override linkedin/search
HOME="$LOCAL_TEST_HOME" webcmd adapter status -f json
HOME="$LOCAL_TEST_HOME" webcmd adapter path linkedin/search
HOME="$LOCAL_TEST_HOME" webcmd adapter source get linkedin/search

Expected:

• override creates a local editable adapter
• status reports its provenance
• adapter path prints an existing source path
• local adapter source get prints the same path, not copied source contents
• executing the command uses the override

Negative checks:

HOME="$LOCAL_TEST_HOME" webcmd adapter source get linkedin/search --output /tmp/should-not-write.js
HOME="$LOCAL_TEST_HOME" webcmd adapter source put linkedin/search /tmp/source.js

Expected:

• both fail clearly in local mode
• guidance points to webcmd adapter path <site>/<command>
• no destination is created or modified

Reset the override and confirm the original command provenance and behavior return:

HOME="$LOCAL_TEST_HOME" webcmd adapter reset linkedin
HOME="$LOCAL_TEST_HOME" webcmd adapter status -f json

Hosted override/source editing

First determine the intended hosted behavior of webcmd adapter override. If direct override is unsupported, it must fail with explicit guidance to browser init / private adapter authoring rather than falling through as an unknown hosted command.

Create or identify an editable private hosted adapter, then run:

HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A" \
  webcmd adapter source get "$TEST_SITE_COMMAND" --output /tmp/hosted-source.js

cp /tmp/hosted-source.js /tmp/hosted-source-edited.js
printf '\n// manual parity edit\n' >> /tmp/hosted-source-edited.js

HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A" \
  webcmd adapter source put "$TEST_SITE_COMMAND" /tmp/hosted-source-edited.js

Expected:

• source is downloaded from the private Cloud package
• source put reinventories the package and reports registered commands
• the edited command executes using the new source
• system/default packages remain read-only
• invalid keys such as ../bad/search and site/../bad are rejected without writing
• source paths never escape the tenant package root

Also verify webcmd adapter path "$TEST_SITE_COMMAND" returns the stable hosted cache destination and does not misrepresent it as the authoritative Cloud storage path.

3. Site memory

Run the same logical workflow in both modes:

webcmd site note add "$TEST_SITE" --text "Manual parity note" --author "manual-tester"

webcmd site endpoint set "$TEST_SITE" search-api \
  --url "https://example.com/api/search" \
  --method GET \
  --params '{"q":"test"}' \
  --rows-path items \
  --fields title,url \
  --notes "Manual endpoint"

webcmd site field-map add "$TEST_SITE" 'items[].title' \
  --meaning "Result title" \
  --source manual-test \
  --force

webcmd site memory list "$TEST_SITE"
webcmd site memory show "$TEST_SITE" --kind notes
webcmd site memory show "$TEST_SITE" --kind endpoints
webcmd site memory show "$TEST_SITE" --kind field-map

Run once with HOME="$LOCAL_TEST_HOME", then with HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A".

Expected locally:

• content persists under the local site-memory root
• list reports notes.md, endpoints.json, and field-map.json
• writes are atomic and stay inside the selected test home

Expected hosted:

• the same CLI commands use Cloud APIs
• no local site-memory files are created
• data persists across CLI invocations
• list/show output is equivalent where the backend semantics match

Mark the endpoint stale and confirm the resulting state:

webcmd site endpoint stale "$TEST_SITE" search-api
webcmd site memory show "$TEST_SITE" --kind endpoints

Fixtures and samples

Create a valid fixture:

{
  "args": {"q": "agent"},
  "expect": {
    "columns": ["title", "url"],
    "notEmpty": ["title"],
    "rowCount": {"min": 1}
  }
}

In both modes:

webcmd site fixture put "$TEST_SITE_COMMAND" /tmp/manual-fixture.json
webcmd site fixture get "$TEST_SITE_COMMAND" --output /tmp/manual-fixture-roundtrip.json
webcmd site sample add "$TEST_SITE_COMMAND" /tmp/manual-fixture.json

Expected:

• fixture round-trips without corruption
• response sample appears under fixtures/
• malformed JSON is rejected
• a fixture with rowCount.min > rowCount.max is rejected
• a failed write leaves the previous valid fixture intact

4. Adapter authoring: browser init and verify

Run in both modes:

webcmd browser init "$TEST_SITE_COMMAND"
webcmd browser verify "$TEST_SITE_COMMAND" \
  --trace retain-on-failure \
  --max-top-level-keys 12

Expected locally:

• scaffold is created in local adapter storage
• adapter path resolves the new file
• local Cloak/daemon browser is used
• verify consumes local site memory/fixture

Expected hosted:

• scaffold becomes a private Cloud adapter package
• source is retrievable through hosted adapter source get
• Browser Use backed infrastructure is used
• maxTopLevelKeys reaches Cloud as numeric 12, not string "12"
• workspace selection is preserved throughout init, source access, fixture access, and verify

Exercise every verify option in both modes:

• --no-fixture
• --write-fixture
• --update-fixture
• --strict-memory
• --seed-args <value>
• --trace off|on|retain-on-failure
• --max-top-level-keys <positive integer>

Negative checks:

• invalid trace mode fails client-side
• zero/negative/non-integer maximum fails client-side
• strict memory fails clearly when notes/endpoints are absent
• write/update fixture actually persists to the correct backend
• retained trace is available on hosted failure

5. Mutable input-output files

Use twitter/bookmarks --resume-file or twitter/likes --resume-file. If the required account is unavailable, create a private test adapter declaring:

file: {
  direction: 'input-output',
  pathKind: 'file',
  multiple: false,
  contentTypes: ['application/json']
}

Test both an existing and missing resume file in local and hosted modes.

Existing file expectations:

• local mode passes the same path directly
• hosted CLI uploads the seed
• Cloud receives an input-output reference containing filename, content type, and inputId
• worker can read and modify only its prepared artifact roots
• returned artifact replaces/materializes to the requested local path

Missing file expectations:

• command can create it
• hosted CLI does not attempt an input upload
• Cloud still reserves the mutable output
• returned artifact materializes to the requested path

Regression checks:

• multiple mutable files for one argument map by filename
• ambiguous receipts fail with HOSTED_FILE_OUTPUT_INVALID
• wrong content type and excessive size fail before execution
• interrupted execution does not corrupt an existing local file

6. Hosted workspace and user isolation

Same user, different workspaces

Write a unique site-memory marker, private adapter edit, fixture, and execution artifact in workspace A. Attempt to read each from workspace B.

Expected:

• B cannot see A's site memory, adapter source, profiles, sessions, or artifacts
• using the same site/command/profile names does not cause state reuse

Different users

Repeat using a second API key with the same workspace ID string.

Expected:

• user B cannot access user A's data
• guessed execution/artifact/package/profile/session IDs return 401/403/404 as appropriate
• no response exposes filesystem paths, storage keys, or another tenant's metadata

Artifact checks must cover upload, run, and download. Browser checks must cover profiles, persistent sessions, and live-view capabilities.

7. Hosted marketplace/catalog

HOME="$HOSTED_TEST_HOME" WEBCMD_WORKSPACE="$WEBCMD_WORKSPACE_A" \
  webcmd plugin search "" -f json > /tmp/hosted-marketplace.json

Expected:

• every enabled official plugin is represented
• every result includes availability and excludedCommands
• fully hosted plugins expose all eligible commands
• mixed plugins expose their hosted subset and list excluded local commands
• local-only plugins are not described as Cloud-runnable
• adding a new enabled plugin to the WebCMD catalog causes the reconciliation job to evaluate it without adding it to a separate Cloud allowlist

Install one hosted plugin and one mixed plugin. Confirm supported commands enter the tenant manifest and excluded commands do not.

8. Regression checks

• local mode never requires Cloud credentials
• hosted mode never scans or executes arbitrary local adapter files
• local adapter execution and browser daemon behavior remain unchanged
• unsupported output formats produce the intended 0.7.1 wording and exit code in both modes
• adapter source lookup rejects stale/nonexistent paths
• system adapter source remains read-only in hosted mode
• local source put remains rejected rather than pretending to update source
• local and hosted site-memory writes cannot escape their storage roots
• commands with ordinary input-only and output-only files still work

Evidence to attach

For every failure, record:

• WebCMD version/SHA
• Cloud deployment SHA
• exact command and mode
• exit code
• stdout/stderr
• workspace ID
• user/API-key label (never the key itself)
• Cloud request ID
• whether the package is system, marketplace, override, or private
• relevant local file paths and before/after hashes for artifact tests

Acceptance criteria

• Local override/status/path/reset workflow verified
• Hosted override behavior explicitly verified and documented
• Local adapter source behavior verified
• Hosted private adapter source get/put verified
• System adapter source remains read-only
• Site notes, endpoints, field maps, fixtures, and samples verified locally
• Same site-memory operations verified through Cloud
• browser init verified locally and hosted
• All browser verify flags verified locally and hosted
• Existing and missing mutable files verified locally and hosted
• Filename-based mutable output mapping verified
• Workspace isolation verified for memory, source, artifacts, profiles, and sessions
• Cross-user isolation verified with two API keys
• Full marketplace catalog and availability metadata verified
• Hosted mode ignores local adapters
• Local mode requires no Cloud auth
• Any intentional mode differences are documented with actionable CLI errors