Repository navigation
feat(e2e): let the e2e harness drive the warehouse flow - #1152
Merged
Merged
Conversation
A `ci` session disables the ask bridge, because no person is there to answer. The e2e harness is the exception: it runs a `ci` session, but its driver loop answers each batch from the program's profile. Without an escape hatch the agent-in-the-loop layer has no pre-merge test at all. `session.e2eAsk` re-enables the bridge. Only the e2e TUI host sets it, from the `E2E_ASK` env var. No CLI flag populates it, so plain `--ci` and `--signup` runs behave exactly as before. The flag also un-gates the orchestrator's seeded warehouse task, which `shouldDisableAsk` gates through the same policy.
…house profile
The profile could answer only `questions[0]` of a `wizard_ask` batch, and
the task-notice overlay had no action at all. Both stop a warehouse run
dead: credential questions arrive in batches, and the seeded warehouse
task opens a notice first.
- `answerQuestions` answers every question in the batch. Resolution order
per question: an `askAnswers` rule, else the first option, else the
`'e2e'` sentinel. A rule matches on the question id first, then on the
prompt. A `multi` question gets an array answer.
- `resolveE2eProfile` folds `E2E_NOTICE` and `${ENV_VAR}` values into the
profile at load. `decideE2eAction` therefore stays pure: it reads no
env and no store, so the flow-snapshot test does not depend on the
shell it runs in.
- `Overlay.TaskNotice` gets a `resolve_notice` action, a `taskNotice`
projection on `CiState`, and a `notice` profile policy. The projection
carries title, items and prompt only.
- The warehouse-source program declares its e2e profile, routing each
credential question to its own `E2E_*` env var.
The e2e-harness tests were excluded from vitest while the harness lived
on a stacked branch. It landed, so run them. That surfaces the gap the
task-notice action closes.
…yload `E2E_RESULT_JSON` reported the screens walked and the dependencies added. It said nothing about what the run was asked, what it answered, or what detection found — the facts a warehouse assertion needs. The payload gains `asks`, `unansweredAsks`, `notices`, `tasks`, `detectedSources`, `reportFile` and `abort`. Every pre-existing key is unchanged. Security rail: only question prompt text and question ids enter the file. No answer value ever does. The decision function reports ids and a keep/decline verdict, so the recorder never holds an answer. A test injects a secret through the real decision path and scans the serialized payload for it. The host also writes the payload from an `exit` hook. An abort exits from inside the runner, so an aborted run used to write nothing at all — and the abort reason is exactly what an assertion needs in that case.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
edwinyjlim
approved these changes
Aug 27, 2026
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The warehouse flow has no pre-merge test coverage of the agent-in-the-loop layer. Unit tests cover the pure logic. Nothing exercises the wizard binary, the downloaded skill, the model and the PostHog MCP together. Every recent warehouse regression lived in that layer.
The e2e harness is the only thing that can drive the real TUI and the real agent. It cannot drive the warehouse flow today, because it runs a
cisession and acisession disables every ask. The warehouse flow is made of asks.Changes
A harness-only escape hatch.
shouldDisableAskbecomes(ci || signup) && !e2eAsk.e2eAskis set only by the e2e host fromE2E_ASK. No CLI flag reaches it and published builds cannot set it. Plain--cibehaviour is unchanged. The same change un-gateswarehouseSeedTasks, which reads the identical predicate.Whole ask batches, and task notices. The profile answered only
questions[0]of a batch. Credential questions arrive in batches, so a warehouse run stopped at the first one.answerQuestionsnow answers every question. Resolution order per question: anaskAnswersrule, else the first option, else thee2esentinel.Overlay.TaskNoticehad no action at all, so the seeded warehouse task could never be accepted; it gainsresolve_notice, ataskNoticeprojection onCiState, and anoticeprofile policy.decideE2eActionstays pure.resolveE2eProfilefoldsE2E_NOTICEand${ENV_VAR}values into the profile once at load, so a decision never reads env or the store.A richer result payload.
E2E_RESULT_JSONreported screens walked and dependencies added. It said nothing about what the run was asked, what it answered, or what detection found. It gainsasks,unansweredAsks,notices,tasks,detectedSources,reportFileandabort. Every pre-existing key is unchanged.The host also writes the payload from an
exithook. An abort exits from inside the runner, so an aborted run wrote nothing at all — and the abort reason is exactly what an assertion needs in that case.Security rail. Only question prompt text and question ids enter the payload. No answer value ever does. The decision function reports ids and a keep or decline verdict, so the recorder never holds an answer.
One thing worth a second look.
vitest.config.tsexcluded**/e2e-harness/**with a comment saying those tests lived on a stacked branch. That branch landed, so the whole harness suite has not been running. This removes the exclusion. Doing so immediately surfaced the missing task-notice action, which is the gap an exhaustiveness test exists to catch.Test plan
tsc --noEmitoutput is byte-identical to a freshorigin/mainworktree: 36 pre-existing errors before and after.shouldDisableAsktruth table; the seeded warehouse task queuing underci + e2eAsk; batch answering, id-before-prompt precedence, case-insensitivity, malformed regex;${ENV}interpolation including unset-to-sentinel; notice keep and decline including theE2E_NOTICEoverride; every screen inE2E_DRIVABLE_SCREENSyielding an action; and the security rail, which answers a real secret through the decision path and then scans the serialized payload for it.Companion PRs: PostHog/wizard-workbench (warehouse fixtures, stub MCP and assertions) and the ref-pinning PRs in this repo and PostHog/context-mill.