[evals] Adding in some vally testing for some basic azd scenarios - #9282
Conversation
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt). I've reduced this down to two buckets: - eval-azd-troubleshooting.yaml, which is a series of prompts that just see what baseline copilot models can provide without any skills or extra materials. It's there mostly just to keep tabs on things and see if the advice changes in an interesting way, or if we start getting bad ad-hoc advice in chat sessions. - eval-azd-deploy.yaml - this is a suite I plan on expanding - right now I've got the node app test uncommented, and I plan on slowly introducing the others once we get this bootstrappable/trackable in some system (TBD, since there's a lot of moving parts/in-flight efforts here).
…s for some common scenarios - troubleshooting (ie, no azd around, but what does the LLM say?) and deploy, which will actually have some azd knowledge and is being asked - how do I deploy this bare-bones app, and other similar questions.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 21 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Replaces the legacy Waza evaluation framework with Vally-based azd scenarios, experiments, reporting, fixtures, and CI integration.
Changes:
- Adds Vally troubleshooting, deployment, and skill-baseline evaluations.
- Adds fixtures and a Markdown report generator.
- Removes legacy Waza tests, graders, and pipelines.
Show a summary per file
| File | Description |
|---|---|
eng/pipelines/eval-waza.yml |
Removes scheduled Waza evaluations. |
eng/pipelines/eval-unit.yml |
Removes legacy ADO unit-test pipeline. |
eng/pipelines/eval-report.yml |
Removes Waza report pipeline. |
eng/pipelines/eval-e2e.yml |
Removes Waza E2E pipeline. |
cli/azd/test/evals/skills/azd/SKILL.md |
Adds evaluation-only azd guidance. |
cli/azd/test/evals/README.md |
Documents Vally suites and commands. |
cli/azd/test/evals/package.json |
Adds suite and reporting scripts. |
cli/azd/test/evals/make-vally-report.go |
Generates Markdown evaluation reports. |
cli/azd/test/evals/fixtures/existing-node-api/package.json |
Defines the Node API fixture. |
cli/azd/test/evals/fixtures/existing-node-api/index.js |
Implements the fixture API. |
cli/azd/test/evals/fixtures/azd-project/staging.env |
Adds staging fixture values. |
cli/azd/test/evals/fixtures/azd-project/production.env |
Adds production fixture values. |
cli/azd/test/evals/fixtures/azd-project/dev.env |
Adds development fixture values. |
cli/azd/test/evals/fixtures/azd-project/config-production.json |
Selects the production fixture environment. |
cli/azd/test/evals/fixtures/azd-project/config-dev.json |
Selects the development fixture environment. |
cli/azd/test/evals/fixtures/azd-project/azure.yaml |
Defines the sample azd project. |
cli/azd/test/evals/eval-azd.experiments.yaml |
Compares skill and model variants. |
cli/azd/test/evals/eval-azd-troubleshooting.yaml |
Adds baseline troubleshooting evaluations. |
cli/azd/test/evals/eval-azd-deploy.yaml |
Adds deployment and lifecycle evaluations. |
cli/azd/test/evals/.gitignore |
Ignores Vally outputs and reports. |
cli/azd/test/eval/tsconfig.json |
Removes legacy TypeScript configuration. |
cli/azd/test/eval/tests/unit/help-text-quality.test.ts |
Removes Waza-era help tests. |
cli/azd/test/eval/tests/unit/flag-validation.test.ts |
Removes legacy flag tests. |
cli/azd/test/eval/tests/unit/command-sequencing.test.ts |
Removes legacy sequencing tests. |
cli/azd/test/eval/tests/unit/command-registry.test.ts |
Removes legacy registry tests. |
cli/azd/test/eval/tests/test-utils.ts |
Removes the old CLI test helper. |
cli/azd/test/eval/tests/human/error-recovery.test.ts |
Removes human error-recovery tests. |
cli/azd/test/eval/tests/human/command-discovery.test.ts |
Removes command-discovery tests. |
cli/azd/test/eval/tests/human/cli-workflow.test.ts |
Removes CLI workflow tests. |
cli/azd/test/eval/tasks/troubleshoot/quota-error.yaml |
Removes the Waza quota scenario. |
cli/azd/test/eval/tasks/troubleshoot/provision-role-conflict.yaml |
Removes the role-conflict scenario. |
cli/azd/test/eval/tasks/troubleshoot/config-error.yaml |
Removes the Waza config scenario. |
cli/azd/test/eval/tasks/troubleshoot/auth-error.yaml |
Removes the Waza authentication scenario. |
cli/azd/test/eval/tasks/negative/raw-azure-cli.yaml |
Removes the raw Azure CLI scenario. |
cli/azd/test/eval/tasks/negative/not-azure.yaml |
Removes the non-Azure scenario. |
cli/azd/test/eval/tasks/negative/general-coding.yaml |
Removes the general coding scenario. |
cli/azd/test/eval/tasks/lifecycle/teardown-only.yaml |
Removes the Waza teardown scenario. |
cli/azd/test/eval/tasks/lifecycle/full-lifecycle.yaml |
Removes the full lifecycle scenario. |
cli/azd/test/eval/tasks/environment/switch-env.yaml |
Removes the environment-switch scenario. |
cli/azd/test/eval/tasks/environment/delete-env.yaml |
Removes the environment-delete scenario. |
cli/azd/test/eval/tasks/environment/create-staging.yaml |
Removes the staging scenario. |
cli/azd/test/eval/tasks/deploy/deploy-python-webapp.yaml |
Removes the Python deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-node-api.yaml |
Removes the Node deployment scenario. |
cli/azd/test/eval/tasks/deploy/deploy-existing-project.yaml |
Removes the existing-project scenario. |
cli/azd/test/eval/reports/.gitkeep |
Removes the legacy report placeholder. |
cli/azd/test/eval/README.md |
Removes Waza framework documentation. |
cli/azd/test/eval/package.json |
Removes Waza/Jest dependencies and scripts. |
cli/azd/test/eval/jest.config.ts |
Removes Jest configuration. |
cli/azd/test/eval/graders/test_graders.py |
Removes Python grader tests. |
cli/azd/test/eval/graders/infra_validator.py |
Removes infrastructure grader. |
cli/azd/test/eval/graders/cleanup_validator.py |
Removes cleanup grader. |
cli/azd/test/eval/graders/azure_auth.py |
Removes grader authentication helper. |
cli/azd/test/eval/graders/app_health.py |
Removes application-health grader. |
cli/azd/test/eval/eval.yaml |
Removes the Waza configuration. |
cli/azd/test/eval/.gitignore |
Removes legacy ignore rules. |
cli/azd/.vscode/cspell.yaml |
Removes obsolete Waza spelling overrides. |
.github/workflows/vally-ci.yml |
Splits Vally CI into two jobs. |
.github/workflows/eval-unit.yml |
Removes legacy GitHub unit workflow. |
Review details
Comments suppressed due to low confidence (1)
cli/azd/test/evals/make-vally-report.go:82
- azd-code-reviewer: This unexported field is initialized but never read. Staticcheck U1000 reports unused struct fields, so preflight will fail until both this field and its initializer are removed.
- Files reviewed: 57/59 changed files
- Comments generated: 5
- Review effort level: Medium
Co-authored-by: richardpark-msft <51494936+richardpark-msft@users.noreply.github.com>
… closer to what it's actually doing.
…re-dev into azd-02-waza-port-evals
…t`, which actually can't create an environment - Use a prompt grader for fuzzy situations (these commands were listed in this order-ish) vs "hey, was the one piece of text mentioned?" which can be an output-matches. - Update the report so it reports _which_ checks failed in a failing test.
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:48
- azd-code-reviewer: The PR description says only the Node app eval is enabled initially, but this second stimulus and all following stimuli are active. Because the workflow runs this file with
runs: 3, the deploy job executes 24 trials rather than the described 3. Either keep the planned gradual rollout by disabling the additional stimuli or update the PR description to reflect the expanded CI scope.
- Files reviewed: 57/60 changed files
- Comments generated: 4
- Review effort level: Medium
…nicode characters on incorrect byte boundaries.
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only
deploy-node-apiis enabled and the other deploy scenarios will be introduced later, but this file activates all eight scenarios at three trials each, and the workflow runs the entire file. That materially expands CI cost and flakiness beyond the stated rollout. Either disable the remaining stimuli or update the PR description if enabling all of them is intentional.
- Files reviewed: 57/60 changed files
- Comments generated: 3
- Review effort level: Medium
…t accidentally flag answers that might mention the 'bad" thing (azd init) in a "well, don't do that" type of answer.
… hopefully do a better job (has a limited rubric, so should be okay)
- Run our vally_report.go after each run, so we get a nice condensed report. It also prints out how it was generated into the report so it's clear it's not a vally thing, just our custom script - Also, made some nice little enhancements - we pick up the skill name when we summarize the tools in our report, etc.... - Added in actual tests for the vally_report.go file - Switch over to more prompt-based grading. A lot of the "hey, what would you do if 'x'" tests can have perfectly valid answers that also include the "don't do this" stuff. Using a prompt is really the only way to go.
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (2)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only the Node app scenario is enabled while the rest will be introduced gradually, but this file activates all eight stimuli with three runs each and the workflow now executes this suite on every matching PR. That changes the intended rollout from 3 to 24 deploy-suite trials. Disable the later stimuli or update the PR description to reflect the expanded CI scope.
cli/azd/test/evals/skills/azd/SKILL.md:27 - azd-code-reviewer:
todo-python-mongo-acacurrently contains a FastAPI API, not Flask. Keeping the framework metadata accurate matters because this file is injected directly into the model's deployment guidance.
- Files reviewed: 62/65 changed files
- Comments generated: 2
- Review effort level: Medium
jongio
left a comment
There was a problem hiding this comment.
Incremental pass on 5c6e092: the toWorkspaceRelative removal is clean. No callers remain anywhere in the module, the os, strings, and filepath imports are all still needed by other functions, and build, vet, and tests are green on cli/azd/test/evals.
While checking the two template problems already flagged in skills/azd/SKILL.md, I resolved every entry in that list against each repo's azure.yaml. Two more are wrong, so this is worth one sweep of the list rather than two line edits:
- line 16:
todo-nodejs-mongois App Service, not Container Apps. - line 19:
app-service-javascript-sap-cap-quickstarthas noazure.yamlat its root.
For the 404 entry on line 29, Azure-Samples/azd-simple-fastapi-appservice is a FastAPI template that exists today, though it's App Service rather than serverless. Azure-Samples/todo-python-mongo-swa-func is the Python Functions one if you want to keep the serverless label.
That puts 4 of the 8 entries wrong. None of it blocks the harness, but the file is injected into every eval-azd-deploy run, so wrong template facts get baked into the baseline you're measuring against.
jongio
left a comment
There was a problem hiding this comment.
One more template label mismatch in the injected skill, same class as the todo-python-mongo line above.
TBH, they're not the best collection of templates, but I'm just trying to get something minimal. The JS ones are definitely lightning focused on specific technologies, like SAP, which isn't as general as I'd like.
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (3)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only the Node app scenario is enabled and the others will be introduced gradually, but this file enables all eight stimuli with three runs each, and the new deploy CI job executes the whole file. That materially changes model-call volume and timeout/flakiness exposure. Disable the other stimuli as described, or update the PR description if running all 24 trials is intentional.
cli/azd/test/evals/eval-azd-deploy.yaml:23 - azd-code-reviewer: With two equally weighted binary graders, a threshold of
0.5lets a trial pass when only the regex grader succeeds and the semantic sequence/safety judge fails. Most stimuli below have exactly that shape, so their rubric guardrails do not gate CI. Use a threshold above 0.5 (or explicit weights) if judge failures should remain regressions.
cli/azd/test/evals/eval-azd-deploy.yaml:141 - azd-code-reviewer: This excludes the canonical
azd init→azd up→azd downflow even thoughazd upis the single-command provision-and-deploy operation (cli/azd/cmd/up.go:63). The judge can therefore reject correct lifecycle guidance. Includeazd upas an alternative to the separate provision/deploy steps.
- Files reviewed: 62/65 changed files
- Comments generated: 3
- Review effort level: Medium
jongio
left a comment
There was a problem hiding this comment.
One non-blocking thing inline about lint coverage on the new evals module. Everything else I'd flagged earlier lines up with the live repos now.
… _just_ that they should run azd up, but also that they've initialized an app.
jongio
left a comment
There was a problem hiding this comment.
One low-severity note on the composite action, inline.
.github/actions/vally-eval/action.yml:go-version-fileis pinned to a literal path whileworking-directoryis an input.
…e the github token, for instance. - Simplify a silly bit of the action.yml, where we were making paths that were relative and then downstream steps had to deal with that.
…ght spot to locate go.mod - Trim evals a bit
There was a problem hiding this comment.
Review details
Files not reviewed (1)
- cli/azd/test/evals/package-lock.json: Generated file
Comments suppressed due to low confidence (1)
cli/azd/test/evals/eval-azd-deploy.yaml:2
- azd-code-reviewer: The PR description says only the Node app eval is enabled and the remaining deploy scenarios will be introduced later, but this file enables all eight stimuli with three runs each, and the new
deployworkflow job executes the whole file. Either disable the seven deferred stimuli or update the PR description to reflect the substantially larger CI scope and model usage.
- Files reviewed: 62/65 changed files
- Comments generated: 0 new
- Review effort level: Medium
jongio
left a comment
There was a problem hiding this comment.
Three new commits since my last pass (68924fd -> 614f294). Verified the report step split works end to end: the qna-vally-results and deploy-vally-results artifacts from run 30320139483 both contain vally_report/ with generated markdown and relative links that resolve.
The azd env remove grader added in d7803cc fails every trial on this HEAD, and the job reported success anyway. That's the one worth a look before this merges.
environment-delete-envscored 0.333 on all 3 trials, verdict 0/3, while "Run vally evals (deploy)" stayed green. The gate only reads vally's exit code, and vally applies its threshold to the eval-wide average.skills/azd/SKILL.mdnever mentionsazd env remove, so that grader has nothing to match against.vally_report.gorenders "trial 1/0" for any eval withruns: 1, which is the whole Q&A suite.eval-azd-qna.yamllost both judge graders, leaving two regex graders as the only signal.
jongio
left a comment
There was a problem hiding this comment.
Verified at 614f294. No code changed since my last pass, so the items from that review are still open. None of them block: the deploy suite is explicitly tuned to tolerate partial failure, and the rest are report cosmetics and dead config.
Re-checked on this head: go vet ./... is clean and go test ./... passes in the evals module.
Still open from my inline comments, all non-blocking:
.github/actions/vally-eval/action.ymlgates on vally's exit code, so a stimulus that fails every trial still reports the job green.cli/azd/test/evals/skills/azd/SKILL.mdhas noazd env remove, which is what pinsenvironment-delete-envto 0.333.cli/azd/test/evals/vally_report.gorenderstrial 1/0forruns: 1evals.cli/azd/test/evals/eval-azd-qna.yamlstill declaresjudge_modelwith no judge grader left to use it.
jongio
left a comment
There was a problem hiding this comment.
One new item on this pass, on the experiments matrix. The rest of my notes from 614f294 are unchanged and still don't block.
eval-azd.experiments.yaml: the model axis can select the same model that's pinned asjudge_model, so one cell ends up grading itself.
jongio
left a comment
There was a problem hiding this comment.
Two things on top of what's already open, neither of them blocking.
cli/azd/test/evals/package-lock.jsondropped thelibcfields on the four@github/copilot-*Linux packages, which leaves the glibc and musl variants indistinguishable to npm.cli/azd/test/evals/testdata/results.jsonlalways setstotalTrials, so nothing in the Go tests reproduces the record shape vally emits forruns: 1.
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
Translating over some of the waza evals into vally evals.
Some evals/tests just didn't make the cut - either they were redundant with our normal unit tests (there were several that were checking help content), or they were evals that seemed to be eval'ing themselves (ie, the answer was encoded in the system prompt).
I've reduced this down to two buckets:
here).
I've tuned down the threshold for some of the flakier evals, just to give us a good reason to investigate if this job starts failing.
There's also an experiments file in there, which isn't yet running in CI, that can be useful for some manual testing as we refine things a bit.
Fixes #9199 (by adding in an experiments file)
Fixes #9186 (by porting waza evals)