feat(projectconfig): accept [tests] and [test-groups] schema in config by bhagyapathak · Pull Request #229 · microsoft/azure-linux-dev-tools

bhagyapathak · 2026-06-04T10:10:19Z

Adds a parse-only schema for declaring tests and reusable test groups in project TOML, plus matching per-component and per-image test reference lists.

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds support for a “new-shape” test configuration schema by introducing first-class project-level test definitions and groups, and enabling images/components to reference them.

Changes:

Introduces TestDefinition, TestGroup, TestRef, and ComponentTestsConfig types for the new schema.
Extends ConfigFile, ImageTestsConfig, and ComponentConfig to carry new test/group references.
Updates fingerprint decision test to exclude component test-selection metadata from build inputs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/projectconfig/tests.go	Adds the new schema types for tests, groups, and references.
internal/projectconfig/configfile.go	Adds top-level `[tests]` and `[test-groups]` maps to the config shape.
internal/projectconfig/image.go	Allows images to reference tests/groups via `ImageTestsConfig.Tests`.
internal/projectconfig/component.go	Allows components to reference tests/groups and deep-copies the new field.
internal/projectconfig/fingerprint_test.go	Excludes `ComponentConfig.Tests` from fingerprint build inputs.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

+func (TestRef) JSONSchemaExtend(schema *jsonschema.Schema) {
+	// Exactly one of name|group: encoded as oneOf with `required` on each and
+	// `not` excluding the other, which forbids both `{}` and `{name, group}`.
+	schema.OneOf = []*jsonschema.Schema{
+		{Required: []string{"name"}, Not: &jsonschema.Schema{Required: []string{"group"}}},
+		{Required: []string{"group"}, Not: &jsonschema.Schema{Required: []string{"name"}}},
+	}


liunan-ms

The new shape seems a superset of tmt, lisa, and pytest, other fields like required-capabilities, reusable groups, and per-component references. Is this intended as the next-generation replacement for test-suites? If yes, is there a migration path designed?

reubeno · 2026-06-24T15:33:12Z

Thanks for your patience -- I went through this in detail yesterday; lots of goodness here. I've created a branch based on the work that also adds some additional proposed changes. I'll be sending it out during my day today.

Some of my main comments/questions:

We need to unify the test-suites and test elements one way or the other.
There's some additional validation for the test references that I think will be helpful to catch typos/dangling references.
Can we better constrain the test kinds?
There are some opportunities for additional unit testing of some of the new code that I think will be worthwhile. Copilot can definitely help with some of it.
Nested test groups will bring some complexity; how do we imagine they'll be used?

I'll follow up with more details later today.

…g files

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

+        "kind": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "title": "Kind",
+          "description": "Test kind hints (e.g. functional or performance)"
+        },


Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

+	// Type identifies the framework/runner. Required, and constrained to the
+	// closed enum in the schema tag at the schema layer. The loader still
+	// accepts unknown values permissively; the resolver is the source of truth.
+	Type string `toml:"type" json:"type" jsonschema:"required,title=Type,description=Test framework type,enum=pytest,enum=lisa,enum=tmt"`


+// TestRef is a reference to either a test (by name) or another group (by name).
+// Exactly one of Name or Group should be set; semantic validation is the resolver's
+// responsibility.


+	if err := validateTestDefinitions(f.Tests); err != nil {
+		return err
+	}
+
+	if err := validateNewTestReferences(f); err != nil {
+		return err
+	}


bhagyapathak · 2026-07-02T15:33:35Z

The new shape seems a superset of tmt, lisa, and pytest, other fields like required-capabilities, reusable groups, and per-component references. Is this intended as the next-generation replacement for test-suites? If yes, is there a migration path designed?

Thank you @liunan-ms for pointing this out.
Yes, that is the intent.
The new tests/test-groups model is designed as the next-generation superset of test-suites: it keeps framework-specific configs (pytest/lisa/tmt) while adding capabilities like required-capabilities, reusable groups, and per-component/per-image references. I am working on migration plan, will update it in some time.

bhagyapathak · 2026-07-02T15:55:43Z

Thanks for your patience -- I went through this in detail yesterday; lots of goodness here. I've created a branch based on the work that also adds some additional proposed changes. I'll be sending it out during my day today.

Some of my main comments/questions:

We need to unify the test-suites and test elements one way or the other.

There's some additional validation for the test references that I think will be helpful to catch typos/dangling references.

Can we better constrain the test kinds?

There are some opportunities for additional unit testing of some of the new code that I think will be worthwhile. Copilot can definitely help with some of it.

Nested test groups will bring some complexity; how do we imagine they'll be used?

I'll follow up with more details later today.

Hi @reubeno
Thanks for the detailed feedback and for taking the time to review this so thoroughly.

For this iteration, I have already tightened a few areas:

Added load-time validation for test references to catch typos and dangling references earlier.
Added restrictions around nested test groups to avoid introducing unnecessary complexity right now.
Constrained test kinds to a tighter allowed set.

One important migration concern is ControlTower compatibility. Today, ControlTower consumes testSuites and derives TestSuiteDetails from that contract. If we remove those fields before ControlTower is updated, we risk missing test metadata and skipping validation task wiring.

To avoid breakage, I propose this rollout:

Make tests canonical internally only.
Continue accepting legacy test-suites input and normalize to canonical tests immediately after parse.
Continue emitting compatibility suite-shaped metadata projected from canonical tests until ControlTower migrates.
Migrate ControlTower to consume canonical tests and test-groups
Remove compatibility projection only after ControlTower rollout is complete.

I agree with your points on unifying test-suites and tests strengthening validation and expanding unit coverage. I will incorporate those as part of the migration work.

I am looking forward to your suggestions on the migration.
cc @liunan-ms For migration plan

Copilot AI review requested due to automatic review settings June 4, 2026 10:10

bhagyapathak changed the title ~~feat(projectconfig): accept [tests] and [test-groups] schema in confi…~~ feat(projectconfig): accept [tests] and [test-groups] schema in config Jun 4, 2026

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread internal/projectconfig/configfile.go Outdated

Comment thread internal/projectconfig/image.go

Comment thread internal/projectconfig/configfile.go Outdated

Copilot AI review requested due to automatic review settings June 4, 2026 16:03

Copilot AI reviewed Jun 4, 2026

View reviewed changes

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread internal/projectconfig/tests.go

Comment thread docs/user/reference/config/tests.md

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f0e30ee to b44cb81 Compare June 5, 2026 13:57

Copilot AI review requested due to automatic review settings June 5, 2026 15:37

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from b44cb81 to f7dbf0e Compare June 5, 2026 15:37

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread internal/projectconfig/image.go

Comment thread internal/projectconfig/configfile.go

Comment thread internal/projectconfig/tests.go Outdated

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f7dbf0e to c01da5d Compare June 8, 2026 11:17

Copilot AI review requested due to automatic review settings June 8, 2026 11:28

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from c01da5d to bd478bd Compare June 8, 2026 11:28

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread schemas/azldev.schema.json

Comment thread internal/projectconfig/component.go Outdated

Comment thread docs/user/reference/config/tests.md

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from bd478bd to fafc0c1 Compare June 8, 2026 11:41

Copilot AI review requested due to automatic review settings June 8, 2026 15:26

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from fafc0c1 to f1e6eb3 Compare June 8, 2026 15:26

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread scenario/__snapshots__/TestSnapshots_config_generate-schema_stdout_1.snap

Comment thread internal/projectconfig/tests.go

Comment thread internal/projectconfig/testsuite_test.go Outdated

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from f1e6eb3 to 86e4307 Compare June 8, 2026 16:34

Copilot AI review requested due to automatic review settings June 9, 2026 07:20

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 86e4307 to 9064645 Compare June 9, 2026 07:20

Copilot AI reviewed Jun 9, 2026

View reviewed changes

christopherco requested a review from liunan-ms June 18, 2026 05:27

liunan-ms reviewed Jun 18, 2026

View reviewed changes

Comment thread internal/projectconfig/tests.go

feat(projectconfig): accept [tests] and [test-groups] schema in confi…

8ab6361

…g files

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 9064645 to 8ab6361 Compare July 1, 2026 11:54

Copilot AI review requested due to automatic review settings July 2, 2026 14:31

Copilot started reviewing on behalf of bhagyapathak July 2, 2026 14:31 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Strengthen test metadata validation and reference integrity checks

c5fb905

bhagyapathak force-pushed the bhagya/azldev-test-schema branch from 7693b03 to c5fb905 Compare July 2, 2026 15:00

bhagyapathak requested a review from Copilot July 2, 2026 15:06

Copilot started reviewing on behalf of bhagyapathak July 2, 2026 15:06 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Uh oh!

Conversation

bhagyapathak commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liunan-ms left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reubeno commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

bhagyapathak commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bhagyapathak commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

bhagyapathak commented Jun 4, 2026 •

edited

Loading

bhagyapathak commented Jul 2, 2026 •

edited

Loading

bhagyapathak commented Jul 2, 2026 •

edited

Loading