Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 17 additions & 16 deletions docs/user/reference/cli/azldev_image_test.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions docs/user/reference/config/components.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ A component definition tells azldev where to find the spec file, how to customiz
| Render config | `render` | [RenderConfig](#render-configuration) | No | Options controlling spec rendering behavior |
| Source files | `source-files` | array of [SourceFileReference](#source-file-references) | No | Additional source files to download for this component |
| Package overrides | `packages` | map of string → [PackageConfig](package-groups.md#package-config) | No | Exact per-package configuration overrides; highest priority in the resolution order |
| Tests | `tests` | [ComponentTests](#component-tests) | No | Test references that apply to this component (see [Tests](tests.md)) |

### Bare Components

Expand Down Expand Up @@ -300,6 +301,24 @@ rpm-channel = "rpm-devel"
rpm-channel = "none"
```

## Component Tests

The `[components.<name>.tests]` subtable lists test or test-group references that apply to this component. Each entry is a [TestRef](tests.md#test-reference) with exactly one of `name` or `group` set.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Tests | `test-suites` | array of [TestRef](tests.md#test-reference) | No | References to `[test-suites.<name>]` entries or `[test-groups.<name>]` entries |

```toml
[components.kernel.tests]
test-suites = [
{ group = "kernel-bvt" },
{ name = "kdump-smoke" },
]
```

Test associations are metadata and never participate in fingerprints — adding, removing, or renaming a test reference does not invalidate a component's build.

## Source File References

The `[[components.<name>.source-files]]` array defines additional source files that azldev should download before building. These are files not available in the dist-git repository or lookaside cache — typically binaries, pre-built artifacts, or files from custom hosting.
Expand Down
3 changes: 2 additions & 1 deletion docs/user/reference/config/config-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ All config files share the same schema — there is no distinction between a "ro
| `components` | map of objects | Component (package) definitions | [Components](components.md) |
| `component-groups` | map of objects | Named groups of components with shared defaults | [Component Groups](component-groups.md) |
| `images` | map of objects | Image definitions (VMs, containers) | [Images](images.md) |
| `test-suites` | map of objects | Named test suite definitions referenced by images | [Test Suites](test-suites.md) |
| `test-suites` | map of objects | Named test definitions referenced by images and components | [Tests](tests.md) |
| `test-groups` | map of objects | Named bundles of test references | [Tests](tests.md#test-group) |
| `tools` | object | Configuration for external tools used by azldev | [Tools](tools.md) |
| `default-package-config` | object | Project-wide default applied to all binary packages | [Package Groups — Resolution Order](package-groups.md#resolution-order) |
| `package-groups` | map of objects | Named groups of binary packages with shared config | [Package Groups](package-groups.md) |
Expand Down
12 changes: 6 additions & 6 deletions docs/user/reference/config/images.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ The `capabilities` subtable describes what the image supports. All fields are op

## Image Tests

The `tests` subtable links an image to one or more test suites defined in the top-level [`[test-suites]`](test-suites.md) section.
The `tests` subtable links an image to tests and test-groups defined in the top-level [`[test-suites]` / `[test-groups]`](tests.md) sections. Each entry is a [TestRef](tests.md#test-reference) with exactly one of `name` or `group` set.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Test Suites | `test-suites` | array of inline tables | No | List of test suite references. Each entry must have a `name` field matching a key in `[test-suites]`. |
| Tests | `test-suites` | array of [TestRef](tests.md#test-reference) | No | References to `[test-suites.<name>]` entries or `[test-groups.<name>]` entries |

## Image Publish

Expand Down Expand Up @@ -85,7 +85,7 @@ description = "Azure-optimized VM image"
definition = { type = "kiwi", path = "vm-azure/vm-azure.kiwi", profile = "azure" }
```

### Image with test suite references
### Image with test references

```toml
[images.vm-base]
Expand All @@ -98,8 +98,8 @@ systemd = true

[images.vm-base.tests]
test-suites = [
{ name = "smoke" },
{ name = "integration" },
{ group = "smoke" },
{ name = "integration-boot" },
]
```

Expand All @@ -117,5 +117,5 @@ channels = ["registry-prod", "registry-staging"]
## Related Resources

- [Config File Structure](config-file.md) — top-level config file layout
- [Test Suites](test-suites.md) — test suite definitions
- [Tests](tests.md) — test and test-group definitions referenced by `[images.<name>.tests]`
- [Tools](tools.md) — Image Customizer tool configuration
133 changes: 0 additions & 133 deletions docs/user/reference/config/test-suites.md

This file was deleted.

112 changes: 112 additions & 0 deletions docs/user/reference/config/tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Tests and Test Groups

The `[test-suites]` section defines named, framework-specific test configurations that components and images can target by name. The `[test-groups]` section bundles those tests under a single name so callers can reference a curated set without enumerating every member.

A `TestRef` is the structured reference used by image and component `tests` subtables. Each ref points at exactly one of a `[test-suites.<name>]` entry or a `[test-groups.<name>]` entry.

> Test references on images and components are metadata and do **not** participate in build fingerprints — adding or removing a reference never invalidates a component's build.

## Test Config

Each entry under `[test-suites.<name>]` describes one configuration of one runner. Framework-specific options live in a typed subtable (`pytest`) whose schema is validated by azldev. `lisa` tests are accepted as a type but have no subtable — they are metadata for external orchestration and are not run by `azldev image test`.

Test names must be simple identifiers (no path separators, traversal segments, or whitespace) since they may be used as path components (for example, each pytest test gets its own Python virtual environment under the project work directory).

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Type | `type` | string | **Yes** | Test framework: `pytest` or `lisa` |
| Description | `description` | string | No | Human-readable description |
| Kind | `kind` | string array | No | Closed enum of classification hints: `functional`, `performance`. May be multi-valued. |
| Long running | `long-running` | bool | No | Hints that this test may run for hours. Declarative cost hint, not a configurable timeout. |
| Pytest | `pytest` | [PytestConfig](#pytest-config) | When `type = "pytest"` | pytest-specific configuration |

The framework subtable must match the declared `type`: `type = "pytest"` requires a `pytest` subtable; `type = "lisa"` forbids it.

> **Note:** Each test name must be unique across all config files. Defining the same test name in two files produces an error.

### Pytest Config

When `type = "pytest"`, a `[test-suites.<name>.pytest]` subtable must be provided. `azldev` runs the test by creating (or reusing) a Python virtual environment, optionally installing dependencies, and invoking `python -m pytest` with the configured arguments.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Working directory | `working-dir` | string | No | Directory used as pytest's CWD. Relative paths are resolved against the config file's directory. Required when `install` is `pyproject` or `requirements`. |
| Test paths | `test-paths` | array of strings | No | Test file paths or directories passed to pytest as positional arguments. Each entry is glob-expanded (including recursive `**`) relative to `working-dir`. Patterns that match nothing are passed through so pytest reports the failure. |
| Extra args | `extra-args` | array of strings | No | Additional arguments passed to pytest verbatim, after placeholder substitution. Use `{image-path}` as a placeholder for the image path. |
| Install mode | `install` | string | No | One of `pyproject`, `requirements`, or `none` (default). |

## Test Group

Each entry under `[test-groups.<name>]` names an ordered list of test references that callers can target as a single unit. Group membership is one layer of indirection: groups list **test names only**, never other groups.

| Field | TOML Key | Type | Required | Description |
|-------|----------|------|----------|-------------|
| Description | `description` | string | No | Human-readable description |
| Tests | `tests` | string array | No | Ordered list of test names (keys in `[test-suites]`) that belong to this group |

Every name in `tests` must resolve to a defined `[test-suites.<name>]` entry; nested group references are not supported.

> **Note:** Each test-group name must be unique across all config files. Defining the same group name in two files produces an error.

## Test Reference

`TestRef` is an inline table used by `[images.<name>.tests.test-suites]` and `[components.<name>.tests.test-suites]`. Exactly one of `name` or `group` must be set:

| Field | TOML Key | Type | Description |
|-------|----------|------|-------------|
| Name | `name` | string | References a `[test-suites.<name>]` entry |
| Group | `group` | string | References a `[test-groups.<name>]` entry |

References must resolve at project load time — an unknown test or group is a config error.

## Example

```toml
[test-suites.bvt-ssh]
type = "pytest"
description = "Basic SSH boot verification"
kind = ["functional"]

[test-suites.bvt-ssh.pytest]
working-dir = "tests/bvt"
test-paths = ["test_ssh.py"]
extra-args = ["--image-path", "{image-path}"]
install = "pyproject"

[test-suites.kdump-smoke]
type = "pytest"
description = "Smoke test for kdump"

[test-suites.kdump-smoke.pytest]
working-dir = "tests/kdump"
test-paths = ["test_kdump.py"]

[test-suites.kdump-lisa]
type = "lisa"
description = "Kdump scenarios driven by external LISA orchestration"
long-running = true

[test-groups.bvt]
description = "Build verification tests"
tests = ["bvt-ssh", "kdump-smoke"]

[images.vm-base.tests]
test-suites = [{ group = "bvt" }]

[components.kernel.tests]
test-suites = [
{ name = "kdump-smoke" },
{ name = "kdump-lisa" },
]
```

## Running tests

Use [`azldev image test IMAGE_NAME`](../cli/azldev_image_test.md) to run all `pytest` tests associated with an image (directly or via test-groups). Use `--test <name>` to run a single named test. `lisa` tests are recognized by the schema but are not currently executed by `azldev image test`.

## Related Resources

- [Components](components.md#component-tests) — per-component `test-suites` references
- [Images](images.md#image-tests) — per-image `test-suites` references
- [Config File Structure](config-file.md) — top-level config layout
- [`azldev image test`](../cli/azldev_image_test.md) — CLI reference for running tests against an image
Loading
Loading