Skip to content

Commit eba9c34

Browse files
atlowChemiaduh95
authored andcommitted
test_runner: add tags option and tag-name filter
Adds a `tags` option to test(), it(), suite(), and describe() that accepts an array of string labels. Tags are canonicalized to lowercase and inherited from suites to nested tests by union. Reporter events expose the tag set on every test, and TestContext exposes the test's tags via `context.tags`. Filtering is done by literal tag name through the new `--experimental-test-tag-filter=<tag>` flag (or `testTagFilters` on run()). The flag may be specified more than once; tests must contain every filter to run. Untagged tests are excluded under any positive filter. The tagging mechanism is gated behind a one-shot ExperimentalWarning. Signed-off-by: atlowChemi <chemi@atlow.co.il> PR-URL: #63221 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Pietro Marchini <pietro.marchini94@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh>
1 parent bdc5713 commit eba9c34

17 files changed

Lines changed: 977 additions & 16 deletions

doc/api/cli.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1412,6 +1412,23 @@ Enable module mocking in the test runner.
14121412

14131413
This feature requires `--allow-worker` if used with the [Permission Model][].
14141414

1415+
### `--experimental-test-tag-filter=<tag>`
1416+
1417+
<!-- YAML
1418+
added: REPLACEME
1419+
-->
1420+
1421+
> Stability: 1.0 - Early development
1422+
1423+
Run only tests whose tag set contains `<tag>`. Tests declare tags via the
1424+
`tags` option on `test()`, `it()`, `suite()`, or `describe()`; tags
1425+
inherit from suites to nested tests by union. Filtering is
1426+
case-insensitive.
1427+
1428+
The flag may be specified more than once; tests must contain **every**
1429+
filter value to run. See [Test tags][] for details on declaring and
1430+
inheriting tags.
1431+
14151432
### `--experimental-vm-modules`
14161433

14171434
<!-- YAML
@@ -4337,6 +4354,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
43374354
[ScriptCoverage]: https://chromedevtools.github.io/devtools-protocol/tot/Profiler#type-ScriptCoverage
43384355
[ShadowRealm]: https://github.com/tc39/proposal-shadowrealm
43394356
[Source Map]: https://tc39.es/ecma426/
4357+
[Test tags]: test.md#test-tags
43404358
[TypeScript type-stripping]: typescript.md#type-stripping
43414359
[V8 Inspector integration for Node.js]: debugger.md#v8-inspector-integration-for-nodejs
43424360
[V8 JavaScript code coverage]: https://v8project.blogspot.com/2017/12/javascript-code-coverage.html

doc/api/test.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -479,6 +479,82 @@ Test name patterns do not change the set of files that the test runner executes.
479479
If both `--test-name-pattern` and `--test-skip-pattern` are supplied,
480480
tests must satisfy **both** requirements in order to be executed.
481481

482+
## Test tags
483+
484+
<!-- YAML
485+
added: REPLACEME
486+
-->
487+
488+
> Stability: 1.0 - Early development
489+
490+
Tags annotate tests and suites with arbitrary string labels. The
491+
[`--experimental-test-tag-filter`][] CLI flag (or the `testTagFilters`
492+
option on [`run()`][]) selects tests whose tag set contains every
493+
provided filter value.
494+
495+
Tags are an alternative to encoding metadata into test names. They are
496+
useful for cross-cutting axes such as subsystem, speed bucket, flakiness,
497+
or environment, where a name pattern would be brittle.
498+
499+
### Authoring tagged tests
500+
501+
Pass a `tags` array on any of `test()`, `it()`, `suite()`, or `describe()`.
502+
Tags inherit from a suite to its child tests by union—a test inside a
503+
suite tagged `['db']` that declares its own `tags: ['integration']`
504+
effectively has both tags.
505+
506+
```mjs
507+
import { describe, it } from 'node:test';
508+
509+
describe('database', { tags: ['db'] }, () => {
510+
it('reads a row'); // tags: ['db']
511+
it('writes a row', { tags: ['integration'] }); // tags: ['db', 'integration']
512+
it('reconnects after disconnect', { tags: ['flaky'] }); // tags: ['db', 'flaky']
513+
});
514+
```
515+
516+
```cjs
517+
const { describe, it } = require('node:test');
518+
519+
describe('database', { tags: ['db'] }, () => {
520+
it('reads a row'); // tags: ['db']
521+
it('writes a row', { tags: ['integration'] }); // tags: ['db', 'integration']
522+
it('reconnects after disconnect', { tags: ['flaky'] }); // tags: ['db', 'flaky']
523+
});
524+
```
525+
526+
Tag values must be non-empty strings. Tags are matched case-insensitively;
527+
the canonical form is lowercase. Duplicates within a single `tags` array
528+
are collapsed on the lowercased form, preserving the first-seen
529+
declaration order.
530+
531+
Hooks (`before`, `after`, `beforeEach`, `afterEach`) do not declare their
532+
own tags. They run as part of their owning suite, which carries the
533+
suite's tags.
534+
535+
### Filtering by tag
536+
537+
Each [`--experimental-test-tag-filter`][] value is a literal tag name. A
538+
test runs only when its tag set contains that name. The flag may be
539+
specified more than once; tests must match **every** filter to run. The
540+
same applies to the `testTagFilters` array on [`run()`][]. Filters are
541+
case-insensitive and AND'd with [`--test-name-pattern`][],
542+
[`--test-skip-pattern`][], and `.only` filtering.
543+
544+
Untagged tests are excluded under any non-empty filter, since the filter
545+
requires the tag to be present.
546+
547+
### Reading tags from inside a test
548+
549+
The [`TestContext`][] object exposes the test's tags as a frozen array
550+
through [`context.tags`][], so tests can branch on their own metadata.
551+
552+
### Errors
553+
554+
A tag value that violates the validation rules above throws
555+
`ERR_INVALID_ARG_VALUE` at the registration site, before any test runs.
556+
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`.
557+
482558
## Extraneous asynchronous activity
483559

484560
Once a test function finishes executing, the results are reported as quickly
@@ -750,6 +826,8 @@ test runner functionality:
750826

751827
* `--test` - Prevented to avoid recursive test execution
752828
* `--experimental-test-coverage` - Managed by the test runner
829+
* `--experimental-test-tag-filter` - Filter values are validated by the parent
830+
process and re-emitted to child processes
753831
* `--watch` - Watch mode is handled at the parent level
754832
* `--experimental-default-config-file` - Config file loading is handled by the parent
755833
* `--test-reporter` - Reporting is managed by the parent process
@@ -1568,6 +1646,9 @@ added:
15681646
- v18.9.0
15691647
- v16.19.0
15701648
changes:
1649+
- version: REPLACEME
1650+
pr-url: https://github.com/nodejs/node/pull/63221
1651+
description: Added the `testTagFilters` option.
15711652
- version:
15721653
- v25.6.0
15731654
- v24.14.0
@@ -1656,6 +1737,10 @@ changes:
16561737
For each test that is executed, any corresponding test hooks, such as
16571738
`beforeEach()`, are also run.
16581739
**Default:** `undefined`.
1740+
* `testTagFilters` {string|string\[]} A tag name, or an array of tag names,
1741+
used to filter tests by their declared tags. Tests must contain every
1742+
listed tag to run. Equivalent to passing [`--experimental-test-tag-filter`][]
1743+
on the command line. See [Test tags][]. **Default:** `undefined`.
16591744
* `timeout` {number} A number of milliseconds the test execution will
16601745
fail after.
16611746
If unspecified, subtests inherit this value from their parent.
@@ -1799,6 +1884,9 @@ added:
17991884
- v18.0.0
18001885
- v16.17.0
18011886
changes:
1887+
- version: REPLACEME
1888+
pr-url: https://github.com/nodejs/node/pull/63221
1889+
description: Added the `tags` option.
18021890
- version:
18031891
- v20.2.0
18041892
- v18.17.0
@@ -1842,6 +1930,10 @@ changes:
18421930
* `skip` {boolean|string} If truthy, the test is skipped. If a string is
18431931
provided, that string is displayed in the test results as the reason for
18441932
skipping the test. **Default:** `false`.
1933+
* `tags` {string\[]} An array of string labels associated with the test.
1934+
Used together with [`--experimental-test-tag-filter`][] to filter which
1935+
tests run. Tags inherit from suites to nested tests by union. See
1936+
[Test tags][]. **Default:** `[]`.
18451937
* `todo` {boolean|string} If truthy, the test marked as `TODO`. If a string
18461938
is provided, that string is displayed in the test results as the reason why
18471939
the test is `TODO`. **Default:** `false`.
@@ -3430,6 +3522,9 @@ Emitted when code coverage is enabled and all tests have completed.
34303522
`undefined` if the test was run through the REPL.
34313523
* `name` {string} The test name.
34323524
* `nesting` {number} The nesting level of the test.
3525+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3526+
and its ancestor suites, in declaration order. Empty for untagged tests.
3527+
See [Test tags][].
34333528
* `testId` {number} A numeric identifier for this test instance, unique
34343529
within the test file's process. Consistent across all events for the same
34353530
test instance, enabling reliable correlation in custom reporters.
@@ -3453,6 +3548,9 @@ The corresponding declaration ordered events are `'test:pass'` and `'test:fail'`
34533548
`undefined` if the test was run through the REPL.
34543549
* `name` {string} The test name.
34553550
* `nesting` {number} The nesting level of the test.
3551+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3552+
and its ancestor suites, in declaration order. Empty for untagged tests.
3553+
See [Test tags][].
34563554
* `testId` {number} A numeric identifier for this test instance, unique
34573555
within the test file's process. Consistent across all events for the same
34583556
test instance, enabling reliable correlation in custom reporters.
@@ -3494,6 +3592,9 @@ defined.
34943592
`undefined` if the test was run through the REPL.
34953593
* `name` {string} The test name.
34963594
* `nesting` {number} The nesting level of the test.
3595+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3596+
and its ancestor suites, in declaration order. Empty for untagged tests.
3597+
See [Test tags][].
34973598
* `testId` {number} A numeric identifier for this test instance, unique
34983599
within the test file's process. Consistent across all events for the same
34993600
test instance, enabling reliable correlation in custom reporters.
@@ -3520,6 +3621,9 @@ Emitted when a test is enqueued for execution.
35203621
`undefined` if the test was run through the REPL.
35213622
* `name` {string} The test name.
35223623
* `nesting` {number} The nesting level of the test.
3624+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3625+
and its ancestor suites, in declaration order. Empty for untagged tests.
3626+
See [Test tags][].
35233627
* `testId` {number} A numeric identifier for this test instance, unique
35243628
within the test file's process. Consistent across all events for the same
35253629
test instance, enabling reliable correlation in custom reporters.
@@ -3577,6 +3681,9 @@ since the parent runner only knows about file-level tests. When using
35773681
`undefined` if the test was run through the REPL.
35783682
* `name` {string} The test name.
35793683
* `nesting` {number} The nesting level of the test.
3684+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3685+
and its ancestor suites, in declaration order. Empty for untagged tests.
3686+
See [Test tags][].
35803687
* `testId` {number} A numeric identifier for this test instance, unique
35813688
within the test file's process. Consistent across all events for the same
35823689
test instance, enabling reliable correlation in custom reporters.
@@ -3616,6 +3723,9 @@ defined.
36163723
`undefined` if the test was run through the REPL.
36173724
* `name` {string} The test name.
36183725
* `nesting` {number} The nesting level of the test.
3726+
* `tags` {string\[]} The flattened lowercased tags declared on the test
3727+
and its ancestor suites, in declaration order. Empty for untagged tests.
3728+
See [Test tags][].
36193729
* `testId` {number} A numeric identifier for this test instance, unique
36203730
within the test file's process. Consistent across all events for the same
36213731
test instance, enabling reliable correlation in custom reporters.
@@ -4119,6 +4229,20 @@ The attempt number of the test. This value is zero-based, so the first attempt i
41194229
the second attempt is `1`, and so on. This property is useful in conjunction with the
41204230
`--test-rerun-failures` option to determine which attempt the test is currently running.
41214231

4232+
### `context.tags`
4233+
4234+
<!-- YAML
4235+
added: REPLACEME
4236+
-->
4237+
4238+
> Stability: 1.0 - Early development
4239+
4240+
* Type: {string\[]}
4241+
4242+
A frozen array of the test's flattened lowercased tags, in declaration
4243+
order, including any tags inherited from ancestor suites. Empty when the
4244+
test has no tags. See [Test tags][].
4245+
41224246
### `context.workerId`
41234247

41244248
<!-- YAML
@@ -4334,6 +4458,9 @@ added:
43344458
- v18.0.0
43354459
- v16.17.0
43364460
changes:
4461+
- version: REPLACEME
4462+
pr-url: https://github.com/nodejs/node/pull/63221
4463+
description: Added the `tags` option.
43374464
- version:
43384465
- v18.8.0
43394466
- v16.18.0
@@ -4364,6 +4491,10 @@ changes:
43644491
* `skip` {boolean|string} If truthy, the test is skipped. If a string is
43654492
provided, that string is displayed in the test results as the reason for
43664493
skipping the test. **Default:** `false`.
4494+
* `tags` {string\[]} An array of string labels associated with the subtest.
4495+
Used together with [`--experimental-test-tag-filter`][] to filter which
4496+
tests run. Tags inherit from the parent test or suite by union. See
4497+
[Test tags][]. **Default:** `[]`.
43674498
* `todo` {boolean|string} If truthy, the test marked as `TODO`. If a string
43684499
is provided, that string is displayed in the test results as the reason why
43694500
the test is `TODO`. **Default:** `false`.
@@ -4512,8 +4643,10 @@ test.describe('my suite', (suite) => {
45124643
```
45134644

45144645
[TAP]: https://testanything.org/
4646+
[Test tags]: #test-tags
45154647
[`--experimental-test-coverage`]: cli.md#--experimental-test-coverage
45164648
[`--experimental-test-module-mocks`]: cli.md#--experimental-test-module-mocks
4649+
[`--experimental-test-tag-filter`]: cli.md#--experimental-test-tag-filterexpr
45174650
[`--import`]: cli.md#--importmodule
45184651
[`--no-strip-types`]: cli.md#--no-strip-types
45194652
[`--test-concurrency`]: cli.md#--test-concurrency
@@ -4539,6 +4672,7 @@ test.describe('my suite', (suite) => {
45394672
[`assert.throws`]: assert.md#assertthrowsfn-error-message
45404673
[`context.diagnostic`]: #contextdiagnosticmessage
45414674
[`context.skip`]: #contextskipmessage
4675+
[`context.tags`]: #contexttags
45424676
[`context.todo`]: #contexttodomessage
45434677
[`describe()`]: #describename-options-fn
45444678
[`diagnostics_channel`]: diagnostics_channel.md

doc/node.1

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -776,6 +776,10 @@ collecting code coverage from tests for more details.
776776
Enable module mocking in the test runner.
777777
This feature requires \fB--allow-worker\fR if used with the Permission Model.
778778
.
779+
.It Fl -experimental-test-tag-filter Ar tag
780+
Run only tests whose tag set contains \fItag\fR. May be specified multiple
781+
times; tests must contain every filter to run.
782+
.
779783
.It Fl -experimental-vm-modules
780784
Enable experimental ES Module support in the \fBnode:vm\fR module.
781785
.

lib/internal/test_runner/harness.js

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ function createTestTree(rootTestOptions, globalOptions) {
4747
globalOptions.testSkipPatterns;
4848
const isFilteringByOnly = (globalOptions.isolation === 'process' || process.env.NODE_TEST_CONTEXT) ?
4949
globalOptions.only : true;
50+
const isFilteringByTags = globalOptions.testTagFilters != null;
5051
const harness = {
5152
__proto__: null,
5253
buildPromise: buildPhaseDeferred.promise,
@@ -76,6 +77,7 @@ function createTestTree(rootTestOptions, globalOptions) {
7677
previousRuns: null,
7778
isFilteringByName,
7879
isFilteringByOnly,
80+
isFilteringByTags,
7981
async runBootstrap() {
8082
if (globalSetupExecuted) {
8183
return PromiseResolve();

0 commit comments

Comments
 (0)