Skip to content

test_runner: extend tag filter with boolean expression DSL #63054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
atlowChemi wants to merge 1 commit into
base: main
Choose a base branch
from
Open

test_runner: extend tag filter with boolean expression DSL #63054

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions doc/api/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1413,22 +1413,28 @@ Enable module mocking in the test runner.

This feature requires `--allow-worker` if used with the [Permission Model][].

### `--experimental-test-tag-filter=<tag>`
### `--experimental-test-tag-filter='<expr>'`
Copy link
Copy Markdown
Contributor

@aduh95 aduh95 May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure adding quotes helps here, to me at least that seems more confusing (e.g. git docs do not do that: https://git-scm.com/docs/git-log)

Suggested change
### `--experimental-test-tag-filter='<expr>'`
### `--experimental-test-tag-filter=<expr>`


<!-- YAML
added: REPLACEME
-->

> Stability: 1.0 - Early development

Run only tests whose tag set contains `<tag>`. Tests declare tags via the
`tags` option on `test()`, `it()`, `suite()`, or `describe()`; tags
inherit from suites to nested tests by union. Filtering is
case-insensitive.
Run only tests that match the provided boolean tag-filter expression. Tests
declare tags via the `tags` option on `test()`, `it()`, `suite()`, or
`describe()`. Tags inherit from suites to nested tests by union.

The flag may be specified more than once; tests must contain **every**
filter value to run. See [Test tags][] for details on declaring and
inheriting tags.
The expression supports boolean operators (`and`/`&&`, `or`/`||`,
Copy link
Copy Markdown
Member

@MoLow MoLow Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isnt there a better way then this tag filtering syntax?.
i.e accept in the run method a string[] or function(string) => boolean and in case you want some complex logic - use the run api without the node cli - That makes much more sense to me

Copy link
Copy Markdown
Member

@MoLow MoLow Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that is more consistent to our name filter flag

Copy link
Copy Markdown
Member Author

@atlowChemi atlowChemi Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the boolean syntax this feature mostly collapses into name-pattern...

The goal (I had in mind) of this feature is to introduce a easy (and common, see prior art) way to filter without the need for a programmatic API. Sure, you could already achieve complex filtering via run() API, but the goal here is to add a built-in logic for this.

Vitest, mocha-tags, and jest-runner-groups all expose such a boolean composition syntax.

Copy link
Copy Markdown
Member

@JakobJingleheimer JakobJingleheimer May 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I even saw Moshe's comment, I was thinking the same thing. But after a minute, I can see the use of it.

I know && and || align with javascript, but it seems a little long, and the simpler & and | look pretty straightforward to me (and aligns with others, like query params and old skool forum query syntax). What happens when you combine them though?

--experimental-test-tag-filter=foo&bar&qux|zed

Is that "foo and bar and (qux or zed)" or "(foo and bar and qux) or zed"? For some reason, my eyes see the former, but syntactically, it's usually the latter. IMO the original spec for syntax made a huge mistake not requiring parentheses when combining operators. If we support combining operators, I think parentheses should be required.

--experimental-test-tag-filter=(foo&bar&qux)|zed
--experimental-test-tag-filter=foo&bar&(qux|zed)

Copy link
Copy Markdown
Member Author

@atlowChemi atlowChemi May 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakobJingleheimer I feel that & vs && is not really "a little long", and I feel using & and | could be confusing with bitwise operators, while removing just a single character...

Copy link
Copy Markdown
Member

@JakobJingleheimer JakobJingleheimer May 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered that before posting: I think nobody would be confused here because bitwise is not appropriate.

&& vs & is a thought musing.

My bigger concern would be parens for combined operators.

Copy link
Copy Markdown
Member

@MoLow MoLow May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thats why I think we should release a initial version that does not include this new langauge

Copy link
Copy Markdown
Member Author

@atlowChemi atlowChemi May 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MoLow @JakobJingleheimer I opened #63221 to separate these concerns.

Copy link
Copy Markdown
Contributor

@aduh95 aduh95 May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it really make sense to have this in core? It adds a lot of complexity, I wonder if having it as a plugin maintained separately would make more sense

Copy link
Copy Markdown
Member Author

@atlowChemi atlowChemi May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aduh95 IMHO it makes sense because it would add a more complex mechanism for filtering tests, which we currently don't have.

I spoke with @MoLow about maybe simplifying by allowing to pass a (tags: string[]) => boolean or similar to testTagFilters. The issue I see with that being that we can't serialize a function, and we couldn't safely call it from child processes when isolation: 'process', making it pretty non-feasible.

wonder if having it as a plugin maintained separately would make more sense

I am happy to do that, but still - I am not sure if/how that would be achievable

`not`/`!`), parentheses for grouping, and `*` wildcards inside identifiers.
Standard precedence applies: `not` binds tighter than `and`, which binds
tighter than `or`. See [Test tags][] for the full grammar and behavior.

The flag may be specified more than once; multiple expressions are combined
with AND, so a test must satisfy every expression to run.

A malformed expression causes the test runner to exit with a non-zero status
before running any tests.

### `--experimental-vm-modules`

Expand Down
99 changes: 76 additions & 23 deletions doc/api/test.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,8 +489,8 @@ added: REPLACEME

Tags annotate tests and suites with arbitrary string labels. The
[`--experimental-test-tag-filter`][] CLI flag (or the `testTagFilters`
option on [`run()`][]) selects tests whose tag set contains every
provided filter value.
option on [`run()`][]) selects tests by a boolean expression over those
labels.

Tags are an alternative to encoding metadata into test names. They are
useful for cross-cutting axes such as subsystem, speed bucket, flakiness,
Expand Down Expand Up @@ -523,37 +523,89 @@ describe('database', { tags: ['db'] }, () => {
});
```

Tag values must be non-empty strings. Tags are matched case-insensitively;
the canonical form is lowercase. Duplicates within a single `tags` array
are collapsed on the lowercased form, preserving the first-seen
declaration order.
Tag values must be non-empty strings that contain no whitespace, no
operator characters (`& | ! ( ) *`), and are not the reserved words
`'and'`, `'or'`, or `'not'` in any casing. Tags are matched
case-insensitively; the canonical form is lowercase. Duplicates within a
single `tags` array are collapsed on the lowercased form, preserving the
first-seen declaration order.

Hooks (`before`, `after`, `beforeEach`, `afterEach`) do not declare their
own tags. They run as part of their owning suite, which carries the
suite's tags.

### Filtering by tag
### Filtering syntax

Each [`--experimental-test-tag-filter`][] value is a literal tag name. A
test runs only when its tag set contains that name. The flag may be
specified more than once; tests must match **every** filter to run. The
same applies to the `testTagFilters` array on [`run()`][]. Filters are
case-insensitive and AND'd with [`--test-name-pattern`][],
[`--test-skip-pattern`][], and `.only` filtering.
The filter expression supports:

Untagged tests are excluded under any non-empty filter, since the filter
requires the tag to be present.
* Identifiers—any non-whitespace, non-operator characters. A literal
identifier matches a tag of the same value (case-insensitive).
* `*` wildcards inside an identifier match any sequence of characters.
A bare `*` matches any tagged test.
* Boolean operators with two equivalent forms:
* `and` / `&&`
* `or` / `||`
* `not` / `!`
* Parentheses for grouping.
Comment on lines +541 to +549
Copy link
Copy Markdown
Contributor

@aduh95 aduh95 May 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too keen on creating a new language for that. Let's either use JS (e.g. --experimental-test-tag-filter='(tag) => tag.startsWith("foo-")'), RegEx (e.g. --experimental-test-tag-filter=/^foo-/), or let's not add it into core

Copy link
Copy Markdown
Member Author

@atlowChemi atlowChemi May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better if we have this conversation at: #63054 (comment)?

Having it in JS means we have a problem (can't serialize functions, so isolation: 'process' is not going to work straightforward). Using regex doesn't solve the problem of "is X or Y but not Z", which JS/DSL would..


### Reading tags from inside a test
The word forms (`and`, `or`, `not`) require whitespace separation; the
punctuation forms do not.

#### Operator precedence

The expression is evaluated with the standard precedence
`not > and > or`. Binary operators are left-associative.

| Expression | Equivalent grouping |
| -------------- | ------------------- |
| `a or b and c` | `a or (b and c)` |
| `not a and b` | `(not a) and b` |

Use parentheses to override:

| Expression | Selects |
| ------------------------------ | ------------------------------------------ |
| `(unit or smoke) and not slow` | unit-or-smoke tests that are not also slow |
| `db && !flaky` | db tests that are not flaky |
| `*` | every tagged test |

#### Untagged tests

Untagged tests behave as if they have an empty tag set. As a result:

| Filter expression | Untagged test | Why |
| ------------------------ | ------------- | ------------------------------------------------ |
| `db` | excluded | Positive match against an empty tag set is false |
| `*` | excluded | The bare wildcard requires at least one tag |
| `db or unit` | excluded | Both branches are false against an empty tag set |
| `not flaky` | included | Negation against an empty tag set is true |
| `not flaky and not slow` | included | Both negations are true against an empty tag set |
| `db or not flaky` | included | The negated branch is true |

For example, `--experimental-test-tag-filter='not flaky'` runs every test
that is not tagged `flaky`, including all untagged tests.

#### Composing multiple filters

[`--experimental-test-tag-filter`][] may be specified more than once on the
command line. Multiple expressions compose by AND—a test must satisfy
every expression to run. The same applies to passing an array to
`testTagFilters` on [`run()`][]. The tag filter is also AND'd with
[`--test-name-pattern`][], [`--test-skip-pattern`][], and `.only`
filtering.

#### Reading tags from inside a test

The [`TestContext`][] object exposes the test's tags as a frozen array
through [`context.tags`][], so tests can branch on their own metadata.

### Errors
#### Errors

A tag value that violates the validation rules above throws
`ERR_INVALID_ARG_VALUE` at the registration site, before any test runs.
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`.
A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`. A malformed
filter expression on the CLI causes the test runner to exit with a
non-zero status before running any test files.

## Extraneous asynchronous activity

Expand Down Expand Up @@ -826,7 +878,7 @@ test runner functionality:

* `--test` - Prevented to avoid recursive test execution
* `--experimental-test-coverage` - Managed by the test runner
* `--experimental-test-tag-filter` - Filter values are validated by the parent
* `--experimental-test-tag-filter` - Filter expressions are validated by the parent
process and re-emitted to child processes
* `--watch` - Watch mode is handled at the parent level
* `--experimental-default-config-file` - Config file loading is handled by the parent
Expand Down Expand Up @@ -1737,10 +1789,11 @@ changes:
For each test that is executed, any corresponding test hooks, such as
`beforeEach()`, are also run.
**Default:** `undefined`.
* `testTagFilters` {string|string\[]} A tag name, or an array of tag names,
used to filter tests by their declared tags. Tests must contain every
listed tag to run. Equivalent to passing [`--experimental-test-tag-filter`][]
on the command line. See [Test tags][]. **Default:** `undefined`.
* `testTagFilters` {string|string\[]} A boolean expression, or an array of
boolean expressions, used to filter tests by their declared tags.
Multiple expressions compose by AND. Equivalent to passing
[`--experimental-test-tag-filter`][] on the command line. See
[Test tags][]. **Default:** `undefined`.
* `timeout` {number} A number of milliseconds the test execution will
fail after.
If unspecified, subtests inherit this value from their parent.
Expand Down
8 changes: 5 additions & 3 deletions doc/node.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -776,9 +776,11 @@ collecting code coverage from tests for more details.
Enable module mocking in the test runner.
This feature requires \fB--allow-worker\fR if used with the Permission Model.
.
.It Fl -experimental-test-tag-filter Ar tag
Run only tests whose tag set contains \fItag\fR. May be specified multiple
times; tests must contain every filter to run.
.It Fl -experimental-test-tag-filter Ar expr
Run only tests that match the boolean tag-filter expression.
The expression supports \fBand\fR/\fB&&\fR, \fBor\fR/\fB||\fR, \fBnot\fR/\fB!\fR,
parentheses, and \fB*\fR wildcards. May be specified multiple times; multiple
expressions are AND'd together.
.
.It Fl -experimental-vm-modules
Enable experimental ES Module support in the \fBnode:vm\fR module.
Expand Down
28 changes: 21 additions & 7 deletions lib/internal/test_runner/runner.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ const {
parseCommandLine,
} = require('internal/test_runner/utils');
const {
validateAndCanonicalizeTagFilter,
parseTagFilterExpression,
} = require('internal/test_runner/tag_filter');
const { Glob } = require('internal/fs/glob');
const { once } = require('events');
Expand Down Expand Up @@ -217,7 +217,7 @@ function getRunArgs(path, { forceExit,
ArrayPrototypeForEach(testSkipPatterns, (pattern) => ArrayPrototypePush(runArgs, `--test-skip-pattern=${pattern}`));
}
if (testTagFilterExpressions != null) {
ArrayPrototypeForEach(testTagFilterExpressions, (value) => ArrayPrototypePush(runArgs, `--experimental-test-tag-filter=${value}`));
ArrayPrototypeForEach(testTagFilterExpressions, (expr) => ArrayPrototypePush(runArgs, `--experimental-test-tag-filter=${expr}`));
}
if (only === true) {
ArrayPrototypePush(runArgs, '--test-only');
Expand Down Expand Up @@ -809,19 +809,34 @@ function run(options = kEmptyObject) {
});
}

// The public contract of testTagFilters is `string | string[]`. The
// parseCommandLine bootstrap path piggybacks the already-parsed AST array
// on the same field, identifiable by the sibling testTagFilterExpressions
// field which only that path sets. When that marker is present and the
// first element isn't a string, treat the array as ASTs and skip the
// public validation loop. Otherwise validate every element as a string,
// so any non-string input throws ERR_INVALID_ARG_TYPE with the offending
// index regardless of position.
let testTagFilterExpressions = null;
if (testTagFilters != null) {
if (!ArrayIsArray(testTagFilters)) {
testTagFilters = [testTagFilters];
}
if (testTagFilters.length === 0) {
testTagFilters = null;
} else if (options.testTagFilterExpressions != null &&
typeof testTagFilters[0] !== 'string') {
// Internal bootstrap: trust the AST array as already-parsed.
} else {
emitExperimentalWarning('Test tags');
testTagFilters = ArrayPrototypeMap(testTagFilters, (value, i) => (
validateAndCanonicalizeTagFilter(value, `options.testTagFilters[${i}]`)
));
testTagFilterExpressions = testTagFilters;
testTagFilterExpressions = ArrayPrototypeSlice(testTagFilters);
testTagFilters = ArrayPrototypeMap(testTagFilters, (value, i) => {
const name = `options.testTagFilters[${i}]`;
if (typeof value !== 'string') {
throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
}
return parseTagFilterExpression(value, name);
});
}
}
testTagFilterExpressions ??= options.testTagFilterExpressions;
Expand Down Expand Up @@ -922,7 +937,6 @@ function run(options = kEmptyObject) {
inspectPort,
testNamePatterns,
testSkipPatterns,
testTagFilters,
testTagFilterExpressions,
hasFiles: files != null,
globPatterns,
Expand Down
Loading
Loading