`./x test clippy` does not honor `--test-args`

[RalfJung]

Summary

./x test clippy does not support filtering tests the way other test suites do, by passing --test-args.

Command used

./x test clippy --stage 1 --test-args line

Expected behaviour

Only the tests with the word "line" in the file get run. (C.f. ./x test miri -- atomic).

Actual behaviour

All tests get run.

More details

The logic that makes this work in Miri is here:

rust/src/bootstrap/src/core/build_steps/test.rs

Line 685 in d617902

cargo.arg("--").args(builder.config.test_args());

There's something in clippy that seems related but I don't understand any part of it:

rust/src/bootstrap/src/core/build_steps/test.rs

Lines 792 to 807 in d617902

	// Collect paths of tests to run
	'partially_test: {
	let paths = &builder.config.paths[..];
	let mut test_names = Vec::new();
	for path in paths {
	if let Some(path) =
	helpers::is_valid_test_suite_arg(path, "src/tools/clippy/tests", builder)
	{
	test_names.push(path);
	} else if path.ends_with("src/tools/clippy") {
	// When src/tools/clippy is called directly, all tests should be run.
	break 'partially_test;
	}
	}
	cargo.env("TESTNAME", test_names.join(","));
	}

Also the logic inside clippy's ui test runner is odd -- usually, Rust test suites take their test filter form the command-line arguments. Here's Miri doing that:

rust/src/tools/miri/tests/ui.rs

Line 159 in db617af

let mut args = ui_test::Args::test()?;

We don't even have to do anything, ui_test just does this by default.

Clippy also does this...

rust/src/tools/clippy/tests/compile-test.rs

Line 112 in 3ee4325

let mut args = Args::test().unwrap();

... but then it overwrites the resulting filter for some reason:

rust/src/tools/clippy/tests/compile-test.rs

Line 118 in 3ee4325

filter_files: env::var("TESTNAME")

Is there a reason clippy makes this different from every other crate I have ever seen?

Cc @flip1995 @oli-obk @Kobzol

[RalfJung]

... but then it overwrites the resulting filter for some reason:

Ah, it's not quite an overwrite, the filters from the argument should still end up being used:

rust/src/tools/clippy/tests/compile-test.rs

Line 133 in 3ee4325

config.with_args(&args);

So TESTNAME just exists on top of the arguments, maybe? The code is hard to follow and sparsely commented, so -- unclear.

But it looks like it may "just" be a matter of actually having bootstrap set these arguments? There's this existing partially_test logic though, whatever that does...

[flip1995]

The TESTNAME thing is probably a legacy thing. That's how we filtered tests before using the ui_test crate