Audit `run-fail` ui tests to see if they need *exact* run stderr/stdout

[jieyouxu]

compiletest triage: I'm not convinced switching error-pattern run-fail ui tests to use only the current form of check-run-results is an improvement over error-pattern unconditionally. If anything, it would be a regression to flaky tests in many cases that are sensitive to exact run stdout/stderr (line numbers, backtraces that can depend on platform-specific unwind mechanism, concrete addresses, version numbers, etc.).

I would be in favor of auditing some of these run-fail tests and sparingly add check-run-results if the exact run stderr/stdout is critical for the intention of the test, but not in a blanket fashion. As such, I'm going to close this issue in favor of a more concrete audit issue.

Originally posted by @jieyouxu in #65865

[jieyouxu]

See also Audit {check-run-results, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives.

[jieyouxu]

E-medium because you need some experience to not accidentally make a test flaky and to know if a test needs to check for exact run stderr/stdout.

[Pyr0de]

Hey,
I'd like to give this issue a shot

By audit do mean I'll need to change the tests or just report on the tests that can use check-run-results? And if I do need to change them, then do I try to normalize output to get a consistent outputs?

[jieyouxu]

By audit, I meant that you'll need to double-check the tests if they really need exact run stderr, or if they actually just care about a specific error pattern. It's less about the actual snapshot, it's more about the test intention.

[Pyr0de]

Here's an example from what I've understood,
A test that tests panic would need the exact stderr but a test which would panic but the intent of the the test is not to test panic would not need exact stderr (not the best example but I hope it's clear enough)