Keyed [NotInParallel] tests ignore Order and ExecutionPriority because they are sorted by hash code #7088

Summary

Keyed [NotInParallel("key", Order = ...)] tests do not run in Order order, and [ExecutionPriority] is not honoured between them. A keyed test with no Order runs before keyed tests with Order = 1, 2 and 3. The same happens inside a [ParallelGroup].

Un-keyed [NotInParallel(Order = ...)] tests are not affected.

Repro

public class KeyedOrder
{
    [Test, NotInParallel("db", Order = 3)] public Task Order3() => Task.CompletedTask;
    [Test, NotInParallel("db")] public Task DefaultOrder() => Task.CompletedTask;
    [Test, NotInParallel("db", Order = 1_000_000_000)] public Task Order1000000000() => Task.CompletedTask;
    [Test, NotInParallel("db", Order = 1)] public Task Order1() => Task.CompletedTask;
    [Test, NotInParallel("db", Order = 50), ExecutionPriority(Priority.High)] public Task HighPriority_Order50() => Task.CompletedTask;
    [Test, NotInParallel("db", Order = 2)] public Task Order2() => Task.CompletedTask;
}

Record the order in which the tests start, for example in a [Before(Test)] hook.

Expected

Highest priority first, then ascending Order:

HighPriority_Order50, Order1, Order2, Order3, Order1000000000, DefaultOrder

Actual

The same on current main in both source-generated and reflection mode, with or without [ParallelGroup] on the class:

Order1000000000, DefaultOrder, HighPriority_Order50, Order1, Order2, Order3

Root cause

TestGroupingService sorts the keyed tests correctly. Then it turns each test's TestPriority (a record struct of Priority and Order) into an int with TestPriority.GetHashCode(). This happens in two places: the top-level keyed list and the keyed list of each constrained parallel group. ConstraintKeyScheduler sorts the tests again on that int, and it uses the int again to choose which waiting test to start when a key is released.

The compiler-generated hash is roughly (int)Priority * -1521134295 + Order, computed with wrap-around. It does not preserve ordering:

• With the default priority (Normal = 2), the hash is 1252698706 + Order. Any Order above about 894,784,941 wraps to a negative number. That includes the default Order (int.MaxValue / 2), so tests without an Order sort ahead of tests with small explicit orders.
• Different priorities are spread across the int range in no useful order, so [ExecutionPriority] is effectively random between keyed tests.

Impact

Anyone who relies on Order or [ExecutionPriority] for keyed [NotInParallel] tests, including tests in a [ParallelGroup], gets the wrong order. The most common trigger is mixing tests that set Order with tests that don't. The existing NotInParallelOrderExecutionTests did not catch this. Every test there uses the default priority and a small Order, and its order check uses IsEquivalentTo, which ignores order.