Failures to resolve selectors are not forwarded by Suite Engine

[mpkorstanje]

Discovered as part of cucumber/cucumber-jvm#3157.

Steps to reproduce

1. Checkout cucumber/cucumber-jvm-starter-gradle-java@0a8c759
2. Use "io.cucumber:cucumber-bom:7.34.0" in build.gradle.kts
3. Put a break point in the catch block of EngineDiscoveryRequestResolution.resolveCompletely
4. Run the tests.
5. Observe that a discovery issue is reported but not printed.

Context

• Used versions (Jupiter/Vintage/Platform): 6.0.2
• Build Tool/IDE: Gradle

Deliverables

• Investigate why the discovery issue is not reported or forwarded.
• Forward the discovery issues from the Suite Engine.

[mpkorstanje]

Discovery issues are not forwarded because the SuiteLauncher invokes EngineDiscoveryOrchestrator.discover without a phase. Thus causing all issues to go unforwarded. Adding configuration parameter junit.platform.discovery.issue.failure.phase=discovery to the suite will result in the issues being forwarded.

Proposed solution: the SuiteLauncher is a Launcher and should provide it's launcher phase. This does require making LauncherPhase public.

[mpkorstanje]

Interestingly enough we test against this in discoveryIssueOfNestedTestEnginesAreReported but in our case the test engine failed to discovery any test and is pruned.

[mpkorstanje]

Additionally while the DiscoveryIssueCollector collects issues through selectorProcessed, the DiscoveryIssueForwardingListener does not forward selectorProcessed calls.

[mpkorstanje]

Got it.

The problem occurs specifically when EngineDiscoveryRequestResolution.resolveCompletely encounters an exception while resolving a selector. This exception is turned into a selectorProcessed event which the DiscoveryIssueCollector then observes and turns into as a discovery issue.

Under normal circumstances, this issue would then be reported by EngineDiscoveryOrchestrator.reportDiscoveryIssues.

But the Suite Engine doesn't use a discovery phase and instead the discovery request created by the suite engine forwards all discovery issues to its parent request.

The problem is that the DiscoveryIssueCollector collects issues through issueEncountered and creates new issues through selectorProcessed. While the former are forwarded, the latter are not. And so they end up being discarded.