trait_selection: Resolve inference variables after deep normalization

[Dnreikronos]

Fixes #160875

Deep normalization can return a value that still contains an inference variable after fulfillment has already resolved it. The fallback replaces the outer alias with a fresh variable and collects an obligation. Fulfillment updates the inference context later, but it doesn't rebuild the folded value, so the caller can still see fn(?3t) after the context knows its final type.

Resolve the value after fulfillment and add a UI test for nested inherent associated types. The test runs with compiler debug assertions because the caller resolves the value before returning, which hides the bug from normal output. I think deep normalization is the right place for the fix since it creates the variable and runs its obligation. After lcnr's review, I traced the fallback end to end and updated the comments and test name around the actual path: the eager outer goal returns NoSolution, then normalize_with_universes retries the original type with ReplaceAliasWithInfer. The invalid body only reaches the MIR assertion. The odd higher-ranked IAT behavior looks separate to me, so I'd rather investigate it outside this fix.

[rustbot]

Some changes occurred to the core trait solver

cc @rust-lang/initiative-trait-system-refactor

[rustbot]

r? @TaKO8Ki

rustbot has assigned @TaKO8Ki.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 75 candidates
• Random selection from 17 candidates

[lcnr]

r? lcnr

Deep normalization is interesting. From the comment of that function

/// Deeply normalize all aliases in `value`. This does not handle inference and expects
/// its input to be already fully resolved.

When do we encounter aliases without any infer vars, whose normalization is ambiguous, then we normalize something else, and suddenly it works?

Or is that a "we normalize and stay ambig, normalize the alias containing this alias, and this then succeeds and returns something which doesn't reference the infer var from the inner alias"? 🤔

Can you say more about why this example with nested inherent assoc types has this behaior or what exactly is going on here?

[lcnr]

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[Dnreikronos]

sup @lcnr :)

I traced this again, and my explanation in the PR body is wrong. This test doesn't hit ambiguity at all.

I saw the ambiguity handling in normalize_with_universes, connected it to the unresolved inference var returned by deep normalization, and assumed that was the path. That was too quick.

The input to deeply_normalize is fully resolved. The inference vars are created inside normalization for the projection results.

The inner alias normalizes first:

Foo<fn(&'b ())>::Assoc -> &'b ()

Then it tries the outer alias with that result folded into it:

Foo<for<'b> fn(&'b ())>::Assoc -> ?T

That returns NoSolution, not Maybe.

Since the fold failed, normalize_with_universes throws away the partial result and retries the original value with ReplaceAliasWithInfer. The inner alias has escaping bound vars, so it gets an obligation but stays in the value. The outer alias is replaced with a fresh inference var. That's where fn(?3t) comes from.

Fulfillment later checks those obligations. The outer obligation still contains the original nested alias, and type relating the higher-ranked equality creates projection goals for that inner alias. This time it succeeds and constrains ?3t to &'?0 ().

So your second guess was basically right. The difference is that the first attempt isn't ambiguous. It returns NoSolution, then the fallback retries the goal in a different shape and succeeds.

The actual bug is simpler than the path that gets us there. We build fn(?3t) before fulfillment runs. Fulfillment updates the inference tables, but it doesn't rebuild that value. The inference context knows what ?3t became, while the value still contains the inference var:

fn(?3t)
fn(&'?0 ())

The old deep normalization code resolved each result after proving its obligation. The newer code proves the obligations together at the end, and the final resolve was lost in that change. I still think resolving after fulfillment is the right fix.

The part that still looks odd to me is the IAT behavior. Eagerly normalizing the inner alias makes the outer goal return NoSolution, while leaving the nested alias for type relating lets it succeed later. My guess is that this is a separate higher-ranked IAT issue.

I was also wrong about the bad function body keeping the goal ambiguous. A valid body takes the same fallback path, but the test doesn't reach the MIR assertion. I'll fix that explanation, the code comment, and probably the test name.

My vote is to keep this PR focused on the missing resolve and investigate the IAT behavior separately. Does that split make sense to you, or do you want the IAT part understood first?

[Dnreikronos]

sup, @lcnr o/

I just pushed the cleanup..the code comment now describes the fallback instead of ambiguity, and I renamed the test and fixed its explanation about the body error.

The path is the one from my last commen, so if you could, please, re-validate: the eager fold normalizes the inner IAT, the outer goal gets NoSolution, and the fallback retries the original type with a fresh inference variable plus obligations. Fulfillment constrains that variable, but it does not rebuild the value, so the resolve after fulfillment picks up what the inference context learned.

I still think the resolve belongs in deep normalization. That is where the variable is created and where its obligations are run, so returning before applying the fulfillment result feels like the wrong boundary.

I kept the strange higher-ranked IAT behavior separate. I still do not have a satisfying answer for why the eager goal fails while the fallback relation works, and I do not want to pretend the missing resolve explains that part. If you would rather have that understood before this lands, I can dig into it first. Otherwise I think a separate follow-up is cleaner.

@rustbot ready

[lcnr]

So the bug here is that the ReplaceAliasesWithInfer visitor doesn't handle aliases with escaping bound vars properly and allows leaking placeholders. So deeply_normalize fails with a leak check error while folding the returned type with ReplaceAliasesWithInfer succeeds?

I feel like this fix papers over a deeper type system bug and we should not do this change ideally. can you look into what's necessary to make the ReplaceAliasesWithInfer by actually changing normalize_with_universes to return a Result. Feel free to start a convo on zulip about this

[lcnr]

@rustbot author

[Dnreikronos]

Closing this in favor of #162618. After lcnr's review, that PR makes normalize_with_universes return the eager normalization failure. Resolving the fallback's inference variable after fulfillment, as this PR did, only hid that failure. The regression test for #160875 moved there too.