Guarantee `handle_alloc_error` never unwinds

[nia-e]

Proposal

Problem statement

Currently, handle_alloc_error cannot unwind on stable. However, the unstable set_alloc_error_hook API would allow that to change, potentially allowing an unwind to happen out of an allocation error.

Motivating examples or use cases

Several soundness bugs have come up due to unwinds midway through allocation, enough that there is a significant push to change this behaviour. Additionally, guaranteeing that we never unwind from an allocation error would remove a lot of possible unwinds from the standard library, such as the Clone impl of Box which currently calls handle_alloc_error.

This also pushes forward allocator_api; for soundness reasons, we currently require that cloning an allocator does not unwind, and Allocator is auto-implemented for Box<A: Allocator, _>. If the standard library impl of Clone for Box never unwinds, it would be a valid Clone impl with respect to Allocator.

Solution sketch

Add a guard to handle_alloc_error that aborts on drop, thus guaranteeing that an unwind from an alloc error hook does not leave handle_alloc_error. Document this as a stable promise.

Alternatives

• Make set_alloc_error_hook unsafe, and require that the caller guarantees the hook passed in never unwinds. Realistically this would also be done with a guard, so we'd just be moving complexity around.
• Keep the status quo and accept that cloning a Box<A: Allocator> may unwind. This makes the justification for impl<A: Allocator> Allocator for Box<A> much weaker, since it would no longer really be a useful allocator.

Links and related work

• https://rust.tf/156490
• https://rust.tf/155746

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[Amanieu]

cc @kornelski based on your comment from rust-lang/rust#147725 (comment)

[nia-e]

We decided to take this in a different direction per today's libs-api meeting; it seems there are some usecases (like the above) for unwinding on OOM, and the API proposal we came up with was splitting set_alloc_error_hook in twain. Namely, set_alloc_error_hook_aborting() and the super-duper-unsafe you-need-to-vet-your-entire-dependency-stack set_alloc_error_hook_unwinding() (names open to bikeshed). I'll close this ACP and go in that direction, but I'll make it clear that using the latter means effectively accepting that there will be vast swathes of library code (including std) that you will need to either vendor or patch upstream to be able to use because this is a usecase that Cloudflare has