First-class technique for splitting input streams or absorbing input events

[alice-i-cecile]

What problem does this solve or what need does it fill?

Typically, when users click on a UI element (including in game units), we don't want to also handle that click in the rest of the world.

The same pattern is observed when typing into a chat box, where the keyboard inputs are stolen.

Right now, the default behavior is to pass all events through.

What solution would you like?

A few solutions immediately come to mind:

1. Centralize this input stream splitting technique into a single standard mega-system that can be customized using function pointers.
2. Assign ownership of input stream events by storing some form of owner (entities? labels?) on the input events as an optional field.
3. Assign ownership of input events using entity-specific events (Suggestion: Entity Events #2070, Per entity Events #2116).
4. Add a split method to ResMut<Events<T>>, which returns two iterators of events based on whether the supplied predicate is true. This doesn't make the chaos any better, but does improve ergonomics.

What alternative(s) have you considered?

Consume events in one-off systems using ResMut<Events<Input<T>>>.
This was the approach taken in vladbat00/bevy_egui#50, specifically vladbat00/bevy_egui@53c1773#diff-420f4e08a25299e0e1fb09f4757e1d5d027a2278f37030c09ab9c06786cfa52eR283.

However this is not a particularly principle or discoverable approach, and risks serious chaos as input events are silently eaten according to decentralized rules based on system ordering.

Additional context

Initially discussed on Discord. Led to #3569. Related reddit post with proposed workarounds.

[rezural]

Here are a few of the Use cases I can see involved, without too much of an opinion on how this could be solved:

There are a few cases which complicate how events may want to be handled:

• PointerHidden: the mouse pointer is hidden, the App probably wants exclusive access to the systems
• PointerVisible: the mouse pointer is showing, users can indicate that they want to interact exclusively with Plugin's based on a couple of schemes:
  • PointerLocation: are we hovering over a plugin's UI region?
  • PointerClicked: the user has clicked within a region, indicating they want to interact with this regions Plugin event code
    • Would need Some way of indicating (via predicate?) that the user has 'escaped' from exclusive interaction (i.e. escape key, or moving out of region, or clicking out of region)
  • Via a KeyCombo
• KeyCombo: The user has entered a key combo which indicates they want to focus on a particular Plugin

Possible, high level solutions:

Event consumption registration:

• Notion of different crates/plugins registering for possibly consuming events
  • The App itself
  • Each relevant Plugin

It is probably most flexible that the App chooses the event splitting.
Bevy could provide boilerplate Event splitting Strategy for i.e. the following:

• PointerHidden: The pointer is hidden, default all events to the App
• PointerVisible: The pointer is visible, either:
  • PointerLocation or PointerClicked strategies defines when a Plugin gets access via the pointer's location
• KeyCombo: Events forward to a particular Plugin, until the user indicates they want to escape this mode (ESC, PointerClicked outside of region etc.)

with Apps able to provide their own event splitting behaviors

Plugins could notify bevy that they may want exclusive access to events based on predicates (is pointer over, is key combo just pressed, pointer + click)
This would seem to be brittle in that Plugins decide the UI workflow that should be within the perview of the App (i.e. hover based etc.)

[mockersf]

I think a plugin should not remove info from the built-in Events<T> or Input<T> from Bevy. I may want to capture mouse click even on top of egui UI because I have a shader that displays a ripple effect on clicks. I may want to capture key events because I want to react to a shortcut even if the player is typing in a text field.

When having only one plugin that want to filter events, I originally thought that it must create a new event type and resend original Bevy events, mapping them to its own type, and filtering them to remove the one it already dealt with but that doesn't scale if more than one plugin want to capture events.

For that case I'm thinking now it could be interesting to be able to attach tags to events, and then being able to filter base on those tags so that you could asks for Events<MouseButton, (Without<Egui>, Without<OtherThing>)> and that starts looking like the Query api...

[rezural]

Yes I agree.

I hope that this serves to illustrate the point that there are currently no relevant abstractions that relieve the need for such egregious hacks.

[HackerFoo]

Here' s my suggestion:

• Leave input events as they are.
• Add a new resource to track focus.
• Each target can claim focus with something like focus.claim(layer, tick), where a higher tick, or the same tick and a higher layer take precedence. tick increases with each input event.
• Systems can ignore events when they don't have focus, if it makes sense.
• A new event could be added that includes the owner entity and focus change events, but this would have to be delayed until after the claiming phase.

I use a more sophisticated system that uses ray casts to assign focus, I've approximated this here with layer. My system also tracks the motion of multiple touch points simultaneously, and so must reset the motion vectors on ownership changes to keep a common frame of reference for each entity. A similar problem may need to be solved when implementing something like drag-and-drop.

Without such a system, gestures would be very hard to implement. Even corner cases would cause trouble, like clicking a button, moving off the button and back (or not), then releasing. So maybe it makes sense to implement a higher level input processing system as I have done, as a separate plugin.

Also, multi-touch and mouse input are quite different, so each might use a different plugin.

[alice-i-cecile]

Add a new resource to track focus.

Okay, interesting. So you'd have something like a:

struct CurrentlyFocused(Option<Box<dyn Focusable>>);

This would be extremely useful for bevyengine/rfcs#41

IMO you have to use a trait object there: enums are the natural fit but they're not composable.

Do we ever want to have multiple objects / areas focused at once? No, not really. And if users need this, they can do make their own weird struct to represent hybrid states.

[PaperCow]

For that case I'm thinking now it could be interesting to be able to attach tags to events, and then being able to filter base on those tags so that you could asks for Events<MouseButton, (Without<Egui>, Without<OtherThing>)> and that starts looking like the Query api...

This is an interesting thought. There have been a number of issues and discussions around events that an API like this could address. I know I have been working around a situation similar to #1431 that this would work perfectly for in my own project. Seems to me this could also handle the same situations as things like #1626 in a very "bevy way".

[HackerFoo]

Do we ever want to have multiple objects / areas focused at once? No, not really. And if users need this, they can do make their own weird struct to represent hybrid states.

The input to my app is multi-touch, so it can assign multiple contact points to multiple objects. For single point input, it might make sense to store all layers to form a hierarchy. For example, clicking a text box might also highlight the box enclosing (in a layer behind) the text box. So you could have:

type Layer = usize;
struct CurrentlyFocused(Vec<(Layer, Box<dyn Focusable>)>);

and keep it sorted by Layer, although I'd use the Entity:

struct CurrentlyFocused(Vec<(Layer, Entity)>);

Claiming would clear entries with the same or higher layer and a lower tick, replacing that part of the hierarchy. No action should be taken based on CurrentlyFocused until all systems have had a chance to claim focus, so these might have to run in a different stage.

[nicopap]

I find @HackerFoo 's suggestion very interesting. I've actually came up with a similar system independently in ui-navigation (although didn't implement it yet). What you call claim I call lock here, and instead of a method on a resource, I have a behavior on a specific entity. To "unlock" you would need to send a NavRequest::Free event. See: nicopap/ui-navigation#7

There is no real concept of ownership, basically the locking mechanism is just a way to toggle off input for the ui-nav system. Without such an escape hatch, the user would need to account for my plugin in their own input management systems, which is not ideal.

For ui-nav, focus claim contention is irrelevant, because it imposes a focus hierarchy and is the sole source of truth for knowing what is focused. This also means the tick and layer parameters are irrelevant, since the ui-nav system knows the hierarchy (see the rfc for details bevyengine/rfcs#41)

Beyond that, there no motion vector clipping or coordinate system adjustments.

[Rust-Ninja-Sabi]

I run into this using Egui and mouse events in my game. Is there any solution now?

Thanks.

Best regards Sabi