Repository navigation
Use UIA for notifications on Windows 10 Fall Creators update and newer - #198
Conversation
|
Unfortunately, the thing still crashes when JAWS is active and a notification is fired. I wonder what JAWS is doing that isn't appreciated by my code. |
|
Ah, now the thing no longer crashes, however it seems JAWS doesn't respond to notification events at all. I need to find out why not. |
|
Some research revealed that JAWS also does not support the UIA Notification events within Alarms and clock and other apps. @jcsteh: Do you think this still has potential, or should I close it? Never mind the work, it was kind of an adventure and I learnt a lot in the process. |
|
I find this very curious. I was led to believe JAWS did support these. I might try to discuss it with Freedom Scientific; I have contacts there. |
|
@jcsteh: I think discussing this with Freedom Scientific could be very helpful, also to find out what they support and what not. |
|
Chaps, Jim Snowbarger is someone closer to home who might be able to advise on this. His email is snowman@snowmanradio.com |
|
@jcsteh, just a heads up to ask whether you have been able to discuss this with someone at freedom scientific? |
|
@jcsteh, you pointed out in #198 (comment) that you might be able to discuss this with @FreedomScientific. Any luck in doing so? |
|
I haven't discussed this with them yet, no. I don't have as much regular contact with them as I did. Still, I'll try to follow up on this at some point. |
|
I finally got around to pinging Freedom Sci about this just now. Obviously can't comment on if/when I'll get an answer. |
|
Apparently, UIA notification events were disabled by default in JAWS because of certain apps which were abusing them. They can be enabled for an app with a short script (not sure about details). FS are going to enable this by default for REAPER in a future update. Unfortunately, we can't just switch over even then because there will probably be a bunch of users using older JAWS versions. Perhaps we can get Snowman to add the necessary code to his scripts. |
|
As an alternative, we could consider using an UIA live region. Downside of that is probably that rapid changes don't interrupt each other. |
|
UIA live regions are fundamentally pretty flawed. Also, does JAWS even
support them?
|
| return E_INVALIDARG; | ||
| } | ||
| if (riid == __uuidof(IUnknown)) { | ||
| // Several examples seem to cast to IRawElementProviderSimple instead of IUnknown in this case. |
There was a problem hiding this comment.
This is done because in C++, where there is multiple inheritance, there can be multiple IUnknowns. All COM interfaces derive from IUnknown. So, for example, if you inherit both IRawElementProviderSimple and IRawElementProviderFragment, there are two different IUnknown pointers. COM requires that when you query to IUnknown, you should always get the same pointer, regardless of the originating interface. Casting to IRawElementProviderSimple first ensures you always get the same IUnknown.
Because you're only inheriting from one interface, this technically isn't necessary, but it's good practice because it's very easy to forget to make this change if you later add an additional interface.
…all its resources
…d before a notification event is raised if a client doesn't query the provider for some other reason, which would mean we couldn't fire notifications. This also gives us better control over the lifecycle of the provider.
…ifecycle is managed entirely by AddRef/Release, so the provider will only ever be deleted when the ref count is 0.
…in active/bypass message.
|
If there are plans to merge this into main soon, there should probably also be an exception to fall back on the old method if someone is using the ZDSR screen reader. It's a small, chinese-made app which is the most widely used screen reader there though it's also slowly starting to gain an international user base because it's more responsive than either JAWS or NVDA in some contexts and has some other, unique features. The developer actually touts Reaper support as a selling point for the commercial version, but it has no support for UIA notifications. I ran the most recent build from this branch and indeed get no speech with it while using ZDSR. The process for it is called ZDSRMain.exe. It has no API that can be used to communicate with it directly so checking if that process is running is the best way to check for it as far as I know. I'll try to reach out to the developer to ask about implementing UIA notifications, because this may be a good impulse for him to do so, but stable releases come out occasionally so users may be caught off guard if OSARA updates and they lose speech. I'm also not sure whether Dolphin Supernova supports these, it may be worth tracking down a reaper+SuperNova user to verify this if there are any out there. |
|
Thanks for flagging this.
I'd really rather not test for the existence of processes; the C++ code to
do that involves a ridiculous amount of boilerplate. Do you know if it
injects an in-process dll and what it's called?
I'll probably just deny-list Supernova. I don't know any OSARA Supernova
users I can ask and I'd say it's unlikely that Supernova supports it.
Another alternative might be to allow-list just NVDA and narrator, since it
seems like these are the only two that actually support this properly.
However, Narrator probably doesn't have an in-process dll I can test for.
Urgh.
Failing all of that, I might need to implement a config option for this,
but the problem is that most users won't understand what it's for.
|
|
Those who will have to understand will understand. Please implement the configuration switch. The developer of Zhengdu is already informed. |
|
Having a configuration option does sound like the best option. I can see the benefit of UIA with the possibility of interruptable messages coming in really useful for something like the parameter change v notifications you're working on. Meanwhile, the current method is more compatible, but even with NVDA I can see heavy braille users opting for it since UIA messages are just flashed in braille, while with the current system they don't disappear so you can read longer ones much easier. So it'd be a case of what someone finds more important |
|
Thinking about it more, a config option is going to be pretty difficult in this case. We don't want it to show up on Mac, but because you have to build the dialogs with manual coordinates, it's pretty hard to have an option on one platform but not the other. I also think it's just totally unintuitive and crappy user experience. I'm just going to blocklist any screen reader we learn doesn't support this. |
|
@LeonarddeR, thanks for all your work on this. I hope you don't mind me taking it over, but since I took so long to get to it (and I had quite a few tweaks I wanted made), I figured I'd take care of it myself. |
No problem at all. Thanks for finishing it, great to see this in now! |
…l surfaces. (#352) This can be disabled using the new "Report changes made via control surfaces" option in the OSARA Configuration dialog.
When on Windows 10 fall creators update, this now make use of UIA to output messages.
This is implemented by creating a new window as a child window of Reaper's main window. The new window is has a size of (0, 0) and is disabled. Notifications are sent to this window with UIARaiseNotificationEvent.
This works with NVDA, in fact with NVDA you don't observe a difference, except that the name of the currently focused control is untouched. Also, the notification comes after the announcement of the edit box when renaming a track, this needs to be fixed in such a way that this still changes the accName.
With Narrator, this no longer announces the role of the currently focused element after changing its name. With JAWS, I experienced crashes with a previous implementation, but that was broken in a major way as it destroyed the pointer to the UIAutomation provider when the reference count was back to zero.
I've also experienced some crashes of Reaper itself in some cases with NVDA, especially when I switched from an installed copy to a source copy. That's why this is still a draft pr. There are some more rough edges, the provider isn't properly destroyed when the window is destroyed, only when UIA support is terminated. I'd like to show this as a work in progress though, and would like some extra feedback about whether I'm on the right track. In any case, it is a wonderful experiment up until now.
Things I've tried as well