Microsoft Accessibility - Part 2: Of Properties and Patterns

 This is continuation of Part 1 of the series on Microsoft Accessibility.

1. Microsoft Accessibility - Part 1: Introduction
2. Microsoft Accessibility - Part 2: Of Properties and Patterns

Properties and Patterns

Any assistive technology (AT) such as Windows Narrator is interested in three things that a control has.

1. Properties
2. Patterns
3. Events

Properties are name-value pairs of data associated with a control. Patterns are methods and interfaces that can be invoked on a control. With events your control can notify the AT about its state changes. We defer discussion on events to the future.

Consider the "Press Me" button in the following MFC dialog.

MFC Dialog

Properties associated with it would be things like its name - "Press Me", Type of control - "Button", etc. Patterns associated would be IInvokeProvider (also known as Invoke Pattern), etc. The IInvokeProvider pattern for eg. will allow an AT to programmatically press the button. Here is what accessibility Insights shows for the button. The upper pane on the right shows properties, while the lower column shows the patterns.

Accessibility insights output

The properties of a control is exposed through a special interface IRawElementProviderSimple::GetPropertyValue().

Specifying properties

For native win32 controls most of the required properties, and patterns are already implemented. There are instances, however, when some of these default properties are missing, or are incorrect. We can use OS API to specify these properties. Note that doing this will work only when IRawElementProviderSimple::GetPropertyValue() interface is implemented by OS. If you are implementing the interface yourself, and not calling the OS version of the interface, then specifying this property via the OS API wont do anything.

Such a technique where the accessibility to a control is provided via proxies (by the OS), but the application (or the control) itself specifies some of the properties is called dynamic annotation. There are three types of dynamic annotation: 

• Direct annotation (IAccPropServices::SetPropValue(), IAccPropServices::SetHwndPropStr(), etc.): Direct annotation is used to specify properties which stay constant over time, and is the one we will be studying here.
• Value-mapped annotation (say for certain slider controls, uses IAccPropServices::SetHwndPropStr(), etc.): Value mapped annotation can be used to specify a mapping for the values/strings displayed in a control to some other value. Eg. in a slider control we may have 0, 1, 2, etc. as ticks. These may represent different resolutions of a display in a setting application. Without annotation AT will read the values as 0, 1, 2, etc. but we can use annotation so that AT will read 0 as 800x600, 1 as 1280x720, and so on.
• Server annotation (IAccPropServices::SetPropServer(), IAccPropServer, etc.). This can be used to associate a class with a control item. The control item may be a whole window, or one of the sub elements of its UI. Whenever the AT gets properties for the item it calls the class methods. This will have a blog post of its own.

Via RC files

For some controls, the name property can be specified from within RC file itself. It is fairly simple and explained here. This method is quite limited as only the name property can be specified , and it can only be used with some controls.

Using COM API

For specifying properties, we can use CAccPropServices COM API (oleacc.h, oleacc.dll).

IAccPropServices* pAccService = nullptr;
CoCreateInstance(__uuidof(CAccPropServices), nullptr, CLSCTX_INPROC, IID_PPV_ARGS(&pAccService));

Now we invoke SetHwndPropStr() on this interface. For example this is how we set name.

TCHAR *NameStr = _T("Press Me");
pAccService->SetHwndPropStr(hWnd, (DWORD)OBJID_CLIENT, (DWORD)CHILDID_SELF, Name_Property_GUID, NameStr);

Here OBJID_CLIENT indicates that we are referring to the whole window (or just the client area if the window has non client area as well) represented by hWnd. CHILDID_SELF indicates that we are referring to the control rather than one of its children.

You must include "initguid.h", and "UIAutomation.h" header files. Include "initguid.h" first.

As these are com API, you will have to initialize COM before using them.

CoInitialize(NULL);

In the Accessibility Insights screenshot given above we can see that the "HelpText" property does not exist. Let us use this COM API to specify it.

TCHAR* HelpTextStr = _T("This button is used to demonstrate accessibility API");
pAccService->SetHwndPropStr(hWnd, (DWORD)OBJID_CLIENT, (DWORD)CHILDID_SELF, HelpText_Property_GUID, HelpTextStr);

Here is what Accessibility insights now reports for this control. Compare the "HelpText" property with the that in the previous screenshot.

Do cleanup when your control is getting destroyed.

MSAAPROPID props[] = { Name_Property_GUID, HelpText_Property_GUID };
pAccService->ClearHwndProps(GetSafeHwnd(), (DWORD)OBJID_CLIENT, (DWORD)CHILDID_SELF, props, ARRAYSIZE(props));
pAccService->Release();
CoUninitialize();

The sample code for all this is present on Github. Check InitAccessibility(), and DeInitAccessibility() methods in 01MFCDialogSimple project.

A note on Object ID (idObject), and Child ID (idChild) ^[3]

Consider the prototype for IAccPropServices::SetHwndPropStr()

HRESULT SetHwndPropStr(
  [in] HWND       hwnd,
  [in] DWORD      idObject,
  [in] DWORD      idChild,
  [in] MSAAPROPID idProp,
  [in] LPCWSTR    str
);

hwnd, idObject, and idChild together refer to a UI element. 'hwnd' refers to the window of a control.

idObject refers to the kind of object within the window (specified by hwnd) we want to refer to.

Here are some of the values it can take.

• OBJID_CLIENT: Refers to window's client area. It excludes non client area (frame, etc of the window)
• OBJID_WINDOW: Refers to the whole window. This includes the client area (OBJID_CLIENT), and non-client area
• OBJID_TITLEBAR; Refers to the title bar of the window
• OBJID_HSCROLL: Windows horizontal scroll bar
• OBJID_VSCROLL: Windows vertical scroll bar
• OBJID_CARET: caret in the window
• OBJID_CURSOR: mouse pointer
• Other values (from WinUser.h): OBJID_SYSMENU, OBJID_MENU, OBJID_CLIENT, OBJID_SIZEGRIP, OBJID_ALERT, OBJID_SOUND, OBJID_QUERYCLASSNAMEIDX, OBJID_NATIVEOM

idChild is used to specify one of the child controls within the Window. The Child ID can either be obtained via either the IEnumVARIANT interface if the control supports it, otherwise the child ID are usually in increasing integral order starting with 1. A special value CHILDID_SELF refers to the object itself rather than one of its children.

Specifying properties for non window (non hwnd) controls

Frequently we encounter controls which don't have a window of their own. For example in the toolbar control of notepad++ we can see that the child controls of the toolbar are not a window in themselves. The toolbar is a window, and draws the child controls as bitmaps within itself.

Notepad++ toolbar

In this image we can see that while Window Detective does not show any children of the toolbar, accessibility insights does. This is because the buttons shown in the toolbar are not individual windows hence Windows Detective is not able to detect them. Accessibility insights on the other hand uses automation to enumerate child controls of the toolbar (perhaps via IAccessible interface), and the toolbar is able to convey information about the children it has.

To make things simpler I have modified our MFC dialog so that it now has a toolbar. The toolbar has a background of red cross lines. It has 2 buttons - Button 1, and Button 2. When you compare outputs of Window Detective, and accessibility insights in the following image you see observations similar to notepad++, ie. the buttons are not listed in Window Detective, but are listed in accessibility insights.

MFC Simple Dialog with toolbar

Let us use this to understand how properties for non-window controls can be specified (Button 1, and 2 above). The toolbar does not support IEnumVARIANT interface (which can be seen from Patterns pane of accessibility insights), thus it is safe to assume that child Id would be 1, and 2 respectively for the two buttons. You can also use tools like Inspect to obtain child id.

Child ID from Inspect.exe

Let us see the properties reported by Accessibility insight for button 1.

Name property for button 1 doesn't exist

Notice that the Name property does not exist. We will now use the IAccPropServices::SetHwndPropStr interface once more to set this, but this time we will pass integers 1, and 2 for the two buttons, instead of CHILDID_SELF.

pAccService->SetHwndPropStr(m_toolbar1.GetSafeHwnd(), (DWORD)OBJID_CLIENT, (DWORD)1, Name_Property_GUID, _T("Button 1.0"));
pAccService->SetHwndPropStr(m_toolbar1.GetSafeHwnd(), (DWORD)OBJID_CLIENT, (DWORD)2, Name_Property_GUID, _T("Button 2.0"));

Take a note of the hwnd, object Id, child ID, and the value of string parameters passed above. Let us see what accessibility insights reports now.

Name property for button 1 now set

The name property is now being reported corrected. 

As mentioned before, the sample code for all this (updated to include the toolbar) is present on Github.

References

1. Sample code accompanying this blog - https://github.com/lihas/WindowsAccessibilitySamples
2. Setting name property from RC file - https://docs.microsoft.com/en-us/accessibility-tools-docs/items/win32/control_name?branch=main
3. Microsoft Active Accessibility V2 (Book) - http://download.microsoft.com/download/activaxs/doc/2.0/w98nt42kmexp/en-us/msaav2.doc, https://web.archive.org/web/20211128090129/https://download.microsoft.com/download/activaxs/doc/2.0/w98nt42kmexp/en-us/msaav2.doc
4. Window Detective (tool) - https://windowdetective.sourceforge.io/, https://github.com/lihas/WindowDetective
5. Code samples for this post on Github - https://github.com/lihas/WindowsAccessibilitySamples

Microsoft Accessibility - Part 1: Introduction

This series of blog posts tries to look at Microsoft Accessibility from a developer's point of view - making sure your product works well with assistive technologies like Windows Narrator. The focus is primarily on native developers - C++/MFC/Win32,etc.

Here is a how Microsoft defines Accessibility.

"... enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop. It enables assistive technology products, such as screen readers, to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI"

Accessibility is a way by which other software can read and understand what is displayed in your window. This is used in assistive technology products like screen readers, and for running automated scripts on your application (eg. for automated UI testing).

Significance of having your application support accessibility technologies

When talking about assistive technologies people often imagine (erroneously) users which have severe disabilities - like complete loss of vision. Accessibility pertains to a wide range of people with a wide range of abilities, not just the folks with disabilities^([1]). Another fallacy is to assume that people who need such technologies are only a small fraction of your overall user base (The myth of the minority user). They can constitute more than 50% of your userbase. Here are some data from Forrester Research 2004 study^([1]).

1. 44% of computer users use some form of assistive technology
2. 57% of computer users could benefit from assistive technologies
3. 1 in 4 people experience visual difficulties
4. 1 in 4 experience a pain in wrists or hands
5. 1 in 5 experience hearing difficulty
6. 57 million users of accessibility technologies in 2003, and it was only expected to grow year on year.

Such data are an eye opener. Not only the people needing assistive technologies is not small, they constitute a big fraction of all computer users. Additionally, many countries have laws which make it mandatory for your software to support various forms of accessibility (see this, and this - Minimize Legal Risk). Norway for example fines commercial websites which fail to support accessibility^([2]).

Thus the time and money you invest in making your product accessible will be worthwhile.

Microsoft accessibility frameworks - Active accessibility, and UI Automation

MS has had native support for accessibility frameworks since Windows 95 days. As of now there are two main frameworks - Microsoft Active accessibility (MSAA), and UI Automation (UIA). Active accessibility is the older framework, which has been there since Win95 days. UI Automation is the newer framework release in 2005 with Vista, and tries to overcome limitations of MSAA. MSAA is simpler to implement, but is also limited in features and performance. If your software supports neither, and you are beginning to support one of these, it is better to implement UIA. Note that the same software if needed can implement both MSAA, and UIA.

See this - Microsoft Active Accessibility and UI Automation Compared

The focus of this series of articles will be mainly UIA.

How to support UI Automation (UIA) framework - add support for accessibility

For win32 applications to support accessibility they have to implement various COM interfaces, and specify certain properties. For instance there is one COM interface which allows an assistive technology (AT) like Screen reader to read the name of a button (IRawElementProviderSimple::GetPropertyValue()), there is another interface which allows an AT to click the button (IInvokeProvider), there is an interface which allows an AT to read, and set values in a text control (IValueProvider), and so on. These interfaces are also known as patterns. MS maintains a mapping of control type, and patterns it must support (See this). It is these mappings which are checked by tools such as Accessibility Insights. Properties are certain pieces of data associated with a control which describe it, like name, type of control, etc. While Patterns are interfaces and methods callable on a control, properties are like name-value pairs of data.

Consider the following image which shows info shown by Accessibility insights for a Dialog based MFC application's "OK" button.

Name, ControlType, etc. are the properties. The Patterns which the control supports are listed in the bottom pane.

Native win32 controls have these patterns, and properties implemented by default. Hence when using Win32, or MFC you will seldom have to tinker with accessibility related stuff. There may be times when the support provided by default isn't enough - say a pattern is missing or a property is not set correctly. You may also have a custom control in your application, which will require the accessibility interfaces to be implemented explicitly from scratch.

In the next few posts we will be diving deeper into some of these concepts.

References

1. Engineering software for accessibility (book)
2. https://www.w3.org/WAI/business-case/#minimize-legal-risk
3. This application handles WM_GETOBJECT, and implements an automation interface - https://github.com/UiPathJapan/RespondingWmGetObject 
4. Implementing automation provider (blog post) - https://vivekcek.wordpress.com/2015/01/09/ui-automation-provider-for-a-custom-control-problem-to-solution-approach/
5. List of pattern ID - https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpattern-ids
6. Application which tries to implement several Automation provider sever interfaces - https://github.com/netide/netide. See this, and this.
7. How WM_GETOBJECT works - https://docs.microsoft.com/en-us/windows/win32/winauto/how-wm-getobject-works
8. Tools - https://docs.microsoft.com/en-us/windows/win32/winauto/testing-tools
9. Handling WM_GETOBJECT - https://docs.microsoft.com/en-us/windows/win32/winauto/handling-the-wm-getobject-message
10. Active accessibility vs UI Automation - https://docs.microsoft.com/en-us/windows/win32/winauto/microsoft-active-accessibility-and-ui-automation-compared
11. https://docs.microsoft.com/en-us/windows/apps/develop/accessibility
12. MS Code samples - https://github.com/microsoft/Windows-classic-samples/search?q=accessibility&unscoped_q=accessibility
13. https://docs.microsoft.com/en-us/windows/win32/accessibility/accessibility-whatsnew
14. https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-howto-expose-serverside-uiautomation-provider 
15. Sample win32 custom control that implements IRawElementProviderSimple - https://github.com/microsoft/Windows-classic-samples/tree/main/Samples/UIAutomationSimpleProvider, https://github.com/microsoft/Windows-classic-samples/tree/27ffb0811ca761741502feaefdb591aebf592193/Samples/UIAutomationSimpleProvider 
16. https://www.linkedin.com/pulse/common-approaches-enhancing-programmatic-your-win32-winforms-barker/
17. https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/WinAuto/uiauto-serversideprovider.md
18. "UI Automation Provider Programmer's Guide" - https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/WinAuto/uiauto-providerportal.md
19. Control to patterns mapping - https://docs.microsoft.com/en-us/windows/win32/winauto/uiauto-controlpatternmapping