Accessibility/AT-SPI Support

Mozilla AT-SPI Support for Linux/UNIX Assistive Technology Developers

This FAQ explains how makers of Linux- or UNIX- based screen readers, voice dictation packages, onscreen keyboards, magnification software and other assitive technologies can support Gecko-based software. The base of our support for these products is AT-SPI (Assistive Technology Service Provider Interface) and the keyboard API/user interface.

For Firefox and all other Gecko-based products: this documentation only applies to up-to-date builds of Firefox -- currently not available on official releases. Grab the current build of Firefox which supports these features.

To Do

• Document dynamic content, such as how to track mutations
• Document text selection
• Update list of supported features for each role
• Go through XXX notes and file bugs or deal with questions
• Add DHTML examples
• Go through Bill's notes in HTML examples and remove or integrate
• Explain embedded object characters and other new things in https://www.mozilla.org/access/unix/new-atk
• Describe how to know where forced and soft line breaks are
• Fix HTML Examples table, the wiki format is messed up

Definitions

Here are some basic definitions that you'll need for this document to make sense:

Gecko:

The rendering engine for Firefox, Thunderbird, Nvu, Mozilla Seamonkey and other applications. Gecko is the internal engine that Mozilla uses to render any kind of web content. It supports HTML, XHTML, Cascading Style Sheets (CSS) and the Document Object Model (DOM).

Assistive Technology Service Provider Interface (AT-SPI)

an API devised by Sun Microsystems so that accessibility aids can track what's going on inside the user interface of any software package that supports it. If you seriously need to understand AT-SPI, you'll need to read the docs on gnome.org and play with the available sample apps and code, such as at-poke. Please note that the web docs are sometimes out of date, and the latest AT-SPI is available on CVS.

DOM: Document Object Model

This is the W3C's specification for how web content is exposed to Javascript and other languages. It covers content, style and events. Inside the Gecko process, code has full access to DOM APIs. However, exposing the entire DOM to external software packages is quite involved, partially because changes to the DOM in Firefox must occur on the main thread. We have chosen a subset of readonly methods in the DOM needed for assistive technology vendors. Events such as focus changes must be tracked through AT-SPI events, rather than DOM events.

XUL: eXtensible User-interface Language

The XML-based language used by Firefox and Mozilla to develop the UI. Similar to HTML in that it can be combined with CSS and Javascript to make powerful applications. Contains more desktop-style widgets than HTML and follows a box layout model, rather than being text-flow based.

AJAX: Asynchronous JavaScript And XML

AJAX is a method of building interactive web applications that process user requests, user actions immediately in real time, unlike an HTTP request, during which users must wait for a whole page to reload or for a new page to load. Data is therefore stored and retrieved dynamically much faster.

Roles, states and events

please read the AT-SPI documentation if you are unfamiliar with these.

 

AT-SPI tree vs. DOM tree - what's the relation?

The AT-SPI tree and the DOM tree are parallel structures, although the AT-SPI tree is a subset of the DOM tree. QueryInterface() can be used to switch between the interfaces (Accessible, AccessibleText, AccessibleValue, etc.).

What is exposed? Any DOM node that ...

• Is focusable
• Is formatted with a blank line before and after (display:block in web jargon)
• Conveys important information about the structure of the document, such as a heading
• Contains a value that can change
• Uses a dynamic content role attribute that is not equal to "presentation"
• Uses one of the universal dynamic content property attributes, which are currently aaa:describedby, aaa:labelledby, aaa:required or aaa:invalid
• Uses an onclick handler
• Is in list of accessible elements in the role table

Linux/UNIX Applications Based on the Gecko Layout Engine

Gecko is a rendering engine that Firefox, SeaMonkey, Netscape and yelp use. Gecko can render a variety of content, not just HTML and supports key web standards such as Cascading Style Sheets, Javascript and the W3C DOM. Gecko also handles the users keystrokes and mouse clicks. Gecko is the core architecture that we are adding accessibility to, in order to support basic accessibility in all applications that are based on it.

The Mozilla Gecko engine must be version 1.9 or later to have high quality AT-SPI support, as documented here.

Embedded Clients

Embedded clients use Gecko only in the content window, at the moment for HTML and generic XML only. They typically use standard Windows controls for their user interface -- the area outside of the client content window, plus the context menu.

• Yelp help viewer. No version is yet known to be available that uses a recent enough version of the Mozilla Gecko engine, for quality AT-SPI support.
• Evolution email: no version is yet known to be available that uses a recent enough version of the Mozilla Gecko engine, for quality AT-SPI support.
• Many more applications exist and are expected to be released

XUL-Based Clients

XUL-based clients make full use of the Gecko architecture, not only for HTML content, as well as for menus, dialogs and the entire user interface via an XML language called XUL (eXtensible User-interface Language). None of the user interface contains standard Windows controls -- not even the menus! This is done to ensure a common look and feel across all supported platforms, and to allow for different skins (appearances).

• Firefox browser (please use recent alpha builds for testing)
• Thunderbird email (please use recent alpha builds for testing)
• Seamonkey suite (please use recent alpha builds for testing)
• Nvu HTML editor: no version is yet known to be available that uses a recent enough version of the Mozilla Gecko engine, for quality AT-SPI support.
• Any Xulrunner application (Songbird media player, etc.)
• Many more applications exist and are expected to be released

Determining if Accessibility is Enabled in a Firefox Installation

See the about:accessibilityenabled Firefox extension.

How to Find the Content Window and Load the Document

In XUL-based clients, screen readers may need to find the content window so that they know where to start grabbing the AT-SPI tree, in order to load the current document into a buffer in their own process.

Use RELATION_EMBEDS on the ROLE_FRAME, which is at the root of each top level window. This will point to the root content accessible(s) for that XUL window.

When you see the content window receive focus, first check the role. If it is a ROLE_DOCUMENT_FRAME then this should be treated as a document for the default modality of the screen reader. If it is a ROLE_EMBEDDED, ROLE_DIALOG or ROLE_ALERT then stay in focus tracking mode -- there is no need to parse the document. In addition, if it is a ROLE_ALERT, a screen reader should treat it as a message box -- that is, to read the entire contents. These roles can occur on content because of the new Accessible DHTML technology which allows the author to specify the type of document or container.

Gecko also helps determine when to load a new window by firing the following document events: document:load-complete, document:reload and document:load-stopped

Supported AT-SPI Interfaces

Supported AT-SPI Roles

The following roles are currently supported:

• XXX Needs updating -- this list is copied from MSAA document

Supported AT-SPI States

The following states are currently supported:

Supported AT-SPI Relations

The following relations are currently supported:

enum {RELATION_LABEL_FOR = 0x1002 };
enum {RELATION_DESCRIPTION_FOR = 0x100f };

These two relations can be used on object to determine what form control is being labelled or desribed.

enum {RELATION_LABELLED_BY = 0x1003 };
enum {RELATION_DESCRIBED_BY = 0x100e };

These two relations are they inverse; they can be used on form controls. If the form control has an accName, you can get the IAccessible that it was labelled by in order to get more formatting information. It is also useful to check for a description.

Note that the label and description relations may be used to prevent redundant information from being presented by the screen reader, since the label and description can occur both on their own, and in the name or description fields of an IAccessible.

enum {RELATION_EMBEDS = 0x1009 };

This relation is used on the root accessible object for a top level Mozilla window, corresponding to what's returned for OBJID_CLIENT for that window. It points to the accessible object corresponding to the root of content in that window. This relation is very useful for finding the content quickly, and will be the proper method for finding content in Firefox 3 and beyond.

enum {RELATION_CONTROLLED_BY = 0x1000 };
enum {RELATION_CONTROLLER_FOR = 0x1001 };

These two relations show what form controls may dynamically change areas of the document, in response to user changes in the form controls themselves. Both controlled_by and controller_for are set in markup from the single dynamic content accessibility dynamic content aaa:controls attribute. The inverse controlled_by relation is automatically calculated.

enum {RELATION_FLOWS_TO = 0x1006 };
enum {RELATION_FLOWS_FROM = 0x1007 };

These two relations allow the reading flow to break out of the normal DOM flow. Both flows_to and flows_from are set in markup from the single dynamic content accessibility aaa:flowsto relation -- the inverse flow_from relation is automatically calculated.

Supported AT-SPI Object Attributes

The following object attributes are currently supported:

Note: see AJAX:WAI_ARIA_Live_Regions/API_Support for additional special object attributes that improve the processing of live changes.

Supported AT-SPI Text Attributes

The following text attributes are currently supported:

Supported AT-SPI Document Attributes

The following document attributes are currently supported:

Supported AT-SPI Events

The following events are currently supported:

• object:focus
• object:property-change ... *** XXX fill in list of properties we support this for ***
• object:state-changed for ... *** XXX fill in list of states we support this for ***
• object:text-changed:delete
• object:text-changed:insert
• object:text-selection-changed
• object:text-caret-moved
• object:selection-changed
• window:activate
• window:deactivate
• document:load-complete
• document:reload
• document:load-stopped
• document:attributes-changed

The event object:link-selected is purposely not supported, but focus events are fired on ROLE_LINK objects instead.

AT-SPI Features We Currently Do Not Support

As just stated, the event object::link-selected is purposely not supported, but focus events are fired on ROLE_LINK objects instead.

See Firefox 3 dependencies, and in particular the New ATK blocking bugs.

Known Differences with Other Applications

*** XXX To Be Done ***

Avoiding Memory Leaks

It is the assistive technology's responsibility to watch for events that indicate when windows or content subtrees are being destroyed, and to release all Accessible objects related to that window. In addition, STATE_DEFUNCT is set on objects that should be released by the assistive technology. XXX *** Indicate what events, I think we need to add object:children-changed:foo support ***

To help developers in that regard, there is Memory Leak monitor, a Firefox extension.

Keyboard User Interface and API

Fortunately, Gecko uses the standard keyboard API's for each supported platform.

The Mozilla keyboard shortcuts for content are similar to what is used in other browsers. Here is a list of Firefox keyboard shortcuts.

HTML Examples

       text="You are a nice

       attribute run for

       text="Here is a *."
       attribute run for "bartending site."

        {child

           text="*bartending site" 
 

     [not sure if we need to dup textattrs

            
       URI="beerglass.GIF"}
     [don't know if the URIs should

       At the moment, not planning to do this, instead plan
         to expose repair text in the name if no alt/title exists ] 

     * This is a list item.
     * This is another list item.

     
       [Bill: we

      <label for="beverage">Make a selection:</label>
       <select id="beverage">
       <option>Water</option>
       <option>Wine</option>
       <option>Whiskey</option>
       </select>
     

Beyond HTML: Other Types of Web Content

• You may have heard of some content types beyond HTML, and want to know if Gecko based products will support them:
• DHTML and AJAX: Gecko and other browsers have long supported dynamic content, where the page appearance changes because of JavaScript. This can be used to create the appearance of desktop-style widgets like menus, spreadsheets and tree views which HTML lacks. Or, it can be used to completely change content on the fly, without loading a new page. Previously it was not posible to make this accessible, but Mozilla supports Accessible DHTML, which allows authors to make advanced widgets and web applications accessible.
• MathML: an XML dialect used to display full math notation on web pages. Here's the ongoing Mozilla project to support MathML . We do not currently have plans to support MSAA for MathML - we will likely suggest use of the external DOM for this, because MSAA simply does not have the right semantics in it to support mathematics. MathML support is currently built into Firefox.
• SVG : Scalable Vector Graphics. Essentially W3C's XML-based version of Flash - mixes well with other markup based content, supports the DOM and has accessibility features. Mozilla's SVG Project has been through several stalls and rebirths -- SVG will probably become more of a priority if other well known browsers support it. We will not know how, or whether, we will support SVG accessibility until a better implementation comes along. Firefox 1.5 supports SVG. Potentially it can make use of the same namespaced role and state attributes as DHTML accessibility, but it may require more powerful author-definable relationships.
• XForms: XForms is the future of online forms as envisioned by the W3C. Drawing on other W3C standards like XML Schema, XPath, and XML Events, XForms tries to address some of the limitations with the current HTML forms model. One of the key features of XForms is accessibility. XForms accessibility development is currently being developed for Mozilla 1.9 (Firefox 3) by Alexander Surkov.
• XUL: The XML-based language used by Firefox and Mozilla to develop the UI. Similar to HTML in that it can be combined with CSS and Javascript to make powerful applications. Contains more desktop-style widgets than HTML and follows a box layout model, rather than being text-flow based. In the future more standalone applications will use XUL via Xulrunner.

Questions or Comments?

Please discuss accessibility issues on the Mozilla Accessibility mailing list or on the Mozilla Accessibility IRC channel.