TV リモコンのナビゲーションを実装する

User experience design for TVs is very different to smart phones. The screen is much larger, but users have to navigate apps using a TV remote control — other more familar mechanisms such as mouse or touch screen are not available. This article explains how the controls in Gaia's TV apps were developed.

マウスを使わずにフォーカスをあてたり視覚を扱う

When the only user navigation is via arrow keys on a remote control, an app has to decide by itself which element should be highlighted next — above the current position, below it, etc. The app also needs to record which element the cursor stops on, to focus it. Such a mechanism is called Spatial Navigation. To implement this, we have to consider the following specifics:

1. How to choose the next element from a number of nearby DOM elements without explicit context.
2. How to trigger the function registered on the currently-focused element when users press the corresponding key (e.g. the <kbd>ENTER</kbd> key to trigger onclick.)

We’ll talk about the first point later on — the Existing libraries section discusses available out-of-the-box libraries inside Gaia, and the Algorithm design section talk about the implementations of the algorithms used in those libraries.

Turning our attention to the second point, we can place the <kbd>ENTER</kbd> key EventListener to identify the current focus, and then call the handler to be executed. This operating way is similar to delegation. Here is the event listener used in Home app:

onEnter: function() {
    ...
    var focusElem = this.focusElem;
    ...
    if (focusElem === this.settingsButton) {
      this.openSettings();
    } else if (focusElem === this.editButton) {
      ...
      this.edit.toggleEditMode();
      ...

The main logic is clear: when the current focus stops on the settingsButton or editButton, we call openSettings() or toggleEditMode() respectively; that is, we simply call the corresponding handler. If there are many objects, we can directly dispatch custom events to trigger EventListeners for each element, which can avoid lengthy enumeration of handlers using if..else if.

既存ライブラリ

Some libraries for keyboard navigation are included inside the Gaia repo. SpatialNavigator and SimpleKeyNavigation keep track of which element on the screen is currently being focused. KeyNavigationAdapter is an event wrapper for arrow keys. Let's explain them in a little more detail.

SpatialNavigator

This library provides a generic algorithm to navigate among a collection of elements. It keeps track of a set of elements and the “currently focused element”, and you can send it a direction (left, right, up, or down) to navigate to other elements close to the current focus. We use “getBoundingClientRect” to determine the real positions of elements on the screen. You can just ask the library to find the next target for you automatically, however sometimes the target may not be the one you expected. In these cases, you have to carry out an “except” condition before calling the navigation algorithm of the library. We will talk about this algorithm in the next section.

SimpleKeyNavigation

Although this library stores an array of elements, it only allows navigation in one dimension at a time (horizontal or vertical.) The order of focus is determined only by the position of elements inside the array rather than their real position on the screen. It also listens to key events automatically. If your user interface is simple, this can be a suitable lightweight choice for navigation.

This library is most suitable for small amounts of elements whose relative positions don't change. It's commonly used for modal dialog button sets, such as alert and confirm dialogs. 

KeyNavigationAdapter

This wrapper class listens to raw key events and organizes them to be handled by higher-level custom events, allowing apps to handle them more efficiently. For example, the keydown events of the four arrow keys are combined into a single move event while the corresponding keyup events become a single move-keyup event. The arguments passed to the event handlers of these two custom events are exactly the same as what SpatialNavigator accepts in its own methods, allowing easier integration of these two libraries in your app.

アルゴリズム設計

Compared with general web pages, it’s much more important for an app to have a highly customizable spatial navigation function because of its complex behaviors. For instance, some authors may prefer moving the focus among elements based on some specified order or intend to adjust the order dynamically. The existing one in Fennec doesn’t do enough for our use cases on TV, so we’ve already filed a meta bug (バグ 1114536.) Before this work is completed, we still need an alternative function to help us develop apps on TV. That’s why we implemented SpatialNavigator.js in Gaia. Let’s look into how it works.

SpatialNavigator maintains a list of focusable elements. You can pass them in at the initialization stage and modify them later as needed. In addition, the module also keeps track of the currently-focused element.

Let’s look at an example — we have five elements, one of which is currently focused (the blue element with the yellow frame represents), and the other four which are to the right of the first one in slightly different places. We split the area around the currently-focused element into 9 slices:

0. Top left.
1. Top.
2. Top right.
3. Left.
4. The currently focused element.
5. Right.
6. Bottom left.
7. Bottom.
8. Bottom right.

When you query SpatialNavigator.js to find out what element to move to if the <kbd>RIGHT</kbd> arrow key is pressed, it will first work out what focusable elements are to the right of the currently-focused element. First, it lengthens the right hand edge of the currently-focused element.

Refering to the numbers in the picture, we all agree that elements located in areas 2, 5 and 8 (top right, right, and bottom right slices) should be considered "on the right". The library calculates the center point of each focusable element in the list and filters the elements with center points located the slices mentioned above (element C is considered to belong to slice 5 since its center point is close to its edge.)

The most important part is how to determine the weight of the candidates and sort them.

In the module, we define three priority levels:

1. Higher priority is given to elements with centers inside the slices that are adjacent to the currently-focused element (slice 5 in this case), than to slices that are diagonally placed (2 and 8).
2. Next, for the adjacent slices we calculate the distance between the closest edge of each candidate (left side in this case) and the nearest edge of the currently-focused element (right in this case) — the red lines in the graphic. The shorter distance, the higher the given priority.
3. Last, the lowest priority — for the adjacent slices we calculate the shortest distance between the edge of each candidate and the horizontal boundary line of the slice to which the candidate belongs — the blue arrows in the graphic. If there are two boundary lines needing to be considered, the upper one or the left one will be chosen first. The same principle is used here: shorter distance means higher priority.
4. For candidates in diagonal slices, priority levels 2 and 3 are awapped over — therefore a shorter vertical distance gives a higher priority than a shorter horizontal distance for diagonals (this is why the order given below is ... A, D and not ... D, A.)

The priority order we end up with is B, C, A, D — the focus will be moved to element B when you press the RIGHT arrow key.

Algorithm issues and improvements

The algorithm above is our first version. However, we encountered a problem when it went live, with situations where you have a large element close to the currently-focused element, but whose center doesn't sit inside the adjacent slice (A in the diagram), and a small element further away, whose center does sit inside the adjacent slice (B in the diagram):

We expect the focus to be moved to element A when pressing the <kbd>DOWN</kbd> key from the currently-focused element, but unfortunately the focus jumps to element B instead — this isn’t intuitive.

Let's look at a similar situation where the <kbd>DOWN</kbd> key is pressed and discuss a potential solution. We altered the algorithm to give equal priority to every element overlapping the adjacent slice — regardless of how much they overlap. However then we encountered another problem. If for example you have two elements below the currently-focused element, one that largely overlaps the adjacent slice but is slightly further away (E in the diagram), and one that only slightly overlaps the adjacent slice but is slightly closer (D in the diagram), the former gets a higher priority and is focused next. Again, this isn't what users would expect — you'd expect E to be focused before D.

To combat this, we added a threshold to determine whether we should treat an element as a candidate or not — a customizable property to adjust the minimum percentage by which an element needs to overlap the adjacent slice before it is granted the associated higher priority.

Example

Now we've discussed the technology we've made available for implementing spatial navigation, let's looks at an example implementation. This section discusses the HOME app from Gaia.

Including the libraries

First, the app includes spatialNavigator and keyNavigationAdapter — this comes from index.html:

...
<!-- Shared TV library for keyboard-based navigating -->
...
<script defer src="shared/js/smart-screen/spatial_navigator.js"></script>
<script defer src="shared/js/smart-screen/key_navigation_adapter.js"></script>
<!-- Specific code -->

(It also partially uses simpleKeyNavigation, but we won't discuss this further.)

Initialization

When activating the app, we initialize the two libraries (see home.js):

...
init: function() {
  ...
  var collection = that._getNavigateElements();
  that.spatialNavigator = new SpatialNavigator(collection);
  that.spatialNavigator.straightOnly = true;
  that.keyNavigatorAdapter = new KeyNavigationAdapter();
  that.keyNavigatorAdapter.init();
  that.keyNavigatorAdapter.on('move', that.onMove.bind(that));
  // All behaviors which no need to have multple events while holding the
  // key should use keyup.
  that.keyNavigatorAdapter.on('enter-keyup', that.onEnter.bind(that));
  ...
  that.spatialNavigator.on('focus', that.handleFocus.bind(that));
  that.spatialNavigator.on('unfocus', that.handleUnfocus.bind(that));

First we call _getNavigateElements(), which returns an array containing HTMLElement and is stored in the variable collection. Second, we send the array to the initialization parameter of SpatialNavigator mentioned before. This tells SpatialNavigator which elements are focusable. We also assign straightOnly as the preference for judging which elements will considered candidates for spatial navigation — diagonally placed elements will not be listed as candidates. You can refer here for options other than straightOnly.

Defining the elements to be navigated

Let’s look at the content of _getNavigateElements():

  navigableIds:
      ['search-button', 'search-input', 'settings-group', 'filter-tab-group'],
  navigableClasses: ['filter-tab', 'command-button'],
  ...
  _getNavigateElements: function() {
    var elements = [];
    this.navigableIds.forEach(function(id) {
      var elem = document.getElementById(id);
      if (elem) {
        elements.push(elem);
      }
    });
    this.navigableClasses.forEach(function(className) {
      var elems = document.getElementsByClassName(className);
      if (elems.length) {
        // Change HTMLCollection to array before concatenating
        elements = elements.concat(Array.prototype.slice.call(elems));
      }
    });
    elements = elements.concat(this.navigableScrollable);
    return elements;
  },

_getNavigateElements() has two forEach loops that iterate through the navigableIds and navigableClasses arrays respectively. The former uses HTML object IDs to indicate which objects should be spatially navigated; the latter does the same thing with HTML object classes.

Receiving key events and finding focus targets

Back in init(), we initialize keyNavigatorAdapter and register two event listeners, move and enter-keyup, to receive keyboard events. The move event will be triggered by pressing one of the <kbd>UP</kbd>, <kbd>DOWN</kbd>, <kbd>LEFT</kbd>, or <kbd>RIGHT</kbd> keys, taking the appropriate left, right, up, and down string as a parameter in each case. We can now just call spatialNavigator in the event listener to find the next focus target:

  onMove: function(key) {
    ...
    var focus = this.spatialNavigator.getFocusedElement();
    if (!(focus.CLASS_NAME == 'XScrollable' && focus.move(key))) {
      this.spatialNavigator.move(key);
    }
  },

Here we first find out the currently-focused object by using spatialNavigator.getFocusedElement(). The HOME app includes a XScrollable object which is an object has its own spatial navigator and manages the element set by itself, so we just hand out the keyboard event to this XScrollable object once it got focus. We skip this specific situation and directly send the key parameter into spatialNavigator.move() to find the next object. After finding the next object, spatialNavigator.move() triggers the focus event:

  handleFocus: function(elem) {
    if (elem.CLASS_NAME == 'XScrollable') {
      this._focusScrollable = elem;
      elem.focus();
      this.checkFocusedGroup();
    } else if (elem.nodeName) {
      switch(elem.nodeName.toLowerCase()) {
        case 'menu-group':
          this.handleFocusMenuGroup(elem);
          break;
        default:
          elem.focus();
          this._focus = elem;
          this._focusScrollable = undefined;
          this.checkFocusedGroup(elem);
          break;
      }
    ...
  },

The code above executes the necessary actions for different focused elements. We'll not say much more for brevity.

Dynamically changing object collections

With these event being handled, we've made a prototype of a TV app. We have not mentioned about dynamically and programmatically changing object collections of spatial navigations. Here is a brief example.

In the HOME app there are menuGroup elements that contain hidden subitems (developed as Web Components — these can be regarded as HTMLElements with custom behaviors.) We want the following behaviors to happen:

1. The user navigates to a menuGroup.
2. The menuGroup expands and shows its sub-items.
3. The sub-items should be added as targets for spatial navigation, while the menuGroup should be removed from the list of targets.

The third step is handled using the following code:

  handleFocusMenuGroup: function(menuGroup) {
    var self = this;
    menuGroup.once('opened', function() {
      self.spatialNavigator.remove(menuGroup);
      var childElement = menuGroup.firstElementChild;
      var firstFocusable = null;
      while(childElement) {
        switch(childElement.nodeName.toLowerCase()) {
          ...
          default:
            firstFocusable = firstFocusable || childElement;
            self.spatialNavigator.add(childElement);
        }
        childElement = childElement.nextElementSibling;
      }
    ...
    if (firstFocusable) {
      self.spatialNavigator.focus(firstFocusable);
    }
 }

After expanding, the menuGroup fires an opened event, triggering the callback function. In the callback function, we first call spatialNavigator.remove(menuGroup) to remove menuGroup from the navigation target list, then we use a while loop to cycle through all the child elements of menuGroup and add them as navigation targets with spatialNavigator.add(childElement). The switch case is used to exclude some exceptions that should not be added.

Finally, we call spatialNavigator.focus(firstFocusable) to programmtically move focus to the first item. This call will trigger spatialNavigator to fire focus events and finally call the handleFocus() function mentioned before.