Interacting with TV remote controls

One of the most notable differences between phones and TVs is the available input mechanisms. In general, a TV doesn’t include a touch screen, so apps can’t be controlled by mouse events or touch events. Instead, your apps will probably need to be operated by button events sent by the TV when its remote control buttons are pressed. This article discusses the basics of how to develop appropriate control mechanisms for your TV app.

Focus

Focus is a major issue that needs to be highlighted when implementing controls for TV apps. It’s important to make sure that the focus is on the right element at all times, since you can’t touch elements in a TV app directly.

When we decided to build B2G for Smart TVs, the first issue we solved was the UI controls — unlike a desktop computer of touchscreen phone, a TV generally has no touchscreen or mouse available. The experience is more like an older feature phone, or controlling web browsing on desktop with only the keyboard — you've got several buttons available on the TV remote control that can be used to provide input. We wanted to redesign the UI for B2G on TVs, and carefully considered the controls as part of this.

Spatial navigation

Generally, such keyboard/button manipulation is called Spatial Navigation. Most people are quite familiar with this mechanism: we always use arrow keys to move around focusable elements on feature phones, and many games console browsers follow the same pattern; Android browsers (including Firefox for Android) generally have the same mechanism available.

Simple spat nav in Firefox for Android

To implement the mechanism, we first studied the existing implementation in Firefox for Android. The source code is quite easy to find, but unfortunately the functionality is too simple to be customized; it is an advanced “Tab” key switching function only. This is not a criticism: we understand that the purpose of supporting Spatial Navigation on Firefox for Android is to make sure every page is “basically workable.” In other words, Firefox for Android doesn’t have to optimize keyboard manipulation; developers rarely create web pages with TV optimization in mind.

However, if we wanted to create TV apps with web technologies, such as the Home screen, Application List, EPG (Electronic program guide), Settings, and so on, “basically workable” is not enough. For example, let’s take a look at the Home screen on a Firefox OS TV.

Based on the UX design, when a user is currently focusing on “Devices” app in the middle of the screen, the "Search" (magnifying glass) icon should be focused next after pressing the Up arrow key. According to the algorithm on Firefox for Android, however, the “Settings" (gear) icon to its right is the next focusing target. As another example, when “Search” is focused, pressing the Down arrow key should move the focus back to “Devices.” However, Gecko will (theoretically at least) always focuses on the nearest icon, which is not what we want.

Directional focus navigation

To solve the issue, we turned to the CSS Basic User Interface Module Level 3 — its Directional focus navigation functionality is what we need. Let’s take a look at how this works:

button { position:absolute }

button#b1 {
  top:0; left:50%;
  nav-right:#b2; nav-left:#b4;
  nav-down:#b2; nav-up:#b4;
}

button#b2 {
  top:50%; left:100%;
  nav-right:#b3; nav-left:#b1;
  nav-down:#b3; nav-up:#b1;
}

button#b3 {
  top:100%; left:50%;
  nav-right:#b4; nav-left:#b2;
  nav-down:#b4; nav-up:#b2;
}

button#b4 {
  top:50%; left:0;
  nav-right:#b1; nav-left:#b3;
  nav-down:#b1; nav-up:#b3;
}

The draft defines four CSS properties: nav-left, nav-right, nav-up, and nav-down. With these properties, you can specify the next element that should be focused when you are focused on a certain element, and move in a certain direction with spatial navigation. Or you can specify auto, which hints to the browser to calculate the correct elements to move to. The specification also defines how to handle moving the focus to elements inside embedded frames.

There is one problem here — the specification is still a draft, and is not yet implemented by most modern browsers (including Gecko).

Even if this functionality was available, some secondary questions would still remain. Going back to the Home screen example, we can of course direct all apps to “Search” with nav-up. However, because the last focused app may vary, we cannot pre-define the nav-down property of "Search", for example. Moreover, if a modal dialog pops up on the screen, we should prevent the focus from moving out of the dialog even though the elements beneath are still visible.

A JavaScript solution: spatial_navigator.js

The above constraints led us to using JavaScript to implement this functionality, and dynamically change the values of nav-up/down/left/right in different contexts. Prior to specifying elements' directional focus with JavaScript, we might need another API to retrieve focus elements based on the same algorithm as the auto value discussed above. Only with this API can we maintain consistancy when combining JavaScript interactions with CSS rules. Unfortunately, the API doesn’t exist yet (bug 449165).

Since we have to write some JavaScript code, why not imitate the aspects of nav-* and implement a library with pure JavaScript? In this way, web pages can support spatial navigation right away before the API is ready.

We implemented a spatial_navigator.js library in the Gaia code base. For a currently focused element, this simple library calculates the nearest element in all four directions. We first applied the library to the TV Home screen, which was under development. During the testing iteration, we gradually improved its navigation algorithm to meet users’ expectations. For more information on the algorithm, please refer to the MDN documentation.

Since this library was originally designed for Gaia only, it was not fully compatiable with other browsers due to usage of ES6 syntax and list dependency on Gaia libraries. Therefore, we came up with a refactoring plan to make it more cross-browser compatible — the result is the js-spatial-navigation library.

Briefly, js-spatial-navigation can be defined as a cross-browser wrapper for the navigating algorithm used in spatial_navigator.js.

A simple example

Let's take a look at a basic example that demonstrates the simplest way of using js-spatial-navigation.

To prepare our example page, let's add some elements into the <body></body> that we want to be navigable. We will add a focusable class to first two <div>s in order to seperate them from the unnavigable <div> element (the last one).

<a href="#">Link 1</a>
<a href="#">Link 2</a>
<div class="focusable">Div 1</div>
<div class="focusable">Div 2</div>
<div>This is not navigable</div>

We need to define styling for each navigable element when focused, so users know which item they are focussing on. Because users are not using the mouse, we have to make the focused element extra easily-identifiable, or they will get lost easily after pressing an arrow key. The styling is as follows:

:focus {
  outline: 2px solid red;
}

After the preparation, we can start to set up js-spatial-navigation. The following example code works only if all navigable elements are rendered, so we recommend that you include this code in a DOMContentLoaded or load event handler.

Now we can move on to implementing the JavaScript.

1. Initialize SpatialNavigation to bind the keyboard events, so it starts to react to the arrow keys.

   SpatialNavigation.init();

2. Define which elements are navigable. The example below demonstrates how to make all <a> elements and other elements containing the focusable class navigable. We add a rule via SpatialNavigation.add(). The selector property is used for filtering the elements you want to navigate. In addition to selector, you can also set other configuration items here. Please refer to this GitHub documentation for more details.

   SpatialNavigation.add({
     selector: 'a, .focusable'
   });

3. <div> elements are not focusable by default — we have to assign a tabindex attribute. To save you the trouble of having to add these attributes manually, you can use the SpatialNavigation.makeFocusable() helper function:

   SpatialNavigation.makeFocusable();

4. Finally, you may want to assign an element that is initially focused when the app is first loaded. SpatialNavigation.focus() can accept an element as its first argument and focus it immediately. If you omit the argument, SpatialNavigation will focus the first navigable element matching the selector defined above.

   SpatialNavigation.focus();

The complete code looks like this:

<head>
  <script src="spatial_navigation.js"></script>
  <script>
    window.addEventListener('load', function() {
      SpatialNavigation.init();
      SpatialNavigation.add({
        selector: 'a, .focusable'
      });
      SpatialNavigation.makeFocusable();
      SpatialNavigation.focus();
    });
  </script>
  <style>
    :focus {
      outline: 2px solid red;
    }
  </style>
</head>
<body>
  <a href="#">Link 1</a>
  <a href="#">Link 2</a>
  <div class="focusable">Div 1</div>
  <div class="focusable">Div 2</div>
  <div>This is not navigable.</div>
</body>

You can find more examples on GitHub.

The library introduces the concept of "section", which can separate all focusable elements on the screen into different scopes and apply different navigating rules to each section. For example, modal dialogs can be an independent section with a limited navigation scope.

Other uses

As well as being used in the key built-in apps in the first version of Firefox OS TV, and being useful for building general web apps, we also used it in the following TV projects.

TV marketplace

The TV-specific marketplace website uses our spatial navigation library to implement the keyboard navigation. Because the library is cross-browser compatiable, you can navigate it with various desktop browsers smoothly.

TV settings app

Meanwhile, for developing and testing purposes, we reworked the Firefox OS phone Settings App to correctly work on TV before building the TV-specific version, using js-spatial-navigation for the controls. The following video shows the real interactions. You can also experience it by checking out Gaia master branch and building the TV profile on your own computer.

A "Youtube" prototype?

During development, we tested the library out by imitating the UIs of existing TV optimized websites — this is helpful for discovering shortcomings so we can improve on them and make the library better. The following is a Youtube TV UI-like demo page (not completed yet). You can find the source code here.

Conclusion

We believe JavaScript-based spatial navigation is just transitional functionality. In the long term, browsers should natively support Spatial Navigation-related APIs. For now, we have made js-spatial-navigation available to help anyone with similar problems to ours. The library still has a long way to go; if you have any ideas or feedback, or wish to contribute to the project, please get in touch with: https://github.com/luke-chang/js-spatial-navigation.

For more information and example code, please refer to the Implementing TV remote control navigation documentation on MDN.