Creating and applying B2G OS update packages

Creating and applying an update is split into four steps:

1. Building incremental update packages from old version(s) to a new version on a build host.
2. Finding the right update package to download on the client.
3. Downloading the update.
4. Applying the update to existing files on the device.

Each of these steps are covered below.

Prerequisites

To build and apply updates you must ensure that your build has the updater and associated update tools enabled. By default those are enabled only in userdebug and user variants. You can however force building them by adding the following line to your .userconfig file:

export B2G_UPDATER=1

To sign update packages you will need a Java runtime environment (JRE) or Java software development kit (JDK) installed and the java command available in the default path.

Types of update

There are two types of updates to know about: FOTA (Firmware Over-The-Air) and Gecko/Gaia OTA (Over-The-Air). Let's look at the differences between them.

FOTA updates

We can update the entire B2G OS system through FOTA updates, the technology behind which is shared with the Android project. The locations on the phone's hard drive that can be changed using FOTA updates include the system partition, kernel, modem baseband, recovery image used for updating, or any other file on the device.

B2G OS does not depend on a particular FOTA client; the interface is abstracted through an API we call librecovery. However, we recommend using the GOTA recovery client (see below for more details), and the discussion here assumes GOTA is being used.

FOTA update packages mainly consist of a file called update.zip. This package consists of

• A set of binary diffs and new files required to update the client to the newer software version
• An "update script" that controls how the diffs and new files are loaded onto the client
• An internal signature used to verify the update package

This format and set of files are the same as those used in normal Android updates, except that B2G OS additionally wraps the update.zip package in a mar wrapper (MAR stands for Mozilla ARchive). This mar wrapper allows an additional level of verification, which is explained below.

Gecko/Gaia OTA updates

Alteratively we can update just the Gecko and Gaia files on a B2G OS device, through a mechanism we call Gecko/Gaia OTA updates. All of the Gecko and Gaia files — comprising the core Gecko runtime and the device's user interface — are in the /system/b2g directory on the device. This is the only directory that OTA updates can make changes to.

Gecko/Gaia OTA updates use the same technology that's used to update the Firefox desktop web browser. Much like the FOTA update.zip packages discussed above, OTA updates consist of a MAR file containing a set of binary diffs and new files needed to update the client to a newer software version.

The Gecko client verifies the integrity of MARs that it downloads, and MARs can be signed by multiple parties.

Why have two update technologies?

OTA updates are not as comprehensive as FOTA updates, but they are a lot more user friendly and easy to apply, and will often be fine for what you need to update:

• Gecko/Gaia OTA updates can be applied "in the background", while B2G OS continues to run normally. This provides a much better user experience because users don't need to reboot their phone and wait while an update is applied. Instead, the update is applied while the user continues to use the phone, and when the update is finished the user only needs to agree to restart the main b2g process. This takes a matter of seconds, instead of the minutes that are usually required to apply FOTA updates.
• Gecko/Gaia OTA update packages can sometimes be smaller than FOTA update packages, though not always; they should never be larger. This means that users can sometimes have less data to download.

Of course, if you need to update files outside Gecko/Gaia, you will have to go for the full FOTA package route.

Let's move on and examine the package building process.

Building update packages

Building updates is the process of generating the files needed to update B2G OS clients from version X of the software to a newer version Y. The update package that's needed to update the client depends on what files have changed between version X and version Y.

• If only files in /system/b2g have changed, we will generate a Gecko/Gaia OTA update
• If any files in a location outside of /system/b2g changed, we will generate a FOTA update

To generate an incremental update package (for both FOTA and Gecko/Gaia OTA updates), our tools require full builds of both version X and version Y. Full build means that the package includes all the files that are needed to Flash a client. When we produce a full build for version X, we don't know which future versions we will be updating to from versionX . Because of that, we build both full FOTA packages and Gecko/Gaia packages for each version. This allows us to generate either a Gecko/Gaia OTA incremental update, or a FOTA incremental update if needed, between version X and all future versions.

At a high level, the process of building an update looks like this:

1. With software version X
   • Generate a complete Gecko/Gaia OTA MAR for the contents of /system/b2g.
   • Generate a complete FOTA target files zip, optionally signing it, for the device partitions. The target files zip is referenced below as DEVICE-target_files-$VARIANT.$USER.zip, and is a zip containing the files to update the phone directories, including SYSTEM/, BOOT/, etc. A full FOTA update.zip can be generated from the target files zip.
2. With software version Y
   • Generate a complete Gecko/Gaia OTA MAR for the contents of /system/b2g.
   • Generate a complete FOTA target files zip, optionally signing it, for the device partitions. The target files zip is referenced below as DEVICE-target_files-$VARIANT.$USER.zip, and is a zip containing the files to update the phone directories, including SYSTEM/, BOOT/, etc. A full FOTA update.zip can be generated from the target files zip.
3. If only files in /system/b2g have changed, generate an incremental Gecko/Gaia OTA update MAR from version X to version Y.
4. Otherwise, generate an incremental FOTA update.zip from version X to version Y. Wrap the incremental FOTA update.zip in a MAR for delivery to the B2G client.
5. Sign the packages as required by delivery agreements.

The subsections below describe how to use B2G's tools to implement each of these steps.

Generating a complete Gecko/Gaia OTA update MAR

To generate a complete OTA update MAR from the last successful b2g build (e.g. that you built yourself), you need to invoke the gecko-update-full target. To place the MAR at $b2g/objdir-gecko/dist/b2g-update/b2g-gecko-update.mar, use the following commands:

$ cd $b2g
$ ./build.sh gecko-update-full
$ cp objdir-gecko/dist/b2g-update/b2g-gecko-update.mar <destination>

Generating a full FOTA update MAR

To generate a full FOTA update MAR from the last successful b2g build (e.g. that you built yourself), you need to invoke the gecko-update-fota-full target. This includes the contents of the entire /system partition. Here are the commands you need:

$ cd $b2g
$ ./build.sh gecko-update-fota-full

This will generate a ZIP file ($PRODUCT_OUT/fota/full/update.zip) and a MAR file ($PRODUCT_OUT/fota-$TARGET_DEVICE-update-full.mar). The ZIP file can be directly used with adb sideload, while the MAR is intended for distribution in the same manner as any other update package.

Generating a FOTA update MAR plus recovery package

As of Firefox OS 2.2 (mid April and beyond) we added a new make target, which can be invoked as follows:

$ cd $b2g
$ ./build.sh gecko-update-fota-fullimg

This is used to produce a recovery package that will dump a set of partitions images. The default set is controlled by the variable B2G_FOTA_FULLIMG_PARTS, defined in gonk-misc/Android.mk (along with most of the other new features seen below.) It's a space-separated string of mountpoint:image instances to include. The default value is "/boot:boot.img /system:system.img /recovery:recovery.img /cache:cache.img".

Along with this we have also introduced some new environment variables to control the production of the two other make targets — gecko-update-fota and gecko-update-fota-full:

• The first is B2G_FOTA_PARTS, which follows the same syntax pattern as B2G_FOTA_FULLIMG_PARTS. This allows us to produce these update packages but arbitrarily dump partition images along with those, e.g. boot partition, modem firmware, etc.
• B2G_FOTA_PARTS_FORMAT provides a way to describe a set of partitions that we want formatted during the installation of the recovery package. It's a space-separated list of mount points to make use of during the formatting.
• We also have two new variables that allow us to wipe caches and/or data during the build procedure:
  • B2G_FOTA_WIPE_DATA
  • B2G_FOTA_WIPE_CACHE

Generating a partial Gecko/Gaia FOTA update MAR

A partial FOTA update uses the same mechanism as a full FOTA update, but by default only includes Gecko/Gaia updates like a regular OTA update. Additional files outside of Gecko/Gaia (such as fonts) can also be included.

The rationale for generating a partial FOTA update package is mainly related to licensing issues: when building a complete FOTA update package, the whole system partition (at least) will be included. This may include blobs that you don't have the authorization to redistribute. However, since MAR distribution is useful and Gecko/Gaia themselves are free software, there is no reason we should not be able to distribute them in this manner. A partial FOTA allows you to only update a subset of the system. An OTA update could be used instead in this scenario but it does come at a cost: OTA updates require enough space on the system partition to hold both the existing Gecko/Gaia files as well as the unpacked update files. A partial FOTA update does not suffer from this limitation as it can overwrite the existing files with the updated ones.

To create a partial FOTA update from the last successful b2g build (e.g. that you built yourself), Invoke the gecko-update-fota target with the following commands:

$ cd $b2g
$ ./build.sh gecko-update-fota

This will generate a ZIP file ($PRODUCT_OUT/fota/partial/update.zip) and a MAR file ($PRODUCT_OUT/fota-$TARGET_DEVICE-update.mar). The ZIP file can be directly used with adb sideload, while the MAR is intended for distribution in the same manner as any other update package.

The construction can be controlled with a couple of environment variables, the most useful of which are documented below:

Generating a complete FOTA target files zip

Invoke the target-files-package target to build a target files zip that can be used to generate both incremental and full FOTA update packages. The target files zip can also be signed by custom keys to ensure that only FOTA updates from known sources can be installed. After signing target files, all images and updates (also OTA) need to be generated again to catch the inserted keys.

The following commands will carry out this step:

$ cd $b2g
$ ./build.sh target-files-package
$ cp out/target/product/$DEVICE/obj/PACKAGING/target_files_intermediates/$DEVICE-target_files-$VARIANT.$USER.zip <destination>

The variable values in the commands listed above should be filled in as follows:

Signing a complete FOTA target files zip

Proper releases should typically be signed by custom release keys only known to the vendor. Having such keys will prevent FOTA updates where the source is unknown from being installed, hence introducing an extra security layer. For this to work, the images flashed to a device need to include public keys while the updates need to be signed by the corresponding private key. 

The first step is to generate custom keys and store them in a safe place. The Android Open Source Project has a script for generating these keys. For full compatibility, get this script from the branch corresponding to the Gonk version of the device in question. Here is the master branch version.

A couple of keys are needed — create them with the following commands. releasekey is the key to use for signing FOTA update packages.

$ development/tools/make_key releasekey '/C=US/ST=California/L=Mountain View/O=Android/OU=Android/CN=Android/[email protected]'
$ development/tools/make_key platform '/C=US/ST=California/L=Mountain View/O=Android/OU=Android/CN=Android/[email protected]'
$ development/tools/make_key shared '/C=US/ST=California/L=Mountain View/O=Android/OU=Android/CN=Android/[email protected]'
$ development/tools/make_key media '/C=US/ST=California/L=Mountain View/O=Android/OU=Android/CN=Android/[email protected]'

With keys present, the target files zip can be signed using the following commands. This will insert the public keys and modify build properties to reflect the fact that it has been signed.

$ cd $b2g
$ ./build/tools/releasetools/sign_target_files_apks \
  --default_key_mappings $RELEASEKEY_FOLDER \
  --replace_ota_keys \
  --signapk_path prebuilts/sdk/tools/lib/signapk.jar \
  $UNSIGNED_TARGET_FILES_ZIP \
  $SIGNED_TARGET_FILES_ZIP

The variable values in the commands listed above should be filled in as follows:

Generating an incremental OTA update MAR

In this example, we're assuming that we're generating an update from software version X to version Y. The location of the full Gecko/Gaia OTA MAR built from software version X using the instructions above will be called $MAR_X below. This might be a path on a build server like /home/build/b2g/versions/X/update.mar. Similarly, the location of the full MAR built from version Y will be called $MAR_Y.

The tool build-gecko-mar.py will generate an incremental Gecko/Gaia OTA update MAR using $MAR_X and $MAR_Y. We'll call the destination of the generated file $GENERATED_INCREMENTAL_MAR_X_Y. Use the following commands for this step:

$ cd $b2g
$ ./tools/update-tools/build-gecko-mar.py --from $MAR_X --to $MAR_Y $GENERATED_INCREMENTAL_MAR_X_Y

Generating an incremental FOTA update zip

In this example, we're assuming that we're generating an update from software version X to version Y. The location of the full FOTA target zip built from software version X using the instructions above will be called $TARGET_FILES_X below. This might be a path on a build server like /home/build/b2g/versions/X/target_files.zip. Similarly, the location of the full FOTA target zip built from version Y will be called $TARGET_FILES_Y.

The tool build/tools/releasetools/ota_from_target_files will generate an incremental FOTA update.zip using $TARGET_FILES_X and $TARGET_FILES_Y. We'll call the destination of this intermediate file $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y.

After this update.zip is generated, the last step is to wrap it in a MAR for delivery to the B2G client. The tool tools/update-tools/build-fota-mar.p does this step. We'll call the destination of this generated file $GENERATED_INCREMENTAL_FOTA_X_Y.

Use the following commands to complete this step:

$ cd $b2g
$ ./build/tools/releasetools/ota_from_target_files -v \
    --incremental_from $TARGET_FILES_X \
    --signapk_path prebuilts/sdk/tools/lib/signapk.jar \
    --package_key $FOTA_SIGNING_KEY \
    $TARGET_FILES_Y \
    $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y
$ ./tools/update-tools/build-fota-mar.py $INTERMEDIATE_FOTA_UPDATE_FOTA_X_Y --output=$GENERATED_INCREMENTAL_FOTA_X_Y

The variable values in the commands listed above should be filled in as follows:

Hosting updates and polling for updates on the client side

B2G OS clients poll for updates by fetching and parsing an update manifest: update.xml. B2G OS clients are configured to poll for updates on specific servers — they query a specially-constructed path on the server. HTTPS is the recommended protocol that the client uses to query the server, however HTTP is also supported. The server and path polled by clients can be changed by shipping an update to existing clients that changes the polling code.

In the examples below, we'll assume that updates are hosted on the server updates.b2g.com.

The URL polled by the client commonly contains the following parameters:

The Firefox client uses the value of its configured update host and these values to construct a URL to poll at runtime. The structure is as follows:

https://aus4.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%PRODUCT_DEVICE%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml

A real example of such a URL is as follows:

https://aus4.mozilla.org/update/3/B2G/37.0a1/20141214040212/flame/en-US/nightly-b2g37/Boot2Gecko%202.2.0.0-prerelease%20%28SDK%2019%29/default/default/update.xml?force=1

If the server returns a "404 Not Found" in response to the client's request, then there is no update available. If the server returns a "200" and a manifest file, then there may be an update available. The manifest describes the newly available build; that is, the build the client would update to. An example manifest is:

<?xml version="1.0"?>
<updates>
  <update type="major" appVersion="19.0" version="19.0" extensionVersion="19.0" buildID="20121210123456"
          licenseURL="https://www.mozilla.com/test/sample-eula.html"
          detailsURL="https://www.mozilla.com/test/sample-details.html"
          isOSUpdate="true">
    <patch type="partial" URL="https://updates.b2g.com/release/unagi1/18.0/20121203123456/update.mar"
           hashFunction="SHA512" hashValue="5111e033875752b7d9b32b4795152dea5ef954cb8a9d4a602dd19a923b464c43521287dcb5781faf3af76e6dc5e8a3dd9c13edea18c1f2c8f3bd89e17d103d6f"
           size="41901319"/>
  </update>
</updates>

This follows the same schema as the B2G build manifest (see updates.xml Format for more details). The fields in the manifest describe:

• Metadata used to show a user interface on the client.
• Metadata about the newly-available version.
• The location of the update package.
• Metadata used to verify the download of the update package.

Using the mechanisms described above, servers can host update packages to update any old client version to the newest version. Or they may host only a "linear update history" in which clients must upgrade through a single path.

The details of the interaction between build servers and the update host is currently beyond the scope of this document. It is highly dependent on the production environment. You can find some more details on our Software Update wiki page.

Verifying and applying updates

After a B2G OS client has successfully polled for an update (handled from within the system), downloaded it, and verified the integrity of the downloaded update package, the final step is to apply the update.

The first step in applying an update is to verify the signatures embedded in the MAR packages (see Generating an incremental FOTA update zip for how these are created). This is done by the B2G OS client itself after checking the integrity of the downloaded package. The code used for this is the same for both FOTA and Gecko/Gaia OTA updates.

After signatures are verified, the process of applying an update diverges between Gecko/Gaia OTA updates and FOTA updates. Let's look at the differences between the two at this point.

Applying Gecko/Gaia OTA updates

The B2G OS client applies these using the updater binary. This is part of the Gecko distribution and is the same code used to apply updates to desktop Firefox. As described above, the update is applied while the B2G OS client continues to run normally. Users are able to make and receive calls, run apps, browse the web, etc. while updates are being applied.

The specific details of the updater binary are beyond the scope of this document, but it works approximately like so:

• It makes a copy of the /system/b2g files.
• It applies binary patches, removes old files, and adds new ones as specified by the MAR file.
• It restarts the main b2g process so that it uses all the new files.

After the b2g process finishes restarting, the user will be running the new version of the B2G client software.

Applying FOTA updates

The FOTA client applies these. The Gecko client "hands off" the update to be applied by calling into the librecovery API. What happens after this step is specific to each FOTA client.

In the implementation of librecovery used for the GOTA client, the downloaded update package is staged to be applied and special commands are enqueued for the recovery client. librecovery then reboots the device into recovery mode. The recovery client then runs the update script in the update.zip to update files and partitions as needed. The recovery client may need to reboot multiple times in order to update all files.

After the final reboot, the device will be running the new version of the B2G OS client software.