Enhanced Extension Installation

Background

There are several flaws with Extension¹ Installation in Firefox² 1.0, including:

• It is very difficult for a third party application with its own managed install process to install an Extension into Firefox. First it must locate the Firefox executable, then run it with the -install-global-extension command line flag, which installs from a XPI into the Firefox application directory. Aside from the work of locating the Firefox executable in the first place (which varies from platform to platform), this is very limiting because:
  1. It forces the third party application to package its Firefox integration hooks as a XPI.
  2. It forces it to have write access to the Firefox directory in order to be installed, which may not always be the case.
  3. It forces its items to be located in different places on the user's disk - some vendors wish to keep all of their installed content within C:\Program Files\Foo\ for example.
  4. There is no clean uninstall procedure, as the -install-global-extension flag was designed only as a means to install items for all user profiles, not as a means for third party installers to register their components.
• The system for installation, upgrade and uninstall is not robust enough. When a newer version fo an existing Extension is installed, files for the newer version are copied into the folder used by the older version, and obsolete files are not cleaned up. This can lead to incompatibilities, mysterious crashes and other problems. There are not enough guards in the upgrade and uninstall process to handle failure and abort the operation, restoring the previous state.
• Items that are installed must be visible in the Extension Manager user interface, even if the Extension type is just an integration hook that has no meaningful UI presence.
• Extension installation and registration is also prohibitively difficult/annoying for developers, who are forced to either dangerously hand-edit all the appropriate manifest files, or package their code as a XPI and install it that way every time they make a change.

The State of the World

Install Locations

In Firefox 1.0 Extensions can be installed into only two locations:

• The Firefox user's profile directory directory, e.g. ~/.mozilla/firefox/abcdefgh.default/extensions/
• The Firefox application directory, e.g. C:\Program Files\Mozilla Firefox\extensions

Extension Metadata

The Extension System stores metadata in both of the two locations, in the following files:

• <location>/extensions/Extensions.rdf - an XML/RDF datasource listing all the Extensions installed at that location.
• <location>/extensions.ini - an INI manifest listing the directories for all the Extensions and Themes at the location (used by the Component Manager, Preferences system, Chrome Registry etc to locate files during the startup process).

In the profile directory, the file compatibility.ini stores information about the version of the application (build info) that last started this profile - during startup this file is checked and if the version info held by the running app disagrees with the info held by this file, a compatibility check is run on all installed items.

When changes are made to the Extensions datasource - new items are installed, old items uninstalled, enabled or disabled, a .autoreg file is written to the profile directory as well, which tells the startup code that the system has been modified, so that it destroys the component registries, finishes pending transactions and regenerates metadata appropriately.

Datasource Structure

The Extension Manager implements a RDF Datasource which contains a composite of the two XML/RDF datasources at the two Install Locations. The Composite datasource handles all read-only information requests, and when data must be written the Extension Manager determines the appropriate datasource to write and flush to. The model looks something like this:

    nsExtensionsDataSource.prototype = {
      _composite            // The composite that manages the two 
                            // datasources at the Install Locations for
                            // read-only information requests
      _profileExtensions    // The RDF/XML datasource for the items at the
                            // profile Install Location
      _globalExtensions     // The RDF/XML datasource for the items at the
                            // global Install Location.
    };

Tracking Install Locations

Since there are only two locations, the installed location of an Extension is expressed throughout the code using a boolean value, often referred to as isProfile - true if the item is installed in the profile directory's extensions folder. This boolean relationship itself is not stored directly in the datasource. Rather, if the system needs to know where an item is located, it checks to see if the item is a member of the the appropriate type container (urn:mozilla:extension:root or urn:mozilla:theme:root) in each of the two datasources. This is clumsy.

Item Type

The Extension System in Firefox 1.0 supports only two item types, Extensions and Themes. This is a by-product of the fact that there are physical per-type containers in the datasource. Sections of UI that display installed items are rooted on these containers.

Tracking Item Type

Item type is tracked for the most part by containment in one or other of the two containers. In situations where an item is not yet a member of one of the containers, type is inelegantly determined by checking for presence of a theme-only property on the item's Install Manifest RDF file.

Installation

Initiation

When an item is installed from the web, XPInstall is invoked and it calls into the Extension System when it discovers that the XPI file contains an install.rdf manifest. The Extension Manager extracts that file and checks to see if the item is compatible with the running version of the application. If it is, a small set of metadata about it is written to the appropriate datasource (name, version, a flag to tell the system to properly install it on the next startup), and it is added to the appropriate type container. The system also writes a .autoreg file to the profile folder which tells the startup system that something has changed on the next restart. A copy of the XPI file the item was installed from is set aside since the XPInstall system cleans up the temporary file it hands to the Extension System.

If the item is not compatible, the Extension System asks the appropriate Update Service (either the one specified by the item, or the default one) if there is remote compatibility information that supercedes the compatibility information held by the item. If there is, and this remote information makes it so that the item is now compatible with the running version of the application, the item is configured in the manner described above (metadata written, added to container).

Finalization

For Themes, the item is immediately installed fully, rather than awaiting the next restart, since Themes do not supply XPCOM components, preferences defaults etc.

For Extensions, on the next startup, the startup system notices the presence of the .autoreg file and starts the Extension manager with a "dirty" flag, telling it to finish any install operations that may be pending.

The Extension manager loads the XML/RDF datasources (this is allowable and necessary because a major change has happened) and gets a list of all items that need to be installed (tracked using a toBeInstalled flag on the item in the datasource). It locates the staged copy of the XPI file, extracts its contents (logging file additions as it goes into {GUID}/uninstall/Uninstall), loads the Install Manifest file and copies all metadata into the appropriate datasource. At this time a new extensions.ini file is written locating the Extension so that the Component Manager, Preferences System and Chrome Registry can locate additional files on the next start. The finalization process then notifies the startup process that finalization has changed this manifest, and that the application must be restarted to pick up changes specified therein.

The startup process receives this "needsRestart" bit when the Extension manager's startup completes, shuts down XPCOM and relaunches the application. On the next restart, the XREDirProvider now supplies a list of directories to requesting systems listing the new item's components, defaults and chrome manifests.

Uninstallation, Disabling, Enabling

These functions work on the same principle as installation - the user requests an action through the UI while the application is running and metadata is written (toBeUninstalled, toBeDisabled, toBeEnabled) and a .autoreg file created in the profile so that on the subsequent startup the Extension System's startup routine can remove files (in the Uninstall case) and write a new extensions.ini file listing the directories for the currently "active" items. ("Active" items are items that are enabled.)

A Whole New World

Install Locations

We have several targets for where items can be installed. These include:

• The application profile directory <profile>/extensions/
• The application install directory <application>/extensions/
• Any location specified in a text file with a {GUID} name placed in one of the above locations, useful for developing extensions at another location, e.g. a NFS home directory and simply placing a text "link" file in the applicable directory above that links to the location of the extension in its installed state at the other location.
• Any location specified in a GUID-to-path registry key mapping, e.g. [HKLM|HKCU]\Software\Mozilla\Firefox\Extensions

Given these goals, and the likelihood of future goals, such as potential for extensions to the XULRunner framework, common drop zones for XPIs etc it makes sense to have the set of install locations be customizable. By default, the application knows about and maintains three Install Locations:

app-profile 

A directory-based install location for items living in the application profile extensions directory.

app-global 

A directory-based install location for items living in the application global extensions directory.

app-registry 

A registry-key based install location for items living at locations specified by a GUID-to-path value set within the registry at a predefined location.

To implement this, we create an interface that the Extension Manager can ask for information about the location, such as what its name is, where its located on disk (if anywhere, in the case of registry based locations which have no root directory), for a list of item directories, for a directory of a particular item, for a subdirectory or file within an item's directory, etc.

Since our needs and the needs of other applications may vary in the future, since we've already effectively generalized the concept of an install location, we can make this set configurable by applications and Extensions. To this end, we scan the category extension-install-locations as the Extension System is started (after profile initialization) and add these to our internal set.

Extension Metadata

The following files are now used to hold metadata:

<profile>/extensions.ini - active items 

This file contains a list of active Extension directories (i.e. directories to extensions that are enabled only), to be used by the XREDirProvider during startup to locate components, preferences and chrome manifests. The format here is similar to Firefox 1.0 except there is no longer a Count value that needs to be kept in sync with the number of lines... the file is written simply as Extension#=<absolute path to item> and is read until there are no more numbers. The other difference is that since there is now only a single instance of this file in the profile location, the file paths are now absolute.

<profile>/extensions.rdf - visible items 

This file contains non-startup and compatibility metadata for extensions that are visible to the user, whether they are enabled or disabled. If an extension is installed at two different Install Locations, the one with the higher importance is what is shown in this file. This replaces the individual Extensions.rdf files at the old locations.

<profile>/extensions-startup.manifest - all items 

This file contains a tab delimited set of lines, one per item. This is a list of all installed items, disabled or not, keyed first by Install Location name, then by GUID. The following information is stored for each entry:

• the persistent descriptor of the path where the item lives.
• the last modified time of that path, used to detect out of process upgrades.
• optionally, an operation key that tells the startup system that there is an install operation pending that needs to be finalized, e.g. needs-install, needs-upgrade, needs-uninstall, needs-enable, needs-disable, needs-install

When the Extension Manager starts, this dataset is read into two data structures:

• The Startup Cache - a hashtable keyed off Install Location name and then GUID, each entry having persistentDescriptor, mtime and id properties.
• The Pending Operations List - a set of entries organized into arrays hashed by operation key, each entry having a locationKey and id properties.

The Startup Cache data structure is used to reflect the extensions-startup.manifest file over the lifetime of the running application, the extensions-startup.manifest file is written from the current state of the cache.

The Pending Operations List is used by the Install Operation Finalization routine (|_finishOperations|) to get a list of items that are to be operated on.

Since all metadata is now stored in the profile directory, there is no longer any need for special Extension System handling of the -register command line flag, so support for that has been removed.

Datasource Structure

Since there is now only a single RDF/XML datasource for storage of all installed items, the staggered datasource structure used by Firefox 1.0 and the use of an internal composite datasource is no longer required. The Extension System retains an object implementing nsIRDFDataSource (so that it can supply special properties in addition to the set stored simply in the XML datasource) and an internal member that holds the single RDF/XML datasource. Properties are read and are written directly to this, there are no shims.

Tracking Install Locations

Since there is now only a single datasource, Install Location information is tracked for every item in the single datasource in the form of a string <em:installLocation> property on each item. The value of this property corresponds to the name property on the Install Location where the item is installed. Code in the Extension System can quickly locate an Install Location for an item using the getInstallLocation method which basically reads this value from the datasource and retrieves the Install Location from the hash of registered Install Locations.

Item Type

install.rdf Install Manifests should now specify a <em:type> property which tells the Extension System what their type is. Types are defined in nsIUpdateService.idl on the nsIUpdateItem interface. The types at the time of writing include:

• 2 - Extension
• 4 - Theme
• 8 - Locale

For backward compatibility the Extension System will continue to assume an item is an Extension by default, and a Theme if the item has a <em:internalName> property, but Extension and Theme authors should be good citizens and upgrade their Install Manifests to include the type.

The type information supplied or inferred will be added to the Extensions datasource.

Tracking Item Type

We still track the item type at this stage by looking at containment. See below in Things This Upgrade Does Not Do.

Installation

Installation is much different. There three kinds of item installation:

1. Installation from a file
2. Installation by a folder or folder-link "appearing" in an Install Location, and
3. A hybrid of the two - an XPI file "appearing" into a directory based Install Location.

Installation From a File

Initiation

When an item is installed from a File (such as, when an item is installed from the web, via XPInstall, upgraded by XPInstall or dropped into a directory-based Install Location), the Install Manifest supplied by the item is expanded into a temporary location and read. The GUID and Version supplied are validated, and then Compatibility is checked for by the new _getInstallData function. This function returns the GUID, version, and type of the item, and also an error code listing either success, or the reason for a failure, such as invalid GUID, version, or incompatible item.

Depending on the failure, the Install function may take several steps. If the item is found incompatible, a Version Update check is performed as before. If updated compatibility information is found this is written into the in-memory representation of the the temporary Install Manifest and the Install function is called recursively, supplying this updated Install Manifest as the source for _getInstallData.

If the item is determined compatible by either of the above processes, a copy of the item's file is staged into the Install Location directory under a hierarchy like so: <staged-xpis>/guid/foo.xpi (where foo.xpi is the original file name of the file) since XPInstall cleans up the file it supplies when the Install function returns. If the item is a Theme, we perform the installation immediately (thanks to Benjamin Smedberg's Chrome Registry changes this operation is now so simple that it can be performed by a function on the ExtensionManager, rather than creating a separate object).

If the item is an Extension, we add it to the Extension container and set the same set of initial setup properties on it that we used to, with the addition of the installLocation key and the type value.

At this stage we rewrite the extensions-startup.manifest and extensions.ini manifests and create the .autoreg file in the profile directory.

Finalization

On the next startup, the Startup system discovers the .autoreg file (as before) and starts the Extension System with the dirty flag, which causes the system to scan for items to be installed as before.

Regardless of whether or not the Extension System is called from the Startup process with the dirty flag set to true (i.e. the Startup process discovered the .autoreg file, before pending operations are finalized the Extension System reads the extensions-startup.manifest and scans all of the Install Locations looking for items that match the following criteria:

1. Item in Install Location is not listed in Startup Manifest (new item being installed by just appearing in the Install Location).
2. Item in Install Location is listed in Startup Manifest, but its mtime is different. (Existing item in Install Location being upgraded).
3. Item in the Startup Manifest does not exist in the Install Location (item uninstalled by removing its folder or registry key).
4. XPI File that is not a staged XPI for a pending install operation has appeared in the Install Location.

For case 1, the same registration steps performed during the Initiation steps of the install-from-file operation are now performed. The configuration sets up initial assertions in the datasource for the item, setting the "toBeInstalled" flag to tell the finalization function to fully register the item.

See below for cases 2 and 3.

If a non-staged XPI is detected, it is passed to the installItemFromFile function, which performs the initial registration so that when finalization routines run the item is properly installed.

Upgrades

When an item is installed (in any of the places described above, e.g. when an item is being installed from a XPI file from the web or from a drop in) the Extension System checks to see if there is an existing version installed, and if so a special routine is called which sets a toBeUpgraded flag on the configured item instead of the usual toBeInstalled flag. On the subsequent startup, this function causes metadata about the old version of the item to be completely removed from the Extensions datasource and the new data from the Install Manifest of the upgrade copied in, to avoid duplicate assertions. After the Upgrade finalization routine executes, it unsets the toBeUninstalled flag and sets the toBeInstalled flag to trigger the item's files to be replaced from the staged XPI.

When an item's files are extracted, a special extraction routine known as a safeInstallOperation is performed. This function moves the existing item's folder to {GUID}-trash file-by-file, while maintaining a list of the files that are moved. If at any point during this move the move of an individual file fails, the process rolls back and the install is aborted. The extraction operation then proceeds into the now-clean {GUID} folder. If this operation fails, the entire operation is rolled back. At the end of the process once everything has succeeded the {GUID}-trash folder is removed.

Uninstallation

If an item is uninstalled via the UI (and has a toBeUninstalled flag set, its data is removed from the datasource and its files removed using a safeInstallOperation with no extraction function, i.e. the item's files are moved to {GUID}-trash and then deleted.

If an item is uninstalled by removing its directory, the Extension startup process detects this prior to finalization and sets the toBeUninstalled flag so that the item's metadata is removed from the datasource.

Uninstall and Staggered Install Locations

If Item Foo 1.1 is installed in the user's profile directory Install Location, and Foo 1.0 is installed in the application directory Install Location, when the user uninstalls Foo 1.1, Foo 1.0 should become visible due to Install Location ordering. This allows the user to shield items installed at lower locations by installing items at higher locations.

When an item is uninstalled, the Extension Manager checks the Startup Cache to see if there is another copy of the item installed at a lower Install Location. If there is one, then the item is uninstalled and an entry is added to the Pending Operations List that indicates the item at the lower Install Location needs to be installed.

The install operation finalization process loops continuously installing, upgrading and uninstalling items at progressively lower locations until the size of the Pending Operations List is finally zero.

Little Things This Upgrade Fixes or Changes

It used to not be possible to move items in the Extension Manager UI up and down if there were items that were from two different Install Locations in the list - that is there were actually two RDF containers and ordering is unique to each. By moving to a unified datasource where Install Location containment is expressed as a property, user-defined ordering can be respected.

Extension authors installing into restricted Install Locations can specify that their item does not show up in the Extensions UI by using the <em:hidden>true<em:hidden> property in their Install Manifests. This property is ignored for Extensions installed into unrestricted locations, such as the profile location, to prevent abuse.

There is more logging for failures in the installation and update process. The set of error logs written is still neither perfect nor complete, but it's a very good start. Log strings are written to the application's error console (in anticipation of that API automatically logging to a file) and to the terminal window.

We now check more effectively for system integrity during startup. We ask each Install Location if it is functioning (for a Directory Install location such as app-profile or app-global the location is functioning if it the directory exists. If any location is not functioning, or any of the manifests listed above does not exist, all manifests are removed and the system rebuilds everything from scratch. This should prevent against random file deletions and allow developers to easily reset their state by deleting one of the required system files.

Since we can now install by simply detecting the presence of a {GUID} folder under the app-profile and app-global Install Locations, we no longer need installed-extensions.txt for registration of pre-configured items such as the default theme. The Firefox build system will just create the {GUID} folder for the default theme and place its Install Manifest inside, and on the first startup of the application it will be automatically registered.

We drop support for the -lock-item/-unlock-item flags, as well as the -list-global-items flag.

Messages that appear in the Extensions UI in response to user-actions such as install, uninstall, enable and disable are now driven by the Extensions datasource itself and are supplied by the em:displayDescription property, rather than being supplied by the front end and several different redundant XBL bindings/style rules.

It is no longer possible to perform any operations on an item that has just been installed. Bugs in the command controller in the Extensions UI made this possible in Firefox 1.0 which could leave the user in bogus states.

The nsIExtensionManager interface now has a single, unified set of installation/enabling interfaces:

• void installItemFromFile(in nsIFile file, in string locationKey);
• void uninstallItem(in string id);
• void enableItem(in string id);
• void disableItem(in string id);

Uninstall logs are no longer written, since the item folder is removed completely on uninstall. Benjamin Smedberg's recent Chrome Registry changes made it unnecessary to have log entries for chrome registration, so the Uninstall log was only being used for file additions. This allows for the removal of the InstallLogReader/Writer objects in the Extension Manager.

It is possible to order Install Locations using the priority property on an Install Location object. This determines whether or not an extension from one Install Location can "trump" the same Extension installed at a different Install Location.

User action requested notifications are now sent with the em-action-requested topic through the observer service:

item-installed 

An item has just been downloaded and configured for installation for the first time.

item-upgraded 

A new version of an installed item has been downloaded and configured for installation.

item-uninstalled 

An item has been configured for uninstallation.

item-enabled 

An item has been configured for enabling.

item-disabled 

An item has been configured for disabling.

Things This Upgrade Does Not Do

Replace RDF/XML as a Storage Format

The RDF/XML datasource back end creates unneccessary and undesirable complexity when saving data. It also makes it more difficult to support multiple instances of a single extension at different Install Locations due to the singleton nature of RDF resources. Extension metadata is structured and not relational in the sense encouraged by RDF, and so a simple text storage format is probably desirable. Making the database file format human readable would also help developers debug their installation problems.

Footnotes

1. The term "Extension" is used in this document to imply any item managed by the Extension Manager, including themes.
2. The term "Firefox" is used in this document to imply any application built on XULRunner that uses the Extension System.