Storage

Storage is a SQLite database API. It is available to trusted callers, meaning extensions and Firefox components only. For a complete reference of all methods and properties of the database connection interface see mozIStorageConnection.

The API is currently "unfrozen," which means it is subject to change at any time. It's likely that the API will change somewhat between Firefox 2 and Firefox 3.

Note: Storage is not the same as the DOM:Storage feature which can be used by web pages to store persistent data or the Session store API (an XPCOM storage utility for use by extensions).

Getting started

This document covers the mozStorage API and some peculiarities of sqlite. It does not cover SQL or "regular" sqlite. You can find some very useful links in the See also section however. For mozStorage API help, you can post to mozilla.dev.apps.firefox on the news server news.mozilla.org. To report bugs, use Bugzilla (product "Toolkit", component "Storage").

Here we go then. mozStorage is designed like many other database systems. The overall procedure for use is:

Open a connection to the database of your choice.
Create statements to execute on the connection.
Bind parameters to a statement as necessary.
Execute the statement.
Check for errors.
Reset the statement.

Opening a connection

C++ users: The storage service's first initialization must be from the main thread. You will get an error if you initialize it the first time from another thread. Therefore, if you want to use the service from a thread, be sure to call getService from the main thread to ensure the service has been created.

C++ example of opening a connection to "asdf.sqlite" in the user's profile directory:

nsCOMPtr<nsIFile> dbFile;
rv = NS_GetSpecialDirectory(NS_APP_USER_PROFILE_50_DIR,
                            getter_AddRefs(dbFile));
NS_ENSURE_SUCCESS(rv, rv);
rv = dbFile->Append(NS_LITERAL_STRING("asdf.sqlite"));
NS_ENSURE_SUCCESS(rv, rv);

mDBService = do_GetService(MOZ_STORAGE_SERVICE_CONTRACTID, &rv);
NS_ENSURE_SUCCESS(rv, rv);
rv = mDBService->OpenDatabase(dbFile, getter_AddRefs(mDBConn));
NS_ENSURE_SUCCESS(rv, rv);

MOZ_STORAGE_SERVICE_CONTRACTID is defined in storage/build/mozStorageCID.h. It's value is "@mozilla.org/storage/service;1"

JavaScript example:

var file = Components.classes["@mozilla.org/file/directory_service;1"]
                     .getService(Components.interfaces.nsIProperties)
                     .get("ProfD", Components.interfaces.nsIFile);
file.append("my_db_file_name.sqlite");

var storageService = Components.classes["@mozilla.org/storage/service;1"]
                        .getService(Components.interfaces.mozIStorageService);
var mDBConn = storageService.openDatabase(file);

Note: The OpenDatabase function is subject to change. It will likely be enhanced/simplified to make it more difficult to get into trouble.

It may be tempting to give your database a name ending in ".sdb" for sqlite database, but this is not recommended. This extension is treated specially by Windows as a known extension for an "Application Compatibility Database" and changes are backed up by the system automatically as part of system restore functionality. This can result in much higher overhead file operations.

Statements

Follow these steps to create and execute SQL statements on your SQLite database. For a complete reference see mozIStorageStatement.

Creating a statement

There are two ways to create a statement. If you have no parameters and the statement doesn't return any data, use mozIStorageConnection.executeSimpleSQL.

C++:
rv = mDBConn->ExecuteSimpleSQL(NS_LITERAL_CSTRING("CREATE TABLE foo (a INTEGER)"));

JS:
mDBConn.executeSimpleSQL("CREATE TABLE foo (a INTEGER)");

Otherwise, you should prepare a statement using mozIStorageConnection.createStatement:

C++:
nsCOMPtr<mozIStorageStatement> statement;
rv = mDBConn->CreateStatement(NS_LITERAL_CSTRING("SELECT * FROM foo WHERE a = ?1"),
                              getter_AddRefs(statement));
NS_ENSURE_SUCCESS(rv, rv);

JS:
var statement = mDBConn.createStatement("SELECT * FROM foo WHERE a = ?1");

This example uses the placeholder "?1" for a parameter to be bound later (see next section).

Once you have a prepared statement, you can bind parameters to it, execute it, and reset it over and over. If you are doing a statement many times, using a precompiled statement will give you a noticable performance improvement because the SQL query doesn't need to be parsed each time.

If you are familiar with sqlite, you may know that prepared statements are invalidated when the schema of the database changes. Fortunately, mozIStorageStatement detects the error and will recompile the statement as needed. Therefore, once you create a statement, you don't need to worry when changing the schema; all statements will continue to transparently work.

Binding parameters

It is generally best to bind all parameters separately rather than try to construct SQL strings on the fly containing the parameters. Among other things, this prevents SQL injection attacks, since a bound parameter can never be executed as SQL.

You bind parameters to a statement that has placeholders. The placeholders are addressed by index, starting from "?1", then "?2"... You use the statement functions BindXXXParameter(0) BindXXXParameter(1)... to bind to these placeholders.

Watch out: The indices in the placeholders count from 1. The integers passed to the binding functions count from 0. This means "?1" corresponds to parameter 0, "?2" corresponds to parameters 1, etc.

You can also use named parameter, like this ":example" instead of "?xx".

A placeholder can appear multiple times in the SQL string and all instances will be replaced with the bound value. Unbound parameters will be interpreted as NULL.

The examples below only use bindUTF8StringParameter() and bindInt32Parameter(). For list of all binding functions see mozIStorageStatement.

C++ example:

nsCOMPtr<mozIStorageStatement> statement;
rv = mDBConn->CreateStatement(NS_LITERAL_CSTRING("SELECT * FROM foo WHERE a = ?1 AND b > ?2"),
                              getter_AddRefs(statement));
NS_ENSURE_SUCCESS(rv, rv);
rv = statement->BindUTF8StringParameter(0, "hello"); // "hello" will be substituted for "?1"
NS_ENSURE_SUCCESS(rv, rv);
rv = statement->BindInt32Parameter(1, 1234); // 1234 will be substituted for "?2"
NS_ENSURE_SUCCESS(rv, rv);

JavaScript example:

var statement = mDBConn.createStatement("SELECT * FROM foo WHERE a = ?1 AND b > ?2");
statement.bindUTF8StringParameter(0, "hello");
statement.bindInt32Parameter(1, 1234);

If you use named parameters, you should use the getParameterIndex method to get the index of the named parameter. Here is a JavaScript example:

var statement = mDBConn.createStatement("SELECT * FROM foo WHERE a = :myfirstparam AND b > :mysecondparam");

var firstidx = statement.getParameterIndex(":myfirstparam");
statement.bindUTF8StringParameter(firstidx, "hello");

var secondidx = statement.getParameterIndex(":mysecondparam");
statement.bindInt32Parameter(secondidx, 1234);

You can of course mix named parameters and indexed parameters in a same query:

var statement = mDBConn.createStatement("SELECT * FROM foo WHERE a = ?1 AND b > :mysecondparam");

statement.bindUTF8StringParameter(0, "hello");
// you can also use
// var firstidx = statement.getParameterIndex("?1");
// statement.bindUTF8StringParameter(firstidx, "hello");

var secondidx = statement.getParameterIndex(":mysecondparam");
statement.bindInt32Parameter(secondidx, 1234);

If you want to use a WHERE clause with an IN ( value-list ) expression, Bindings won't work. Construct a string instead. If you're not handling user input it's no safety concern:

var ids = "3,21,72,89";
var sql = "DELETE FROM table WHERE id IN ( "+ ids +" )";

Executing a statement

The main way to execute a statement is with mozIStorageStatement.executeStep. This function allows you to enumerate all the result rows your statement produces, and will notify you when there are no more results.

After a call to executeStep, you use the appropriate getter function in mozIStorageValueArray to get the values in a result row (mozIStorageStatement implements mozIStorageValueArray). The example below only uses getInt32().

You can get the type of a value from mozIStorageValueArray.getTypeOfIndex, which returns the type of the specified column. Be careful: sqlite is not a typed database. Any type can be put into any cell, regardless of the type declared for the column. If you request a different type, sqlite will do its best to convert them, and will do some default value if it is impossible. Therefore, it is impossible to get type errors, but you may get weird data out.

C++ code can also use AsInt32, AsDouble, etc. functions which return the value as a more convenient C++ return value. Watch out, though, because you won't get any errors if your index is invalid. Other errors are impossible, because sqlite will always convert types, even if they don't make sense.

C++ example:

PRBool hasMoreData;
while (NS_SUCCEEDED(statement->ExecuteStep(&hasMoreData)) && hasMoreData) {
  PRInt32 value = statement->AsInt32(0);
  // use the value...
}

Javascript example:

while (statement.executeStep()) {
  var value = statement.getInt32(0); // use the correct function!
  // use the value...
}

mozIStorageStatement.execute() is a convenience function for when you are getting no data out of the statement. It steps the statement once and resets it. This can be useful for insert statements because it really cleans up the code:

var statement = mDBConn.createStatement("INSERT INTO my_table VALUES (?1)");
statement.bindInt32Parameter(52);
statement.execute();

This Image:TTRW2.zip is a simple, but complete, JavaScript and XUL example of how you run an SQL SELECT against a database.

Resetting a statement

It is important to reset statements that are no longer being used. Un-reset write statements will keep a lock on the tables and will prevent other statements from accessing it. Un-reset read statements will prevent writes.

When the statement object is freed, its corresponding database statement is closed. If you are using C++ and you know that all references will be destroyed, you don't have to explicitly reset the statement. Also, if you use mozIStorageStatement.execute(), you don't need to explicitly reset the statement; this function will reset it for you. Otherwise, call mozIStorageStatement.reset().

JavaScript callers should ensure that statements are reset. Be particularly careful about exceptions. You will want to make sure to reset your statements even if an exception is fired, or subsequent access to the database may not be possible. Resetting a statement is relatively lightweight, and nothing bad happens if it's already reset, so don't worry about unnecessary resets.

var statement = connection.createStatement(...);
try {
  // use the statement...
} finally {
  statement.reset();
}

C++ callers must do the same. There is a scoped object in storage/public/mozStorageHelper.h called mozStorageStatementScoper which will ensure that a given statement is reset when the enclosing scope is exited. It is hightly recommended that you use this object if possible.

void someClass::someFunction()
{
  mozStorageStatementScoper scoper(mStatement)
  // use the statement
}

Last insert id

Use the lastInsertRowID property on the connection to get the id (rowid) from the last INSERT operation on the db.
This is useful if you have a column in your table set to INTEGER PRIMARY KEY or INTEGER PRIMARY KEY AUTOINCREMENT in which case SQLite automatically assigns a value for each row inserted if you don't provide one. The returned value is of type number in JS and long long in C++.

lastInsertRowID JS example:

var sql = "INSERT INTO contacts_table (number_col, name_col) VALUES (?1, ?2)"
var statement = mDBConn.createStatement(sql);
    statement.bindUTF8StringParameter(0, number);
    statement.bindUTF8StringParameter(1, name);
    statement.execute();
    statement.reset();
    
var rowid = mDBConn.lastInsertRowID;

Transactions

mozIStorageConnection has functions for beginning and ending transactions. If you do not explicitly use transactions, an implicit transaction will be created for you for each statement. This has major performance implications. There is overhead for each transaction, especially for commits. You will therefore see a large performance win when you are doing multiple statements in a row if you put them in a transaction. See Storage:Performance for more performance information.

The major difference between other database systems is that sqlite does not support nested transactions. This means that once a transaction is open, you can not open another transaction. You can check mozIStorageConnection.transactionInProgress to see if a transaction is currently in progress.

You can also just execute "BEGIN TRANSACTION" and "END TRANSACTION" directly as SQL statements (this is what the connection does when you call the functions). However, use of mozIStorageConnection.beginTransaction and related functions are strongly recommended because it stores transaction state in the connection. Otherwise, the attribute transactionInProgress will have the wrong value.

sqlite has several types of transactions:

mozIStorageConnection.TRANSACTION_DEFERRED: The default. The database lock is acquired when needed (usually the first time you execute a statement in the transaction).

mozIStorageConnection.TRANSACTION_IMMEDIATE: Get a read lock on the database immediately.

mozIStorageConnection.TRANSACTION_EXCLUSIVE: Get a write lock on the database immediately.

You can pass this type of transaction to mozIStorageConnection.beginTransactionAs to determine what kind of transaction you need. Keep in mind that if another transaction has already started, this operation will not succeed. Generally, the default TRANSACTION_DEFERRED type is sufficient and you shouldn't use the other types unless you really know why you need them. For more information, see the sqlite documentation about BEGIN TRANSACTION and locking.

var ourTransaction = false;
if (!mDBConn.transactionInProgress) {
  ourTransaction = true;
  mDBConn.beginTransactionAs(mDBConn.TRANSACTION_DEFERRED);
}

// ... use the connection ...

if (ourTransaction)
  mDBConn.commitTransaction();

From C++ code, you can use the mozStorageTransaction helper class defined in storage/public/mozStorageHelper.h. This class will begin a transaction of the specified type on the specified connection when it comes into scope, and will either commit or rollback the transaction when it goes out of scope. If a transaction is already in progress, the transaction helper class will not do anything.

It also has functions for explicitly committing. The typical use is that you create the class defaulting to rollback, and then explicitly commit the transaction when processing has succeeded:

nsresult someFunction()
{
  // deferred transaction (the default) with rollback on failure
  mozStorageTransaction transaction(mDBConn, PR_FALSE);

  // ... use the connection ...

  // everything succeeded, now explicitly commit
  return transaction.Commit();
}

How to corrupt your database

Read this document: File locking and concurrency in sqlite version 3, especially the section on corruption.

Open more than one connection to the same file with names that aren't exactly the same as determined by strcmp. This includes "my.db" and "../dir/my.db" or, on Windows (case-insensitive) "my.db" and "My.db". Sqlite tries to handle many of these cases, but you shouldn't count on it.

Access a database from a symbolic or hard link.

Open connections to the same database from more than one thread (see "Thread safety" below).

Access a connection or statement from more than one thread (see "Thread safety" below).

Open the database from an external program while it is open in Mozilla. Our caching breaks the normal file-locking in sqlite that allows this to be done safely.

SQLite Locking

SQLite locks the entire database; that is, any active readers will cause an attempt to write to return SQLITE_BUSY, and an active writer will cause any attempt to read to return SQLITE_BUSY. A statement is considered active from the first step() until reset() is called. execute() calls step() and reset() in one go. A common problem is forgetting to reset() a statement after you've finished step()'ing through.

While a given SQLite connection is capable of having multiple statements open, its locking model limits what these statements can do concurrently (reading or writing). It is in fact possible for multiple statements to be actively reading at one time. It is not possible, however, for multiple statements to be reading and writing at one time on the same table -- even if they are derived from the same connection.

SQLite has a two-tiered locking model: connection level and table level. Most people are familiar with the connection (database) level locking: multiple readers but only one writer. The table-level (B-Tree) locks are what can sometimes be confusing. (Internally, each table in the database has its own B-Tree, so "table" and "B-Tree" are technically synonymous).

Table-level locks

You would think that if you have only one connection, and it locks the database for writing, you could use multiple statements to do whatever you want. Not entirely. You must be aware of table-level (B-Tree) locks, which are maintained by statement handles traversing the database (i.e. open SELECT statements).

The general rule is this: a statement handle may not modify a table (B-Tree) which other statement handles are reading (have open cursors on) -- even if that statement handle shares the same connection (transaction context, database lock, etc.) with the other statement handles. Attempts to do so will still block (or return SQLITE_BUSY) .

This problem often crops up when you attempt to iterate over a table with one statement and modify records within it using another statement. This will not work (or carries a high probability of not working, depending on the optimizer's involvement (see below)). The modifying statement will block because the reading statement has an open cursor on the table.

Working around locking problems

The solution is to follow (1) as described above. Theoretically, (2) actually shouldn't work with SQLite 3.x. In this scenario, database locks come into play (with multiple connections) in addition to table locks. Connection 2 (modifying connection) will not be able to modify (write to) the database while the Connection 1 (reading connection) is reading it. Connection 2 will require an exclusive lock to execute a modifying SQL command, which it cannot get as long as Connection 1 has active statements reading the database (Connection 1 has a shared read lock during this time which prohibits any other connections from getting an exclusive lock).

Another option is to use a temporary table. Create a temporary table containing the results of the table of interest, iterate over it (putting the reading statement's table lock on the temp table) and then the modifing statement can make changes to the real table without any problem. This can be done with statements derived from a single connection (transaction context). This scenario sometimes happens behind the scenes anyway as ORDER BY can produce temporary tables internally. However, it is not safe to assume that the optimizer will do this in all cases. Explicitly creating a temporary table is only safe way to do perform this latter option.

Thread safety

The mozStorage service and sqlite are threadsafe. However, no other mozStorage or sqlite objects or operations are threadsafe.

The storage service must be created on the main thread. If you want to access the service from another thread, you should be sure that you call getService from the main thread ahead of time.

You can not access a connection or statement from multiple threads. These storage objects are not threadsafe, and the sqlite representations of them are not thread safe either. Even if you do locking and ensure that only one thread is doing something at once, there may be problems. This case hasn't been tested, and there may be some internal per-thread state in sqlite. It is strongly advised that you don't do this.

You can not access a single database from multiple connections from different threads. Normally, sqlite allows this. However, we do sqlite3_enable_shared_cache(1); (see sqlite shared-cache mode) which makes multiple connections share the same cache. This is important for performance. However, there is no lock for cache access, meaning it will break if you use if from more than one thread.

It's worth noting, however, that authors of JavaScript browser extensions are less impacted by these restrictions than it might first appear. If a database is created and used exclusively from within JavaScript, thread safety usually will not be an issue. SpiderMonkey (the JavaScript engine run within Firefox) executes JavaScript from a single persistent thread, except when the JavaScript runs in a different thread or is executed from a callback made on a different thread (e.g. via some networking or stream interfaces). Barring incorrect use of multi-threaded JavaScript, problems should occur only if a database already in use by a non-JavaScript, system-level thread is accessed through mozStorage.