XPCOM Stream Guide

What are streams?

In Mozilla code, a stream is an object which represents access to a sequence of characters.  It is not that sequence of characters, though: the characters may not all be available when you read from the stream. 

Think of a water tank with a spout:  the tank may be full, or it may be half-full, or it may be empty. The spout controls how much water you can get out of the tank at a time. The spout is a source of water. If we think of the water as data, then the spout represents an input stream: a controller for data coming out of something. At the same time, there may be some way to put water into the tank - say, a main water line. That water line represents an output stream: a controller for data going into something.

The actual data going in to and coming out of an object isn't usually important to a stream. Streams simply provide a way to read or write data to some object.

In basic C++ programming for a console application, we usually talk about streams to access files, or to interact with the user. Mozilla's use of streams is more complex. On the one hand, we don't interact with the user through streams. On the other hand, we use streams to access files within a ZIP archive, to store and provide data coming from websites, even to talk to other programs on the same computer through "pipes" (more on that later). We even have streams that take input from other streams and transform the data within them to something more useful.

Primitive streams

There are streams for several different types of data storage.  Each of these implements nsIInputStream.

Similarly, each of these implements nsIOutputStream.

Channels have streams too

Any implementation of nsIChannel will have an input stream as well, but unless you own the channel, you shouldn't try to read from the input stream. There are two ways of getting the input stream: by calling the channel's .open() method for a synchronous read, or by calling the channel's .asyncOpen() method with an nsIStreamListener object.

You can get an nsIChannel object from the IO Service's newChannel(spec, charset, baseURI) method or its .newChannelFromURI(uri) method.

The "safe file output stream"

Mozilla provides a "safe file output stream" implementation. This implementation actually writes the contents of the file you're trying to create to a temporary file. If everything goes well, calling the stream's .finish() method overwrites the original file with the new file.

Using Streams in C++

Using Streams in JavaScript

Input Streams

Input streams are not scriptable - you cannot directly call .read() on them, for example. To solve this, there is a special nsIScriptableInputStream interface and "scriptable stream" wrapper. If you have an input stream called nativeStream, you can use code like this:

var stream = Components.classes["@mozilla.org/scriptableinputstream;1"]
                       .createInstance(Components.interfaces.nsIScriptableInputStream);
stream.init(nativeStream);

The stream provides .read(count), .available(), and .close() methods.

Output Streams

Output streams are usually scriptable:  you can call .write(chars, chars.length) as you wish. However, it is usually better to create an input stream that you then feed to the output stream:

var outStream = Components.classes["@mozilla.org/storagestream;1"]
                          .createInstance(Components.interfaces.nsIStorageStream)
                          .getOutputStream();
var inStream = Components.classes["@mozilla.org/io/string-input-stream;1"]
                         .createInstance(Components.interfaces.nsIStringInputStream);
var data = "Hello World";
inStream.setData(data, data.length);
while (inStream.available()) {
    outStream.writeFrom(inStream, inStream.available());
}

Note this is an inefficient example:  the only important part is how to feed the output stream.  This is also a synchronous (blocking) operation, so if you're in JavaScript, consider using NetUtil.jsm as described below.

A note about Unicode strings versus nsIInputStream

nsIInputStream and nsIOutputStream work with 8-bit characters.  However, JavaScript strings contain 16-bit characters.  This can mean if you have characters beyond ASCII code 255, you risk losing data using nsStringStream, for example.

To get an input stream for JavaScript strings safely, try this.

var converter = Components.classes["@mozilla.org/intl/scriptableunicodeconverter"]
                          .createInstance(Components.interfaces.nsIScriptableUnicodeConverter);
converter.charset = "UTF-8";
var stream = converter.convertToInputStream(string);

JavaScript modules are your friends

There are several useful JavaScript modules at your disposal. For streams, the most obvious are NetUtil.jsm and FileUtils.jsm.  NetUtil.jsm provides APIs for copying an input stream to an output stream (the asyncCopy() method), getting an input stream from another source (the asyncFetch() method), and reading an input stream into a string (the readInputStreamToString() method).  FileUtils.jsm provides APIs for getting output streams for files, with the .openFileOutputStream(file, modeFlags) and .openSafeFileOutputStream(file, modeFlags) methods, and for closing those output streams with the .closeSafeFileOutputStream(inputStream) method.

As we mentioned earlier, you can use the IO service to get any channel, and from there an input stream. The IO service is available through the Services.jsm module as the .io property.

Stream Listeners

A stream listener is an object you build to let you know when there is data in a stream ready for you to consume. Most streams are asynchronous: they make no assumptions that all the data a resource provides is available immediately. (This is particularly true of streams that reach out over a network connection, like HTTP and FTP channels.)

Stream listeners implement three methods. From the nsIStreamListener interface, the .onDataAvailable(request, context, inputStream, offset, count) method gives you the input stream and the number of bytes available. The stream listener must read exactly count bytes before exiting. From the nsIRequestObserver interface, the .onStartRequest(request, context) method tells you when the request begins, while the .onStopRequest(request, context) method tells you when the request ends. A request will have one .onStartRequest(request, context) call, followed by at least one .onDataAvailable(...) call, followed by one .onStopRequest(request, context) call.

The context argument will be something passed from whoever invokes the request to the .onStartRequest(), .onDataAvailable(), and .onStopRequest() methods of the listener.

As for passing in the stream listener and starting the request: that will vary depending on the use case.

Seekable Streams

Some streams are "seekable": they let you specify where in the stream you are reading from (instead of requiring it be from the beginning). They can also report their current position in the file. Seekable streams implement the nsISeekableStream interface.

The following stream types are known to implement nsISeekableStream:

• nsStorageInputStream
• nsStringInputStream
• nsMultiplexInputStream
• nsPipeInputStream
• nsFileInputStream
• nsBufferedInputStream
• nsFileOutputStream
• nsBufferedOutputStream

Complex stream types

Several stream types leverage primitive stream types to do specialized work. These work by taking the input from another stream, and providing a stream interface to access that underlying stream's data.

Similarly, there are complex output streams which build from primitive output streams:

Additional Stream Interfaces

• The nsILineInputStream interface supports a .readLine() method for reading a single line from an input stream. The nsFileInputStream and nsPartialFileInputStream classes implement this interface.
• (XXX nsIUnicharInputStream interface)
• (XXX nsIUnicharLineInputStream interface)
• (XXX nsISearchableInputStream interface)

Stream Converters

(TBD: @mozilla.org/streamConverters;1)

Forcing an input stream to be read

Suppose you already have an input stream, and something to read from that input stream...but the reader doesn't do anything with the stream. Suppose that reader also implements nsIStreamListener. Mozilla provides an "input stream pump" component to feed data from the stream into the reader.

There are two parts: initializing the pump, and telling it to asynchronously read data into the stream listener:

var pump = Components.classes["@mozilla.org/network/input-stream-pump;1"]
                     .createInstance(Components.interfaces.nsIInputStreamPump);
pump.init(stream, -1, -1, 0, 0, true);

pump.asyncRead(listener, context);

nsIPipe

Code Examples

File input and output

For file input, see Code Snippets: Reading from a file.  For file output, see Code Snippets: Writing to a file.

ZIP input and output

For getting an input stream from a ZIP archive, see the nsIZipReader interface:

// file is an nsIFile object mapping to a ZIP archive
var zipReader = Components.classes["@mozilla.org/libjar/zip-reader;1"]
                          .createInstance(Components.interfaces.nsIZipReader);
zipReader.open(file);
var stream = zipReader.getInputStream("/path/to/zipped/file");

// process the stream

// when we don't need the zipReader anymore
zipReader.close();

For writing from an input stream to a ZIP archive, see the nsIZipWriter interface:

// file is an nsIFile object mapping to a ZIP archive
var zipWriter = Components.classes["@mozilla.org/zipwriter;1"]
                          .createInstance(Components.interfaces.nsIZipWriter);

zipWriter.open(file, ioFlags);

// stream is the output stream
zipWriter.addEntryStream("/path/to/zipped/file", modTime, compression, stream, queueForLater);

// if queued for later operations, and all operations are queued
zipWriter.processQueue();

// when we don't need the zipWriter anymore
zipWriter.close();

Concatenating input streams

var StringStream = Components.Constructor("@mozilla.org/io/string-input-stream;1",
                                          "nsIStringInputStream",
                                          "setData");

function buildStream(data) {
  return new StringStream(data, data.length);
}

function run_test() {
  var str1 = buildStream("We ");
  var str2 = buildStream("will ");
  var str3 = buildStream("rock ");
  var str4 = buildStream("you!");

  var check = "We will rock you!";

  var multi = Components.classes["@mozilla.org/io/multiplex-input-stream;1"]
                        .createInstance(Components.interfaces.nsIMultiplexInputStream);
  multi.appendStream(str1);
  multi.appendStream(str2);
  multi.appendStream(str3);
  multi.appendStream(str4);

  var inStream = Components.classes["@mozilla.org/scriptableinputstream;1"]
                           .createInstance(Components.interfaces.nsIScriptableInputStream);
  inStream.init(multi);
  var data = inStream.read(inStream.available());

  // check == data;
}

Creating copies of an input stream

var StringStream = Components.Constructor("@mozilla.org/io/string-input-stream;1",
                                          "nsIStringInputStream",
                                          "setData");

var InputStream = Components.Constructor("@mozilla.org/scriptableinputstream;1",
                                         "nsIScriptableInputStream",
                                         "init");

function buildStream(data) {
  return new StringStream(data, data.length);
}

Components.utils.import("resource://gre/modules/NetUtil.jsm");

function run_test() {
  var check = "We will rock you!";
 
  var baseInputStream = buildStream(check);
 
  var store = Components.classes["@mozilla.org/storagestream;1"]
                        .createInstance(Components.interfaces.nsIStorageStream);
  /* In practice, your storage streams shouldn't be this small.  The first argument,
   * which represents capacity per segment, is far too small for practical use.
   */
  store.init(64, 64, null); 

  var out = store.getOutputStream(0);

  NetUtil.asyncCopy(baseInputStream, out, function(status) {
    if (status != Components.results.NS_OK)
      return;
    /* Due to a crash, we can't create input streams until the storage output stream
     * has some data in it.
     *
     * Also, baseInputStream has been completely consumed at this point, so we
     * shouldn't read from it anymore. (However, because baseInputStream is an
     * nsStringInputStream, it is also a seekable stream...so we could go to
     * the beginning if we wanted to.)
     */
    var str1 = store.newInputStream(0);
    var d1 = new InputStream(str1);
    // d1 has a complete copy of baseInputStream's original contents.

    var d2 = new InputStream(store.newInputStream(0));
    // d2 has a complete copy of baseInputStream's original contents.
  });
}

Parsing a DOM document from an input stream