B2G IPC internals

Architecture

In FirefoxOS we have a Multi-process Architecture where the apps on the phone are running in a different process which has the least amount of privileges. On a running system we have a single parent process called b2g. The b2g process has a child process called nuwa, this process will be used to fork the app processes. Whenever a new app is being started on the phone, b2g tells nuwa that we need a new process. By default,  child processes are run with a minimal set of privileges. For every action they want to perform (that needs more privileges), they need to go through the parent process (b2g). This is where the Inter-process Communication (IPC) comes into place. Each child has a connection, using an IPC channel, to the parent which will be used for its communication. The process layout is illustrated on the right.

Setup

Now that you have a rough overview of how everything is laid out, we will go into more detail on how this communication actually works.
We use Unix sockets, created with the socketpair system call, to send messages across the process boundary. For writing and reading, the system calls sendmsg and recvmsg are being used. Each process has its own dedicated thread that handles the socket operations,which is called an IOLoop. Each IOLoop thread has its own outgoing message queue, which is used by the main thread to send a new message across the channel.

IOLoop

An IOLoop thread is created for the parent during b2g startup, and for each child process after nuwa forks.

Parent

In the parent process (b2g) the IOLoop is created very early on; more precisely it happens in the NS_InitXPCOM2 function. This is the code snippet that actually initiates the thread start:

...
scoped_ptr<BrowserProcessSubThread> ioThread(
    new BrowserProcessSubThread(BrowserProcessSubThread::IO));
...
ioThread->StartWithOptions(options)
...

ioThread is an instance of BrowserProcessSubThread which inherits from base::Thread, itself a subclass of PlatformThread::Delegate.

StartWithOptions is actually defined in base::Thread. The call will lead to a couple of more calls which will eventually end up at pthread_create. The function started in the new thread is ThreadFunc. ioThread object is passed along the calls and ioThread->ThreadMain() is called in the new thread.

ThreadMain is now running in the IOLoop thread, it will call Init() of the current instance and also create a MessageLoop instance to call the Run() method. We will <come any="" back="" is="" it="" later="" not="" of="" p="" part="" since="" startup="" that="" the="" thread="" to=""> </come>revisit this part later on, since it is not involved further in thread startup.

...
// The message loop for this thread.
MessageLoop message_loop(startup_data_->options.message_loop_type);
...
// Let the thread do extra initialization.
// Let's do this before signaling we are started.
Init();
...
message_loop.Run();
...

Child

For the child IOLoop thread spawn, we have to look at the nuwa process as a child of b2g and also at the forked processes of nuwa.

Exactly how b2g spawns nuwa will be covered later; for now we assume that the nuwa process already exists. Once nuwa is created, we eventually reach the XRE_InitChildProcess function. This function is responsible for creating the IOLoop thread at this line:

process = new ContentProcess(parentHandle);

In the ContentProcess constructor, the ProcessChild constructor is called, which leads to a call of the ChildProcess constructor. The important part to note here is that the ChildProcess constructor gets passed a new instance of IOThreadChild. Within the constructor, the Run() is called on the passed IOThreadChild object:

ChildProcess::ChildProcess(ChildThread* child_thread)
    : child_thread_(child_thread),
    ...
{
    ...
        child_thread_->Run();
}

From there, the StartWithOptions function is called. At this point it follows the same code path as for the main IOLoop startup. The only exception is that it is an IOThreadChild instance and not a BrowserProcessSubThread (see the b2g process illustration above for reference).

Below is an illustration of the IOLoop thread spawn in the  nuwa process :

This is the case for the original nuwa process. All future children will be forked from nuwa, and since fork only copies the thread it was called in to the new process, all threads so far would be lost.

We want to have all threads from nuwa (along with the IOLoop) in the forked process. In order to do that, pthread_create is not called directly, instead, the call is routed to __wrap_pthread_create which wraps the real pthread_create. The purpose of the wrapper function is to maintain a static list of startup information for all created threads (sAllThreads). This list will be copied to the new process, and the new process will then call RecreateThreads to restore all threads based on the information maintained in the list.

Channel

In order to be able to send and receive messages, we have to create a channel between the parent and the child. This section covers the classes used for this - the actual setup between parent and child will be covered once we get to the process spawning part.

Here is a short illustration of the call flow:

1.  This flow illustrates the creation of an IPC::Channel instance (the process of creating the instance will be covered later on). This class has two important attributes:
   • channel_impl_ which is the actual implementation of the channel (platform specific)
   • listener_ which is used to pass incoming messages to
2. The posix class for the channel_impl_ object can be found here (Channel::ChannelImpl). Channel::ChannelImpl has the following important attributes:
   • pipe_ the file descriptor of the pipe (created by socketpair) for the parent side
   • client_pipe_ the client end of the pipe
   • listener_ the object that receives the incoming messages
   • output_queue_ a queue where all outgoing messages are pushed to
3. Channel::ChannelImpl has two overloaded constructors which can be used to create an object. One of them takes a file descriptor as the first argument which will be stored in pipe_. The more interesting constructor is the one which takes a channel_id (which can also be empty). Both of them also take a Mode and a Listener* pointer as second and third argument. Mode just specifies if we are the server or the client. When the constructor with the channel_id is called, CreatePipe will be called from there. We have to distinguish two different cases from here:
   • Mode == MODE_SERVER: In this case, socketpair will be called. One end of the pipe will be stored in pipe_ the other in client_pipe_. If channel_id is not empty, we insert a new entry in a PipeMap where we associate client_pipe_ with the given channel_id.
   • Mode != MODE_SERVER: In this case, we call ChannelNameToClientFD, which looks inside the PipeMap for an entry with the given channel_id. The result will be stored in pipe_.
4. After the object creation is completed, the Connect method can be called. This method will tell libevent to notify us whenever something has been written to pipe_ and is ready to be received.
5. OnFileCanReadWithoutBlocking is the callback for this event. This function will then call a function to read the message from the file descriptor, and then the message will be passed to the OnMessageReceived function inside the listener_ (this will be covered later).

Spawning

In the previous section, we learned how IOLoop is created and how a channel is created. Throughout the last sections, we made the assumption that a process has already been started. This section will cover how those processes actually get started and how they connect to the IPC::Channel. We will again have to distinguish between the nuwa process and the children of nuwa.

At this point, if you are not already familiar with IPDL consider reading the IPDL Tutorial because from this point on we will reference some of the classes generated from those IPDL files.

Nuwa

Creating the process

Throughout the initialization phase of the b2g process, an instance of the singleton class PreallocatedProcessManagerImpl will be created. This instance is mainly accessed through a couple of static functions defined in the PreallocatedProcessManager class. The purpose of this manager is to keep track of pre-allocated processes. This will be explained in more detail in the #Preallocated section.

The implementation class has two important attributes:

• mSpareProcesses which is an array that contains the preallocated processes (which will be important later on)
• mPreallocatedAppProcess which will be the nuwa process

This initialization happens inside the ContentParent::StartUp function when executing the following code:

...
// Try to preallocate a process that we can transform into an app later.
PreallocatedProcessManager::AllocateAfterDelay();
...

This call will lead to the creation of the one and only instance of PreallocatedProcessManagerImpl (located inside the PreallocatedProcessManagerImpl::Singleton function). Right after the constructor call, the Init function is invoked. Following the call flow from there, we will end up in Enable. Enable will then schedule the nuwa fork, with a 1 second delay (DEFAULT_ALLOCATE_DELAY), by calling ScheduleDelayedNuwaFork. This gives the b2g process enough time to finish its initialization.

As soon as the delay time has passed, the DelayedNuwaFork function is called inside the main thread. Inside the function, we will call ContentParent::RunNuwaProcess which returns a pointer to a ContentParent object; this object represents our nuwa process.

Inside the ContentParent constructor, a couple of interesting things happen.

• we insert the new ContentParent into the global static list, called sContentParent
• we create a GeckoChildProcessHost instance
• we call the LaunchAndWaitForProcessHandle method of GeckoChildProcessHost

The LaunchAndWaitForProcessHandle method will schedule a task inside the IOLoop thread. In the IOLoop thread, RunPerformAsyncLaunch is called. After a few calls, we will end up in the LaunchApp function. This is where the forking happens. After the fork, it will call execve in the child to re-execute itself.

Connecting to the channel

We covered the actual spawning. What's left is the part where the parent (b2g) and the child (nuwa) connect to the same IPC channel. We have two important calls for that on the parent side: one is made before the child is spawned and the other after the spawn. The first one is in the RunPerformAsynchLaunch function. Before actually calling PerformAsynchLaunch (the position is marked with a 'x' in the above diagram), we call InitializeChannel, and this will call CreateChannel. At this point a new IPC::Channel object is created, so please check out the #Channel section above.

The GeckoChildProcessHost object created inside the ContentParent constructor serves as the listener_ inside the IPC::Channel object, thus, GeckoChildProcessHost will supply the OnMessageReceived function. There is nothing done there; it just saves all the incoming messages.

At this point we can consider the parent process to be connected to the channel. This was the first important call.

The second one is called as soon as LaunchAndWaitForProcessHandle returns (nuwa process is running at this point). Since the current OnMessageReceived handler doesn't do any good, we will have to assign a new listener_. In order to do that, this is being executed (mSubprocess is an instance of GeckoChildProcessHost which is created in ContentParent):

Open(mSubprocess->GetChannel(), mSubprocess->GetOwnedChildProcessHandle());

What happens now is a little complicated to explain just with text. I will try to illustrate the process in the end, so you might want to follow the process again with the illustration.

ContentParent actually extends PContentParent (this class is generated from the *.ipdl files that, unfortunately, I can't reference to any github or mxr location) which is defined in ./objdir-gecko/ipc/ipdl/PContentParent.cpp relative to the root directory of FirefoxOS. PContentParent has a member variable which will be important during the Open (defined in PContentParent.cpp) call.

• mChannel instance of MessageChannel

MessageChannel takes a MessageListener object as the one and only argument. mChannel is created during the PContentParent construction and passes this as the MessageListener object to MessageChannel (ContentParent extends MessageListener shown below).

Open gets the IPC::Channel instance taken from mSubprocess (TODO: GetOwnedChildProcessHandle???) and calls Open on mChannel

auto PContentParent::Open(
        Channel::Transport* aTransport,
        ProcessHandle aOtherProcess,
        MessageLoop* aThread,
        mozilla::ipc::Side aSide) -> bool
{
    mOtherProcess = aOtherProcess;
    return (mChannel).Open(aTransport, aThread, aSide);
}

As a side note, Channel::Transport is a typedef of IPC::Channel. aThread and aSide are set to 0 and UnknownSide by default if not specified. Open inside of the MessageChannel class will create a ProcessLink. This is passed to the constructor which will set mChan inside the MessageLink class which ProcessLink inherits from.

After an instance of ProcessLink is created, we call Open on it. This sets the member variable mTransport (which represents the IPC::Channel). So mTransport is the IPC::Channel pointer we retrieved from mSubprocess.

Since Connect has already been called when the IPC::Channel was created, we will schedule IOLoop to run OnTakeConnectedChannel. The channel state inside the MessageChannel object will be set to ChannelConnected, and we also call set_listener on the IPC::Channel instance to let it know that ProcessLink will handle incoming messages. That means OnMessageReceived inside ProcessLink is called. Those received messages will be passed to the MessageChannel and from there to the OnMessageReceived funciton inside the PContentParent class.

Now everything has been set up on the parent side. Let's get to the nuwa side.

After the fork, nuwa inherited all the open file descriptors from the parent b2g, and one of them is its end of the pipe. Every child expects its end of the pipe to be file descriptor 3 (kClientChannelFd). In order to guarantee that file descriptor 3 will be the child's end of the pipe, we call ShuffleFileDescriptors. This function will dup the child end of the pipe to 3. It also makes sure that in case 3 is a file descriptor needed by the child that it will be remapped to the next available.

After the file descriptors have been remapped, we call CloseSuperfluousFds to close all the ones that nuwa doesn't need. When everything is done the child will call execve to re-execute itself.

From here please have a look at the #Child section. The actual channel connection will happen inside the ThreadMain function inside the IOLoop thread. ThreadMain will call Init defined in ChildThread. Inside this function, we will create a new IPC::Channel object, and you can refer to the #Channel section from here. You will have to consider the part Mode != MODE_SERVER, therefore ChannelNameToClientFD will be called, which returns our magic file descriptor 3 (remember the child end of the pipe is mapped to this fd).

So now, we have a connection to the channel. What is left is to set the correct listener for the incoming messages. To do so, the Init funciton inside ContentProcess is called. From there it will call Open in the PContentChild class, and this will lead to the same call flow as for the parent in the above diagram. The only exception we have is that these calls originate from ContentChild which inherits from PContentChild.

Preallocated

Creating the process

A preallocated process is a process fork()ed from nuwa which b2g keeps around to turn it later into a app process. At some point in time, b2g tells the preallocated process to turn into an app that has been started on the phone. The process will then be assigned to the privileges of the app, and the app will be able to access privleged functionality via API calls to the parent. As soon as a preallocated process has been turned into an app, b2g will tell nuwa to create a new preallocated process. So how exactly does b2g tell nuwa to create a new preallocated process? (NB: at this point we have an active channel connection between b2g and nuwa.)

For initiating the fork, b2g sends a message through the IPC channel, and nuwa sends one back once it creates the new process. The message contains the PID of the new process. On the right is a little illustration of the message flow.

Sending the initial message is initiated inside the NuwaFork function. Rememeber the mPreallocatedAppProcess is an instance of ContentParent and the invoked SendNuwaFork function is actually implemented inside PContentParent.

As soon as the child receives this messages, it will call RecvNuwaFork which is defined in ContentChild. Following the calls from there, we will end up in ForkIPCProcess.

From there we call PrepareProtoSockets which calls socketpair to create a new pipe for the new child and the parent.
It is also where fork is called and our new process is born. After the fork we will call AddNewProcess inside nuwa. AddNewIPCProcess will then be responsible for initiating the second message. The message contains the parent side file descriptor for the pipe as well as the PID of the new process.

On the b2g side the RecvAddNewProcess will handle the message. It will create a new ContentParent instance. This time we call the second ContentParent constructor. It will create the ContentParent instance based on the already existing one and with the information of the new process. This also leads to a new instance in our static sContentParents list. This basically means that we have a ContentParent instance for each process that is running.

There is also a check in place which makes sure that only the nuwa process actually sends Msg_AddNewProcess:

...
if (!IsNuwaProcess()) {
...

IsNuwaProcess checks a member variable of ContentParent whether or not it is nuwa who sent the message. This member variable can only be set from the parent side, and it does this during the creation of the ContentParent for the nuwa process.

After initialization is finished, RecvAddNewProcess will then call PublishSpareProcess to add the process to the managers list which will keep track of all existing Preallocated processes. Whenever a Preallocated process will be turned into an app process, it will be removed from that spare processes list. Here an illustration of the process:

Connecting to the channel

TODO: write it