Firefox for Android automation using Robocop

Robocop is a Java-based testing framework which is an extension of the Android Robotium test tool with features needed to perform automation on Firefox for Android builds.

Before starting to write tests you need to Set up the Automation Environment.
Here are more resources on how to set up the environment, and run and create automated tests using Robocop:
If you have a Pandaboard you need to set up, check out the Pandaboard setup Wiki page. This gives information on how to install the automation build on the board, some useful adb commands to help you work with it, and some info on the SUTAgent tool.

Limitations of Robocop and where it can be used

All Robocop tests will be run on Tegra and Panda boards. In order to make sure the test runs correctly, make sure it works on phones or tablets running Android 2.3 - this would ensure it runs on the Tegraboards (Android 2.3 and Gingerbread phones UI) - and on 7" Tablets running Android 4.0 - this would ensure it runs on the Pandaboards (Android 4.0.4 and ICS 7" Tablet UI).
No real URLs should be used because the time it takes to load them is not constant, the pages are not static, and they aren't always available. Only use local webpages created for the purpose of the test.
Because it is based on the Robotium test API, it can be used to test the UI elements of Firefox for Android.
Because it is a UI test tool, it can be used to test basic functional tests, but is not the best tool for performance or endurance testing.
The Firefox for Android Reader Mode feature is not completely supported yet.
Robocop receives limited information about the Gecko layer so any Gecko content (Web content) is seen as a picture. You can tap on the content, get the color of a pixel in the content, and drag the content. However, you don't get any information about the scroll position, whether elements on the page are displayed or have been loaded, whether everything on the page is where it should be, whether the content is correctly displayed, or about what type of content it is (text, picture, empty space, video, flash).
It works on some devices but not all. This is also caused by the Android fragmentation, the different Android OS versions and builds.
There are a lot of issues with event timing. The tests will always have to wait for events to happen before doing other actions.

Getting started with Robocop tests

Please make sure to look over this page for some general information about Robocop tests.
The existing test cases can be found in the source directory under the folder mobile/android/base/tests.
The class that will cover the text must be named the same as the test file.
The test file name should be test<Feature_to_be_tested>.java.in.
The test must be included in the same package as all other Robocop tests.

Start writing a new test

Download sources. More info can be found here about how to do this.
If you already have the sources downloaded, update to the latest version by running the following command in the sources directory:

hg pull -u

Make the build by running in superuser (sudo su) and make sure the PATH variable is updated with the Android SDK location:

make -f client.mk

Create a new file named test followed by the test name and with the extension java.in (for example, testBookmarks.java.in).
Copy the basic structure of the test presented in the next section to the test, making sure to match the test name and the class and constructure names.
Create a new Mercurial queue patch on the repository by running in the source folder the following command:

hg qnew <patch_name>

Add the new test file to the queue by running the following command:

hg add mobile/android/base/tests/<test_file_complete_name>

Open robocop.ini from mobile/android/base/tests and add after the last Robocop test a new line with the test name between sqare brackets ( "[" and "]").
Write the test, compile robocop (here is how) and run the test in order to test that it works.
After the test works, follow the instruction to create a patch so the test can be uploaded to a bug and integrated in Mozilla.
Before updating the sources or after the patch is completed, make sure to return the sources to the unmodified version by running the following command in the source folder until there are no patches in the queue:

hg qpop

Basic structure of a test

#filter substitution
package @[email protected];

import @ANDROID_PACKAGE_NAME@.*;
 
/*
Insert info about the test here - a basic description of what the test does
*/
public class test<Feature_to_be_tested> extends BaseTest {
    @Override
    // This method is needed in the BaseTest class
    protected int getTestType() {
        return TEST_MOCHITEST; // Test type should remain Mochitest
    }
    /* 
    * This is the class constructor 
    * This method will contain the test that will be run
    */
    public void test<Feature_to_be_tested>() {
   
    }
}

This is a basic structure of a Robocop test.
Each test class must have three methods:
- protected void setUp() - Starts Fennec and sets up commonly used member variables. This is usually very similar for every test class.
- public void test[YourFeature]() - Your test code goes here. Use the Robocop API to access Fennec elements, click, send keys, and assert conditions.
- public void tearDown() - Clean up.
If your tests extends BaseTest then you will not need to add the methods setUp() and tearDown() as they are already defined in BaseTest.
If you need to do extra cleanup, such as removing entries you added in a database, the tearDown() method can be overwritten, but make sure to call super.tearDown() as needed in the method. The same is true for setUp().

APIs

Robotium itself provides a rich API through the Solo class - javadocs for Solo are available at [1].

Robocop provides an additional API to make common tasks easier. The main interfaces are Actions, Element, and Driver.

Actions provides commonly used non-element-specific actions that can be taken on the application, such as dragging and sending key events.

// Actions
  //This will cause this process to spin until Gecko fires a specific JSON event, such as DOMContentLoaded
  void waitForGeckoEvent(String geckoEvent);
  //Clicks the given Key (Actions.SpecialKey.[DOWN|UP|LEFT|RIGHT|ENTER])
  void sendSpecialKey(SpecialKey button)
  //Sends a string of characters to the system. (Most have been implemented but not all)
  void sendKeys(String keysToSend);
  //Sends a drag action across the screen
  void drag(int startingX, int endingX, int startingY, int endingY)
  // Run a SQL query on the specified database
  public Cursor querySql(String dbPath, String sql);

Element represents each of the available UI objects in Fennec including the Awesomebar, the tabs button, and different lists and menus.

// Element
  //To click the element
  void click()
  //Returns true if the element is currently displayed
  boolean isDisplayed();
  //Returns the text currently displayed on the element, or direct sub-elements
  String getText();

Driver finds elements and provides info about the UI.

// Driver
  //This is used to find elements given their id's name
  Element findElement(String name);
  //This is used for getting information on scrolls. NOTE: It must be used for the next three methods to return useful information
  void setupScrollHandling();
  int getPageHeight(); //The total height of the page
  int getScrollHeight(); //How far down the screen the client has scrolled
  int getHeight(); //The height of the client's view

  //The following are used to give information on the graphical location of the Gecko view on the screen
  int getGeckoTop();
  int getGeckoLeft();
  int getGeckoHeight();
  int getGeckoWidth();

Finally, an evolving set of test base classes - BaseTest, PixelTest, etc. - can be leveraged for some types of tests.

Usable IDs

The following is a list of IDs that can be used with Driver.findElement(). Most of the following IDs have not been tested, so you might have unexpected results, or require increased APIs for them. To know how a given object is used, in mobile/android/base, grep R.id.[id-name] * (from Objdir/mobile/android/base/R.java#id)

abouthome_content
add_tab
addons
address_bar
agent_mode
all_pages_list
awesome_bar
awesome_screen
awesomebar_button
awesomebar_tabs
awesomebar_text
background
bookmark
bookmark_icon
bookmark_title
bookmark_url
bookmarks_list
browser_toolbar
close
container
doorhanger_choices
doorhanger_container
doorhanger_title
favicon
forward
gecko_layout
grid
history_list
info
list
main_layout
notification_image
notification_progressbar
notification_text
notification_title
outline
plugin_container
preferences
quit
recommended_addon_list
reload
save_as_pdf
screenshot
select_list
share
site_security
stop
tabs
tabs_count
title
url

Basic code sequences to do different actions

Most of the code is already covered by existing methods but these code bits will help you understand how Robocop works. Besides looking over these, you should also check out the existing tests.

Load Page

Note that the tests need to use local pages in order to run on the Mozilla setup. Outside pages are not accessible.
Using a URL:

String url = "<insert_link_here>";
loadUrl(url);

Using the robocop tests blank page:

String url = getAbsoluteUrl("/robocop/robocop_blank_01.html");
loadUrl(url);

Check Page Title

There has been a method implemented for this in BaseTest.java.in called verifyPageTitle(String title) but the core code to verify the title is:

Element awesomebar = mDriver.findElement(getActivity(), "awesome_bar_title"); // search the awesomebar title
mAsserter.isnot(awesomebar, null, "Got the awesomebar"); // log "Got the awesomebar" in the logs
assertMatches(awesomebar.getText(), "<insert_title_here", "page title match"); // do a match between expected and actual title

Open app menu

There has been a method implemented for this in BaseTest.java.in called selectMenuItem(String item) but the core code for opening the menu is:

mActions.sendSpecialKey(Actions.SpecialKey.MENU);
//Open the More menu for devices with old style menus
if (mSolo.waitForText("^More$")) { 
mSolo.clickOnText("^More$");
}

Simulate a Back key press:

mActions.sendSpecialKey(Actions.SpecialKey.BACK);

Set up a listener to catch a new page load in a new tab

This is included in the addTab(String url) method from BaseTest.java.in:

Actions.EventExpecter tabEventExpecter = mActions.expectGeckoEvent("Tab:Added"); // Look for a new tab event
Actions.EventExpecter contentEventExpecter = mActions.expectGeckoEvent("DOMContentLoaded"); // Look for page load complete

Wait for the tab and page load

tabEventExpecter.blockForEvent();
contentEventExpecter.blockForEvent();

Wait for Gecko to complete loading

Include this code at the start of each test to wait for Firefox to load completely before starting to do any other actions:

blockForGeckoReady();

Open the Bookmarks tab

private ListView openBookmarksList() {
    Activity awesomeBarActivity = clickOnAwesomeBar();
    // Click the "Bookmarks" tab to switch to bookmarks list
    mSolo.clickOnText("Bookmarks");

    TabHost tabHost = (TabHost)mSolo.getView(TabHost.class, 0);
    return (ListView)tabHost.getCurrentView();
   }

Open the bookmark context menu

View child = list.getChildAt(<nr>); //bookmark nr- numbering starts at 0 
mSolo.clickLongOnView(child); //long tap to open context menu

Similarly you can open the context menu for History and Top Sites items

Open a new tab and check the tab count

expectedTabCount = <nr_of_expected_tabs>;
tabCountText = tabCount.getText();
tabCountInt = Integer.parseInt(tabCountText);
mAsserter.is(tabCountInt, expectedTabCount, "Number of tabs increased");

Loging info in the Android Logs

The Assert class can be found in the source folder in the subfolder /obj-android/build/mobile/robocop.
Please see the Java code for the Assert methods for more info about how they work.
Each Assert will determine if a test fails or passes.

Test that an object is not the same as a second object

Method signature:

public void isnot(Object a, Object b, String name)

Example for checking that there is a title set for the current page:

Element awesomebar = mDriver.findElement(getActivity(), "awesome_bar_title");
mAsserter.isnot(awesomebar, null, "Got the awesomebar"); // Checks that the awesomebar title is not null and posts in the logs

Test that an object is the same as a second object

Method signature:

public void is(Object a, Object b, String name)

Example for checking the default bookmarks:

ListView list = getAllPagesList(url);
mSolo.waitForText(url);

mAsserter.is(list.getChildCount(), 5, "all pages list has 5 children (the default bookmarks)"); // Checks that 
there are 5 entries in the list and posts in the logs

Write text in the Logs

Method signature:

public void dumpLog(String message) // writes text in the logs

Log info about test progression

Method signature:

public void ok(boolean condition, String name, String diag)

Example

ListView list = getAllPagesList(url);
mSolo.waitForText(url);

mAsserter.ok(list != null, "checking that all pages list exists", list.toString()); // tests if the list is null and
                                                                                    // if it isn't it prints the list 
                                                                                    // after printing the message of 
                                                                                    // what the test does

Updating your test directory

It is not always necessary to rebuild Fennec to use new tests. If you are updating the build with your own test, ensure that your test has been added to the manifest in /PATH/TO/FENNEC/SOURCE/mobile/android/base/tests/robocop.ini.

Otherwise, you can retrieve tests from the repository by executing:

hg pull
hg update

Now you can build the Robocop part of Fennec by executing:

cd objdir
sudo make -C build/mobile/robocop/
sudo make package

Testing and submitting the test as a patch

Make sure your code follows the basic coding style guidelines. In this article there are a few guidelines that should be followed.
Make sure you run the test at least five times to make sure there are no intermitent fails caused by timing.
If you have access to the tryserver (Commit access level 1) run the patch on the tryserver to see if it passes the tests.
Once you are sure the test works you can create a patch and add it to a bug. If there isn't a bug to cover the creation of the test, then create one.
If you need help creating a patch please read the documentation covering this from this wiki page.