Investigating memory regressions

Investigating any memory regressions detected by endurance tests takes time. It can often involve building Firefox to determine when regressions are introduced. Identifying and raising a memory regression is already covered here.

Once a regression has been identified and raised, the next step is to replicate the regression locally using the Nightly builds. For obvious reasons this should be done on a platform that has demonstrated the regression. Download the latest appropriate Mozmill environment, extract it and then execute run.sh (or run.cmd for Windows). From the resulting command line prompt you can download the last known good build by running the following command:

mozdownload --type=daily --build-id=BUILD_ID --directory=BUILD_ID

BUILD_ID relates to the build ID of Firefox, so simply repeat this for the first build that demonstrates the regression. Then you can run the endurance tests for each build, one after the other, using the following command line: testrun_endurance --iterations=10 --entities=10 --report=https://mozmill-crowd.blargon7.com/db/ BUILD_ID .

Substitute BUILD_ID for the values used when downloading the binaries.

Once this has run you can check the reports here to see if you've reproduced the regression. If you haven't, then the problem might be related to something specific on the Mozmill CI build machines. In this case, please update the regression bug in Bugzilla with your findings. If you have replicated the memory regression then that's excellent news! What we need to do now is identify a smaller regression range. If this regression was detected recently then you may have the tinderbox builds available to you, but if not you may need to build Firefox... Using the changeset of the last good build and the changeset of the first bad build, you can get a push log. Open this page and enter the from/to changesets in the boxes provided. Look through the changes and see if anything jumps out at you that may have caused the regression - it might not be (and rarely is) this easy, but it's worth a shot!

If there is a concerning changeset, then you should download the tinderbox builds surrounding that change to see if it caused the regression. Tinderbox builds aren't always available for long so the sooner we react to a memory regression the better. If you need to bisect and build Firefox then you'll find these instructions valuable. If you have any difficulties with any of these steps, come speak to us in #automation on irc.mozilla.org.

Identifying memory regressions

The endurance tests are a good way to notice memory regressions. These run every day on nightly builds, and essentially run a small snippet over and over. You can read more about the endurance tests on the project page. Results from each of the testruns are plotted in a chart on our dashboard, which can be filtered by date, Firefox version, and platform. Any spike in duration or memory persisting for two or more consecutive days should be considered a regression and filed in Bugzilla for further investigation.

Reporting memory regressions

As soon as you've identified a regression we need a bug filed. What we look for in a good bug report:

• Build ID, platform, and date the regression first appeared
• Link to the Endurance report(s) displaying the regression
• Mozilla changeset reported in the endurance report
• Name of tests which cause the memory spike
• If you want, you can use this template
• Make sure you CC ashughes and davehunt on the bug

Once you have filed the bug, we will work with you to investigate if A) it's a problem with a test or B) it's a Firefox regression.