Performance tests for tamarin are located in the test/performance directory. Similar to the acceptance tests set the environment variables AVM, ASC, BUILTINABC. The runtests.py test harness works by parsing the abc stdout for lines like: 'metric time 566'. The testconfig.txt file defines groups of tests to run (e.g. sunspider, scimark, language). By default the time for each test is output. Use the --memory flag to capture the maximum private bytes memory (high water mark) for a test. By default each test is run a single time. To run each test multiple times use the --iterations flag. When the number of iterations is greater than 2 all times are displayed in [], faster time is displayed in the output column, and the 95% confidence interval is calculated. The 95% confidence interval is the calculated percentage of the mean value to fall within 95% of the measured results. For example if the mean is 100 and the 95% confidence interval is 2%, 95% of the expected values should fall between 98 and 102. To compare two vms against each other export AVM2=<path to 2nd vm> or use --avm2=<path to 2nd vm>. The percentage difference is calculated between the two vms. See the --help usage for more details about performance tests.
# to run a group of tests (e.g. sunspider) $ ./runtests.py sunspider/*.as sunspider/as3/*.as test avm metric sunspider/access-binary-trees.as 584 time sunspider/access-fannkuch.as 364 time ... # to set vm args (e.g. interp) $ ./runtests.py --vmargs="-Dinterp" sunspider/*.as test avm metric sunspider/access-binary-trees.as 690 time sunspider/access-fannkuch.as 467 time ... # to measure memory $ ./runtests.py --memory sunspider/*.as test avm metric sunspider/access-binary-trees.as 2.4M memory sunspider/access-fannkuch.as 1.5M memory ... # change the iterations $ ./runtests.py --iterations=10 sunspider/*.as test avm 95% conf metric sunspider/access-binary-trees.as 565 1.0% time [565, 586, 573, 577, 567, 567, 579, 576, 565, 585] sunspider/access-fannkuch.as 361 0.3% time [365, 366, 362, 363, 366, 361, 363, 363, 364, 364] ... # compare 2 builds $ ./runtests.py --avm2=build2/avmshell sunspider/*.as test avm avm2 %sp metric sunspider/access-binary-trees.as 565 564 0.2 time sunspider/access-fannkuch.as 366 362 1.1 time ... # compare 2 builds with > 2 iterations $ ./runtests.py --iterations=10 --avm2=build2/avmshell sunspider/*.as
test avm avm2
min : max avg min : max avg %diff metric
----------------------- ----------------------- -----
sunspider/access-binary-trees.as [ 567.0 : 585] 575.9 [ 564.0 : 580] 572.0 0.5 time
sunspider/access-fannkuch.as [ 363.0 : 386] 370.0 [ 362.0 : 399] 371.8 0.3 time
...