Moonlight Test Harness

The Developer Regression Test List (drtlist)

Tests are specified in an XML file called a drtlist. The drtlist should contain a <DRTList> root element and each test should be specified with the <Test> element. The following attributes can be set on a test. Any other attributes will be ignored.

id A readable name to identify the test
inputFile Mandatory: The test file, either a .xaml or .htm(l) file
masterFile masterFile10 or masterFile11 Mandatory: The master image that the result file will be compared to. If the test does not need an image comparison use "None" as the value for masterFile. Only one of masterFile, masterFile10 and masterFile11 should be specified. In the future, multiple master files will be allowed to test multiple versions of Moonlight.

Master files can be either PNGs or TIFFs. PNGs should be used for simgle image capture tests and TIFFs should be used when multiple images are captured. Each capture will be saved as another layer in the TIFF.

codebehind A C# codebehind file can be specified for the the 2.0 tests in the format
[, dll name]. For example codebehind="nested-canvas.cs" or codebehind="nested-canvas.cs, mylib.dll"
|- valign="top"
|categories
| A comma separated list of categories that the test is a member of.  Categories are explained further in the Running Tests section.
|- valign="top"
|expectedError
|The ErrorType of ErrorArgs passed to the error handler when an error is encountered in the test.  If the error does not occur, or the error is of the wrong type the test will fail.  This attribute will only work for xaml tests, since the error handling functionality is built into the html template generated for the .xaml tests.
|- valign="top"
|knownFailure
|Tests the implement a feature that is known to be broken or not implemented should be marked as a known failure.  Marking tests as a known failure makes it easier to differentiate between failures that are regression and failures that are known.
|- valign="top"
|failureReason
|Failure reasons should be given for all knownFailures.  This allows people triaging tests to easily understand why tests are failing.
|- valign="top"
|ignore
|Tests that may hang or crash the test harness should be marked as ignore.  These tests will not be run during a test run.
|- valign="top"
|ignoreReason
|Tests that are marked as ignored should have a reason assigned to them.
|- valign="top"
|captureSpecifier
|A capture specifier is normally used for taking multiple screenshots.  The capture specifier is a compacted method of setting the captureInterval maxImagesToCapture and initialDelay.
|- valign="top"
|captureInterval
|When taking multiple screenshots the captureInterval is the time to wait inbetween captures.
|- valign="top"
|maxImagesToCapture
|The total number of screenshots that should be captured.
|- valign="top"
|initialDelay
|The time to wait before beginning image capture.
|- valign="top"
|captureWidth
|The width of the image to capture.
|- valign="top"
|captureHeight
|The height of the image to capture.
|- valign="top"
|imageCompareTolerance
|The tolerance for image comparisons can be overriden with this attribute.  The default tolerance is two percent. To make the comparison 1% you would use imageCompareTolerance="0.01".
|- valign="top"
|xspExecDir
|Some tests may want to have xsp running in the background.  For example tests that attempt to connect to special web handlers or need to interact with an aspx page.  The xspExecDirectory is the directory that xsp should be run from.
|- valign="top"
|remote
|Some tests may reside on remote file systems / on the web. This flag (valid value is 'True') tells the test harness that it shouldn't try to modify the test to make it run correctly, nor try to resolve any paths, prepend current directory, etc, which would otherwise be done.
|}
 
== How to run tests ==
 
If you are running the test harness from the moon/test directory there are a couple of make targets you can use:
 
{| border="1"
|- valign="top"
|make run-tests
|This will run all of the tests in moon/test/xaml/drtlist.xml that are not marked as ignore or knownFailure.
|- valign="top"
|make run-all-tests
|This will run all of the tests in moon/test/xaml/drtlist.xml that are not marked as ignore.
|- valign="top"
|make list-known-failures
|This will list all of the tests marked as knownFailure in moon/test/xaml/drtlist.xml and their reason for failing.
|- valign="top"
|make run-drtlist
|This will run the drtlist specified in the environment variable DRTLIST, otherwise it'll work as 'make run-tests'
|- valign="top"
|make run-valgrind
|This will run the drtlist specified in the environment variable DRTLIST with valgrind, otherwise it'll work as 'make run-tests'
|- valign="top"
|make run-gdb
|This will run the drtlist specified in the environment variable DRTLIST with gdb, otherwise it'll work as 'make run-tests'
|}
 
 
If you are running the test harness outside of the make targets, these options are available to you:
 
{| border="1"
|-
| -v, -verbose
|Increase the verbosity level.
|- valign="top"
| --fixture=, -fixtures=
|Specify one or a list of fixtures to run.
|-
| -categories=
|Specify one or a list of categories to run.
|- valign="top"
| -drtlist=
|Specify the path to the drtlist.  If the drtlist is not specified, the test runner will attempt to load drtlist.xml from the current directory.
|- valign="top"
| -agviewer=
|Specify the path to the agviewer binary.  If the agviewer binary is not specified, the harness will attempt to find agviewer by searching recursively down from the current directory and using the PATH environment variable.
|- valign="top"
| -no-console-report
|Disable reporting results to the terminal
|- 
| -html-report, -generate-html-report
|Create an HTML report for the test run
|-
| -run-known-failures
|Run tests marked as known failures
|-
| -list-known-failures
|List tests marked as known failure and their reason for failing.
|-
| -compare-to-moon
|Compare to the moon master files instead of the master files specified in the 
drtlist.
|-
| -valgrind
|Execute moonlight with valgrind. Valgrind will be executed with --tool=memcheck, unless MOONLIGHT_VALGRIND_OPTIONS is set, in which case it will be executed with that value.
|-
| -gdb
|Execute moonlight with gdb. There is currently no way to pass options to gdb. If -valgrind is specified, this flag is ignored.
|-
| -generate-master-files	
|Generate master files to be used with the compare-to-moon option.
|}
 
 
There are two ways of running a subset of the drtlist.xml tests.  You can use categories or specify the tests you would like to use:
 
=== Categories ===
 
Test categories are specified in the drtlist.xml with the categories="<cat one>,<cat two>,<cat three>,...." attribute.  If a test was both a text test and a brushes test, it's XML would look like this:
 
     <Test id="MyTest"
          inputFile="MyTest.xaml"
          masterFile="MyTestMaster.png"
          categories="text,brushes"
      />
 
Category names are case sensitive strings. A new category can be created by adding it's name to the categories attribute. Currently these categories are used in our tests:
 
* text
* animation
* brushes
* parser
 
There are two ways of running category tests.  If you are using the make targets mentioned above, the easiest way is to specify the categories with the MOON_DRT_CATEGORIES=<cat name>,<cat name>,... environment variable.  To run the text and animation tests using this variable your command line would look like this:
 
    $ MOON_DRT_CATEGORIES=text,animation make run-tests
 
If you are not using the make targets you can specify the categories you would like to run on the command line with the -categories flag:
 
    $ mono test-runner.exe -categories=text,animation
 
You can also exclude categories, if you want to run all tests except text and brushes tests, you could do:
 
    $ MOON_DRT_EXCLUDE_CATEGORIES=text,brushes make run-tests
 
 
=== Specifying tests ===
 
If you would like to limit a test run to just a single test, or a small set of tests, that aren't in the same category, you can specify the tests to run.  If you are using the make targets listed above, the easiest way to specify a single or set of tests to run is with the MOON_DRT_FIXTURE=<test id> and MOON_DRT_FIXTURES=<test id>,<test id>,<test id>,... environment variables.  To run tests 2 and 5 using this method your command line would look like this:
 
     $ MOON_DRT_FIXTURES=2,5 make run-tests
 
If you aren't using the make targets you can specify the tests on the command line with the -fixture or -fixtures flags:
 
     $ mono test-runner.exe -fixtures=2,5
 
Id's are specified in the drtlist.xml with the id attribute:
 
     <Test id="2"
          inputFile="MyTest.xaml"
          masterFile="MyTestMaster.png"
          categories="text,brushes"
     />
 
You can also specify a different drtlist.xml, this will run the drtlist in media/video/drtlist.xml:
 
     $ DRTLIST=media/video/drtlist.xml make run-drtlist
 
=== Using Xvfb ===
When running the test suite, you may be interested in using Xvfb. Xvfb is a virtual buffer that can be used for screen captures instead of your screen. This allows you to continue working while the test suite is running in the back ground. 
Setup the virtual buffer like so:
<pre>
     $ Xvfb -fp /usr/share/fonts/misc -ac -screen 0 1024x1024x24 -nolisten tcp :3
</pre>
Then in another terminal, run the test suite and redirect the display to the virtual buffer.
<pre>
     $ DISPLAY=:3 make run-tests
</pre>
 
== While the tests are running ==
 
Depending on the verbosity level and the reporting options, test results will print out on the console as they are completed.  The following legend is for the default verbose level and reporting options:
 
{| border="1"
|-
|.
|The test passed
|-
|F
|The test failed
|-
|K
|The test was marked as a known failure.
|-
|I
|The test was ignored, either because the master file couldn't be found, or because it was marked as ignore in the drtlist.
|}
 
If you would like to abort a test run, press the Escape key while a test is running.  This will signal the test harness to stop the run once this test has completed.  If you press the escape key again, it will cause the currently running test to abort.
 
== When the test has finished ==
 
Generally any test will shutdown the browser when it has finished (the test harness waits for the browser to shut down before executing the next test). However, there may be times when you don't want the browser to shut down when the test has finished (to inspect state, you have other tabs open, etc). In this case you can use the MOONLIGHT_SHOCKER_DONT_DIE flag, which will prevent the browser from shutting down:
 
    $ MOONLIGHT_SHOCKER_DONT_DIE=1 make run-tests
 
== Reading test output ==
 
By default the tests will output a summary of the test run on stdout. The verbosity level will effect the amount of data in the summary.  The verbosity levels are:
 
#Show data from the test plugin
#Show data printed to stderr during the test run
#Show data printed to stdout during the test run
 
If the -html-report option is used (our make targets use this option) there will also be an html_report/ directory created with an html report and all of the master files, result files and image comparison files.   Multiple image capture which are normally saved as tiffs will be converted into a png mosaic of all of the layers of the tiff.