WinForms

The purpose of this Strongwind example is to familiarize the reader with the specific testing practices of the UIA Mono Accessibility team.

This tutorial demonstrates the writing of tests for a Gtk application. In practice, of course, we write our tests for WinForms applications. Using a Gtk application in this tutorial allow us to demonstrate testing techniques effectively, without having to worry about the early stage of the WinForms providers implementation at the time this tutorial was written.

Strongwind

Introduction

This tutorial will not have in-depth explanations about the Strongwind-specific parts of the code in our test. This information is covered in Getting Started with Strongwind (http://medsphere.org/projects/strongwind/getting_started.html) and Strongwind Basics. In this tutorial, our focus will be on how our Mono accessibility tests and environment are unique.

Requirements

GNOME (http://www.gnome.org)
Python (http://www.python.org) (>=2.5, <3.0)
Enable "Assistive Technologies" from the GNOME Control Center
pyatspi (packaged with AT-SPI (http://ftp.gnome.org/pub/GNOME/sources/at-spi))
Accerciser (http://live.gnome.org/Accerciser)
Strongwind (http://medsphere.org/projects/strongwind)
uia2atk code. The QA-related code is in the test directory. There are short README files in the test directory and each of its subdirectories. Read these README files if you are confused about the files and directories you are seeing. The code can also be checked out anonymously (using subversion) by running svn co svn://anonsvn.mono-project.com/source/trunk/uia2atk.

Strongwind Introduction

This tutorial assumes the reader already has at least a basic understanding of Strongwind and can comfortably write non-trivial tests using it. If you are already familiar with writing Strongwind tests, please skip this introduction. Otherwise, a brief introduction to is needed before we can begin.

First of all, read Getting Started with Strongwind (http://medsphere.org/projects/strongwind/getting_started.html) to get a basic idea of what it's all about. Also notice there are several Gtk tests in uia2atk/test/testers that can be used as additional examples.

Next, read the Strongwind Basics page.

After reading these two documents, you will be ready to proceed with this tutorial.

Strongwind Test Harness

We have developed a test harness for our Strongwind test harness. The purpose of the test harness is to facilitate the execution and logging of multiple Strongwind tests on multiple machines. Using the Strongwind test harness we are able to execute one command that runs all of our tests--either on your local machine or on all of our test machines simultaneously--and stores all of the logs to their proper locations.

The test harness files are located in uia2atk/test/harness; here is a list of the files with their descriptions:

tests.py: A list of "enabled tests," that is, tests that should be executed when running remote_run.py or local_run.py.
machines.py: A dictionary of machines on which the enabled tests should be run. This file also contains the directory where the tests reside on the machines in machines.py, directory where logs should be stored on the machines in machines.py, and the username for the machines in machines.py.
remote_run.py: Run the enabled tests on the machines specified in machines.py.
local_run.py: Run the enabled tests on the local machine.

Here is a brief overview of how tests are executed using our Strongwind test harness:

local_run.py

When using local_run.py to execute tests, the following occurs:

The list of enabled tests is retrieved from tests.py
If a test is specified in tests.py, but it cannot be found on the machine, a warning is issued and that test will not be run.
Each test from tests.py is execute in order
Logs are stored in uia2atk/test/logs unless otherwise specified (by using the [-l|--log=] option.

Information is also printed to the terminal at each step unless the [-q|--quiet] option is used.

remote_run.py

When using remote_run.py to execute tests, the following occurs:

The list of test machines is retrieved from machines.py
If a machine is specified in machines.py, but it cannot be pinged successfully, a warning is issues and the test(s) will not run on that machine.
remote_run.py is executed via SSH on each available machine from machines.py. There are three important things to remember:
1. remote_run.py's default log directory is overridden and logs are stored in LOG_DIR as defined by machines.py.
2. The user used to perform the SSH operation is defined by USERNAME in machines.py
3. remote_run.py looks for the test directory (i.e., uia2atk/test) in TEST_DIR as defined by machines.py

Information is also printed to the terminal unless the [-q|--quiet] option is used.

The information normally displayed to the terminal by local_run.py is dumped in /tmp/uiaqa unless otherwise specified using the [-l|--log] option.

"Official Tests"

Currently, the "official tests" are run on the "official" test machines, which are located at Novell in Provo. These are the machines that are found in the machines.py in our source code and are only available internally to Novell employees.

We encourage anyone to help us write tests! If you want to submit a test to be added to the official tests and you have not received permission from a QA hacker to check patches into our Subversion repository, please e-mail our mailing list (mono-a11y@forge.novell.com) and attach a patch with the following:

Sample application modifications (recommended) or new sample application. Not applicable if you are writing a test for an already existing application that does not need modified. NOTE: If you are modifying a sample application, make sure it doesn't break any of the existing tests.
New or modified Application wrapper. Not applicable if you are writing a test for an already existing application wrapper that does not need modified.
Test script
Your new test added to tests.py

Please only submit one test per e-mail, and begin the subject line with [QA PATCH].

Official Test Machine Configuration

The official tests are run on the official test machines in the official lab :), which is located at Novell in Provo. Thus, this information is probably only useful to Novell employees.

The test machines are actually virtual machines (running on VMware Workstation) and are hosted on the physical machines in Provo. Here is a list of the test machines and the physical machines on which they are hosted:

Platform	Address	VMware Host
openSUSE 11 32-bit	suse32v0.sled.lab.novell.com	d755a.sled.lab.novell.com
openSUSE 11 64-bit	suse64v0.sled.lab.novell.com	d755b.sled.lab.novell.com
Ubuntu 8.04 32-bit	ubuntu32v0.sled.lab.novell.com	d755c.sled.lab.novell.com
Ubuntu 8.04 64-bit	ubuntu32v0.sled.lab.novell.com	d755d.sled.lab.novell.com
Fedora 9 32-bit	fedora32v0.sled.lab.novell.com	d755e.sled.lab.novell.com
Fedora 9 64-bit	fedora64v0.sled.lab.novell.com	d755f.sled.lab.novell.com

All of the test machines have a VNC server installed. The host machines have NX Server (http://www.nomachine.com) installed, which can be connected to using NX Client (http://www.nomachine.com/download.php); VMware could be running in the background. Running vmware will open the backgrounded session. Both the test machines and the VMware hosts have a user a11y. For more login details contact a QA hacker.

Each official test machine has a tests directory and a logs directory. TEST_DIR in machines.py is set to the tests directory and LOG_DIR in machines.py is set to the logs directory.

These directories are, in truth, mounted directories on uiaqa.sled.lab.novell.com.

**uiaqa.sled.lab.novell.com**
Directory	CIFS Share Path	Description
/var/qa/code/test	//uiaqa.sled.lab.novell.com/test	The uia2atk test code is checked out at this location
/var/qa/logs	//uiaqa.sled.lab.novell.com/logs	QA logs, which can be accessed from http://uiaqa.sled.lab.novell.com/uiaqa_logs/

//uiaqa.sled.lab.novell.com/test is mounted by TEST_DIR and //uiaqa.sled.lab.novell.com/logs is mounted by LOG_DIR. As the above table states, the test code (uia2atk/test) is checked out at /var/qa/code/test. This means that the test code can be updated on all test machines by simply running svn up at /var/qa/code/test on uiaqa.sled.lab.novell.com. The QA logs for our project are stored in /var/qa/logs, which is also pointed to by http://uiaqa.sled.lab.novell.com/uiaqa_logs. This is where all the "official" logs are stored when our tests are run on the official test machines.

QA Architecture Diagram

This diagram summarizes how WinForms testing is performed using Strongwind and the UI Automation test harness.

Example Sample Application

Our team does not test a specific application. Instead, our developers are writing code to make sure that Mono WinForms applications are accessible. This means that we can writer our testing against whatever application(s) we wish. The approach we take is to create small and simple "sample applications." Our sample applications can be found in our code repository in the test/samples directory.

At this point you should check out the uia2atk code if you haven't already.

After you have checked out the uia2atk code, you can find several sample applications in uia2atk/test/samples. For this example, we will use uia2atk/test/samples/gtktutorial.py.

Take a moment to run the gtktutorial and familiarize yourself with the it; start to think of how you could write its application wrapper. You will notice that the main window (or frame) has two buttons. The first button opens a tree view window with several parents and children. The second button opens a window with check boxes and a quit button, just like the application from the Strongwind Basics tutorial.

Example Application Wrapper

As usual, we will begin by writing the application wrapper for our testable application. In this case our testable application will be gtktutorial.py. We will begin by writing our __init.py__ file which describes how to open the testable application and then returns an object of it. After completing the __init__.py piece of the application wrapper we will code the second piece (gtktutorialframe.py), which describes how we can interact with the widgets of the testable application.

init.py

This __init__.py file is very similar to previous ones we have created, and it has the same function: launch the testable application and return an object of it. We begin just like we have in other __init.py examples:

from strongwind import *
 
from os.path import exists
from sys import path
 
def launchTreeView(exe=None):
    """Launch gtktreeview with accessibility enabled and return a TreeView
    object.  Log an error and return None if something goes wrong"""

There is one part, however, that is unique to our UIA QA team. This code is shown below. The purpose of this portion of code is the same as in other examples: make sure we can find the sample application and that it exists. The difference is that now we find the sample application based on where the test file is being run.

path[0] is using sys.path[0], which is "the directory containing the script that was used to invoke the Python interpreter."

So if we execute a test from /home/a11y/uia2atk/test/testers, we know to look for the sample app in ../samples (i.e., /home/a11y/uia2atk/test/samples).

    if exe is None:
        # make sure we can find the sample application
        harness_dir = path[0]
        i = harness_dir.rfind("/")
        uiaqa_path = harness_dir[:i]
        exe = '%s/samples/gtktreeview.py' % uiaqa_path
        if not exists(exe):
          raise IOError, "Could not find file %s" % exe

The remainder of the __init__.py code is the same as always:

    args = [exe]
 
    (app, subproc) = cache.launchApplication(args=args)
 
    treeview = GtkTreeView(app, subproc)
    cache.addApplication(treeview)
 
    treeview.gtkTreeViewFrame.app = treeview
 
    return treeview
 
# class to represent the application
class GtkTreeView(accessibles.Application):
    def __init__(self, accessible, subproc=None):
        'Get a reference to the Tree View window'
        super(GtkTreeView, self).__init__(accessible, subproc)
        self.findFrame(re.compile('^Tree View'), logName='Gtk Tree View')

gtktutorialframe.py

The second portion of the application wrapper (gtktutoriaframe.py) is no different than previous examples. The code below is aggressively documented:

import sys
import os
 
from strongwind import *
from gtktreeview import *
 
 
# class to represent the main window.
class GtkTreeViewFrame(accessibles.Frame):
 
    # constants
    # the available widgets on the window
    COLUMN_ZERO = "Column 0"
    PARENT_ONE = "parent 0"
    PARENT_TWO = "parent 1"
    PARENT_TREE = "parent 2"
    PARENT_FOUR = "parent 3"
 
    # the ordering of table cell names we expect when we call the
    # findAllTableCells method
    ASCENDING = ('parent 0', 'parent 1', 'parent 2', 'parent 3',
                 'child 0 of parent 0', 'child 1 of parent 0',
                 'child 2 of parent 0', 'child 0 of parent 1',
                 'child 1 of parent 1', 'child 2 of parent 1',
                 'child 0 of parent 2', 'child 1 of parent 2',
                 'child 2 of parent 2', 'child 0 of parent 3',
                 'child 1 of parent 3', 'child 2 of parent 3')
    DESCENDING = ('parent 3', 'parent 2', 'parent 1', 'parent 0',
                  'child 2 of parent 3', 'child 1 of parent 3',
                  'child 0 of parent 3', 'child 2 of parent 2',
                  'child 1 of parent 2', 'child 0 of parent 2',
                  'child 2 of parent 1', 'child 1 of parent 1',
                  'child 0 of parent 1', 'child 2 of parent 0',
                  'child 1 of parent 0', 'child 0 of parent 0')
 
    # Use the constructor to find some of the accessibles we will be testing
    # and store an object for each accessible in a variable that can be used
    # from the test script.
    def __init__(self, accessible):
        super(GtkTreeViewFrame, self).__init__(accessible)
        self.column0 = self.findTableColumnHeader(self.COLUMN_ZERO)
        self.parent0 = self.findTableCell(self.PARENT_ONE)
        self.parent1 = self.findTableCell(self.PARENT_TWO)
        self.parent2 = self.findTableCell(self.PARENT_TREE)
        self.parent3 = self.findTableCell(self.PARENT_FOUR)
 
    # expand accessible by performing an "expand or contract" action.
    # this method will also contract the accessible if used out of order
    def expand(self, parent):
        procedurelogger.action('expand %s.' % parent)
        parent.expandOrContract()
 
    # contract accessible by performing an "expand or contract" action.
    # this method will also expand the accessible if used out of order
    def contract(self, parent):
        procedurelogger.action('contract %s.' % parent)
        parent.expandOrContract()
 
    # perform a click action for TableColumnHeader
    def tchClick (self, thc):
        procedurelogger.action('click %s.' % thc)
        thc.click()
 
    # assert that the accessible has the "contracted" state after performing
    # the "expand or contract" action.  states can be found on the "status" list
    # on the "interface viewer" tab in Accerciser.
    def assertContracted(self, accessible):
        'Raise exception if accessible does not match the given result'   
        procedurelogger.expectedResult('%s is %s.' % (accessible, "contracted"))
        def resultMatches():
            return not accessible.expanded
        
        assert retryUntilTrue(resultMatches)
 
    # assert that the accessible has the "expanded" state after performing
    # the "expand or contract" action.
    def assertExpanded(self, accessible):
        'Raise exception if accessible does not match the given result'   
        procedurelogger.expectedResult('%s is %s.' % (accessible, "expanded"))
        def resultMatches():
            return accessible.expanded
        
        assert retryUntilTrue(resultMatches)
 
    # assert that the sorting of the TreeView is ascending by comparing the 
    # order in which the table cells are found to the order we expect them
    # to be in (self.ASCENDING).
    def assertAscending(self):
        'Raise exception if the sorting of the tree view is not ascending'   
        procedurelogger.expectedResult('TreeView sorting is ascending')
        self.table_cells = self.findAllTableCells(None, checkShowing=False)
        tcs = [table_cell.name for table_cell in  self.table_cells]
 
        def resultMatches():
            self.table_cells = self.findAllTableCells(None, checkShowing=False)
            return tuple(tcs) == self.ASCENDING
 
        assert retryUntilTrue(resultMatches)
 
    # assert that the sorting of the TreeView is descending by comparing the 
    # order in which the table cells are found to the order we expect them
    # to be in (self.DESCENDING).
    def assertDescending(self):
        'Raise exception if the sorting of the tree view is not descending'   
        procedurelogger.expectedResult('TreeView sorting is descending')
        self.table_cells = self.findAllTableCells(None, checkShowing=False)
        tcs = [table_cell.name for table_cell in  self.table_cells]
 
        def resultMatches():
            self.table_cells = self.findAllTableCells(None, checkShowing=False)
            return tuple(tcs) == self.DESCENDING
 
        assert retryUntilTrue(resultMatches)

Example Test Script

The test script is also no different than previous examples. The code below is aggressively documented and should be fairly easy to follow:

from strongwind import *
from gtktreeview import *
from sys import argv
from os import path
 
app_path = None 
try:
  app_path = argv[1]
except IndexError:
  pass #expected
 
# open the treeview sample application
try:
  app = launchTreeView(app_path)
except IOError, msg:
  print "ERROR:  %s" % msg
  exit(2)
 
# make sure we got the app back
if app is None:
  exit(4)
 
# just an alias to make things shorter
tvFrame = app.gtkTreeViewFrame
 
#expand parent 0
tvFrame.expand(tvFrame.parent0)
sleep(config.SHORT_DELAY)
tvFrame.assertExpanded(tvFrame.parent0)
 
#contract parent0
tvFrame.contract(tvFrame.parent0)
sleep(config.SHORT_DELAY)
tvFrame.assertContracted(tvFrame.parent0)
 
#expand parent 1
tvFrame.expand(tvFrame.parent1)
sleep(config.SHORT_DELAY)
tvFrame.assertExpanded(tvFrame.parent1)
 
#contract parent1
tvFrame.contract(tvFrame.parent1)
sleep(config.SHORT_DELAY)
tvFrame.assertContracted(tvFrame.parent1)
 
#expand parent 2
tvFrame.expand(tvFrame.parent2)
sleep(config.SHORT_DELAY)
tvFrame.assertExpanded(tvFrame.parent2)
 
#contract parent2
tvFrame.contract(tvFrame.parent2)
sleep(config.SHORT_DELAY)
tvFrame.assertContracted(tvFrame.parent2)
 
#expand parent 3
tvFrame.expand(tvFrame.parent3)
sleep(config.SHORT_DELAY)
tvFrame.assertExpanded(tvFrame.parent3)
 
#contract parent3
tvFrame.contract(tvFrame.parent3)
sleep(config.SHORT_DELAY)
tvFrame.assertContracted(tvFrame.parent3)
 
# we should also make sure that clicking on the table header column reorders
# the tree view
 
# evidently, in Gtk the first click just enables ordering
tvFrame.tchClick(tvFrame.column0)
sleep(config.SHORT_DELAY)
# the treeview should still be ascending at this point
tvFrame.assertAscending()
 
# next time we click the sorting should change to descending
tvFrame.tchClick(tvFrame.column0)
sleep(config.SHORT_DELAY)
tvFrame.assertDescending()
 
# and now the sorting should go back to ascending
tvFrame.tchClick(tvFrame.column0)
sleep(config.SHORT_DELAY)
tvFrame.assertAscending()
 
print "INFO:  Log written to: %s" % config.OUTPUT_DIR
 
# close the app using Strongwind's altF4 method (from accessibles.py).
# this is the standard way of closing an application that doesn't have a
# clickable "quit" option., 
tvFrame.altF4(tvFrame)

Orca

Requirements

It is recommended that you use a Virtual Machine (VM) for testing. You should take VM snapshots before and after you have it set up for testing Orca so you can easily revert if something goes wrong later.

GNOME (http://www.gnome.org)
Python (http://www.python.org)
Enable "Assistive Technologies" from the GNOME Control Center
pyatspi (packaged with AT-SPI (http://ftp.gnome.org/pub/GNOME/sources/at-spi))
uia2atk code. The QA-related code is in the test directory. There are short README files in the test directory and each of its subdirectories. Read these README files if you are confused about the files and directories you are seeing. The code can also be checked out anonymously (using subversion) by running svn co svn://anonsvn.mono-project.com/source/trunk/uia2atk.
Install intltool >=0.40.0 (http://ftp.acc.umu.se/pub/GNOME/sources/intltool/0.40/intltool-0.40.3.tar.gz).
Install Orca revision 4178 (version 2.23.92) from source (svn co -r 4178 http://svn.gnome.org/svn/orca/trunk orca) so you can follow along with the examples. We must check out the code fron svn because the test code is in the svn trunk but it not in the tarballs or source packages. Additionally, we want to make sure we use the same revision on all test machines so we do not get varying test results.
Set orca.debug.debugLevel = orca.debug.LEVEL_INFO in your ~/.orca/user-settings.py file. This is explained in Writing Orca Tests (http://live.gnome.org/Orca/RegressionTesting/WritingTests), which is mentioned below and should be read prior to writing tests for Orca.
Install Accerciser (http://live.gnome.org/Accerciser) from source. Here (http://bgmerrell.blogspot.com/2008/07/buildling-accerciser-from-source-on.html) are some instructions to do this easily.
Install Macaroon by running configure, make, and make install in accerciser/macaroon.
Add the uia2atk/test/samples (explained later) directory to PATH. You can do this by running export PATH=/home/a11y/code/uia2atk/test/samples/:$PATH. Add it to your profile script (e.g., /etc/profile.local in openSUSE) and log out for universal and permanent affect.

Testing

Orca already has its own test harness that the Orca team uses to perform regression tests on Orca itself. We will use their harness to execute tests on our own sample WinForms applications to ensure that they are accessible. The documents linked below will help you get familiar with Orca testing. Read them first.

Orca Regression Testing (http://live.gnome.org/Orca/RegressionTesting)
Writing Orca Tests (http://live.gnome.org/Orca/RegressionTesting/WritingTests)

In the uia2atk/test directory we have created a directory named keystrokes. This is where the Orca test scripts for our WinForms sample applications will reside.

It is fairly simple to use Orca's test harness to execute our test scripts. Logs are stored in a directory whose name is of the form YY-MM-DD_HH:MM:SS (e.g., 2006-11-29_20:21:41)

The code below will execute all our test scripts inside the the uia2atk/test/keystrokes/gtk directory. To execute all our WinForms test scripts, that path uia2atk/test/keystrokes/winforms would be used instead.

Results (as explained in the Orca Regression Testing (http://live.gnome.org/Orca/RegressionTesting) document) will be stored in the current/working directory unless otherwise specific with the -r option.

cd /home/a11y/code/orca/test/harness; ./runall.sh \
-k /home/a11y/code/uia2atk/test/keystrokes/gtk > runall.out 2>&1

Important: Each directory name inside of the uia2atk/test/keystrokes/gtk and uia2atk/test/keystrokes/winforms directory is used to execute the application to be tested. Because of this, each directory in the uia2atk/test/keystrokes/gtk and uia2atk/test/keystrokes/winforms directories should have the same name as the sample application that will be tested. Thus, the keystroke test scripts for that sample application are then stored in the directory that has the same name as the sample application that will be tested by that test script. This is required because Orca runs each application to be tested by issuing the name of each directory in uia2atk/test/keystrokes/gtk or uia2atk/test/keystrokes/winforms as a command. That is, the directory uia2atk/test/keystrokes/gtk/gtktutorial.py means that Orca will executed the command gtktutorial.py when the runall.sh is used and the uia2atk/test/keystrokes/gtk is passed as the keystrokes directory argument (-k). This is why we needed to add uia2atk/test/samples to PATH (as discussed in the Requirements section).

Additionally, the following code can be used to execute a single Orca tests. Logs are stored in the current/working directory.

cd /home/a11y/code/orca/test/harness; ./runone.sh \
/home/a11y/code/uia2atk/test/keystrokes/gtk/gtktutorial/gtktutorial_example_1.py \
/home/a11y/code/uia2atk/test/samples/gtk/gtktutorial.py 0 > runone.out 2>&1

For the sake of completeness, it should be noted that Orca's runone.sh script does not require that each directory in the uia2atk/test/keystrokes/gtk uia2atk/test/keystrokes/winforms directory have the same name as the sample application that will be tested. This is because the application to be tested is specified explicitely. However, we should always give each directory in uia2atk/test/keystrokes/gtk and uia2atk/test/keystrokes/winforms the name of the sample application to be tested, so runall.sh always runs successfully.

In the above commands, the test output, which is what we are interested to see test results, is redirected using the ">" character. This is optional. On the official test machines we redirect the test output to the appropriate directory, so it can be viewed easily after the test has been run.

As explained in the Orca Regression Testing (http://live.gnome.org/Orca/RegressionTesting) document, the 0 that follows gtktutorial.py means that Orca is not running at the time the test is run. Therefore, using the above command, Orca will be started and the test will run. Replacing the 0 with a 1 will execute the test if Orca is already running, however, there are some Orca peculiarities (related to settings) that make this approach more difficult. Using 0 should be fine for all of our tests.

Test Writing

Gtk applications are already accessible in Linux. Our approach to testing WinForms via Orca will be to:

Create a Gtk application
Create a WinForms application that mirrors the Gtk application. That is, create a WinForms application that uses the same controls (or at least controls that are very similar and have the same accessible roles), text, title, etc., as the Gtk application
Create a keystrokes test (e.g., uia2atk/test/keystrokes/gtk/gtktutorial/gtktutorial_example_1.py) for the Gtk application.
Use the keystrokes test to test the WinForms application to ensure that the WinForms application is accessible in the same way that the Gtk application is accessible.

Two directories have been created for this purpose:

uia2atk/test/keystrokes/gtk: keystroke tests for Gtk applications
uia2atk/test/keystrokes/winforms: keystroke tests for WinForms applications that mirror the Gtk applications.

Of course, when we run our tests, we will only run the WinForms keystroke tests. We are only saving the Gtk keystroke tests for references, comparison, and examples.

Moonlight

The work to make Moonlight accessible has not yet begun. Check back in 2009!

Mono

Accessibility: Testing Howto

Table of Contents

WinForms

Strongwind

Introduction

Requirements

Strongwind Introduction

Strongwind Test Harness

local_run.py

remote_run.py

"Official Tests"

Official Test Machine Configuration

QA Architecture Diagram

Example Sample Application

Example Application Wrapper

init.py

gtktutorialframe.py

Example Test Script

Orca

Requirements

Testing

Test Writing

Moonlight