A test plan for the Accessibility team's efforts in bringing accessibility to mono by implementing the managed UI Automation framework.
No meetings scheduled
UIA---Microsoft UI Automation. A managed code application programming interface (API), exposing user interface controls for test automation and assistive technology. Part of the .NET framework starting at 3.0. Successor of MSAA (Microsoft Active Accessibility)
UIA Clients---Applications such as screen readers and testing frameworks written in managed code (e.g., C#/VB).
UIA Providers---UI implementations or application controls such as checkboxes. Written in managed code or C/C++.
AT---Assistive technology. A generic term that includes assistive, adaptive, and rehabilitative devices and the process used in selecting, locating, and using them.
AT-SPI---A toolkit neutral way of providing accessibility facilities in applications. AT-SPI can also be used to automate testing of user interfaces. AT-SPI is currently supported by GTK+2, JAVA/Swing, Mozilla, and StarOffice/OpenOffice. For our product, AT-SPI will act as the equivalent of the UIA core.
ATK---Accessibility toolkit. A developer toolkit that allows programmers to use common GNOME accessibility features in their applications.
ATK/UIA Bridge---Mapping of ATK to the UIA provider APIs.
UIA/at-spi Bridge---Mapping of AT-SPI to the UIA provider APIs.
WinForms---One of the many GUI Toolkits for use with Mono, working towards compatibility with Microsoft's System.Windows.Forms.
Moonlight---The Mono-based implementation of Silverlight.
Accerciser---An interactive Python accessibility explorer for the GNOME desktop. It uses AT-SPI to inspect and control widgets, allowing you to check if an application is providing correct information to assistive technologies and automated test frameworks.
Orca---Open source scriptable screen reader. Using various combinations of speech, braille, and magnification, Orca helps provide access to applications and toolkits that support the AT-SPI (e.g., the GNOME desktop).
IronPython---Implementation of the Python programming language, targeting .NET and Mono. It makes .NET libraries easily available to Python programmers, while maintaining full compatibility with the Python language.
CPython---The default, most-widely used implementation of the Python programming language. It is written in C, hence the name CPython.
Strongwind---A GUI test automation framework inspired by dogtail. Strongwind is object-oriented and extensible. Strongwind is written in Python and uses the pyatspi library to manipulate and query the state of applications.
This document is a comprehensive test strategy to be used by testers to ensure proper and exhaustive testing. It also acts as a reference for all stakeholders wishing to obtain information on current and past testing efforts.
Test plan identifier
The purpose of this section is to identify the current and previous versions of the test plan, its level, and the version of software it pertains to.
|ID||Level||Software Version||Modify Time||Author|
|Accessibility-TP-V0.1||Draft||N/A||03-25-2008||Brian & Calen|
|Accessibility-TP-V1.0||Initial Release||N/A||03-25-2008||Brian & Calen|
Testing efforts will be related to the project goals, which are:
- Make mono WinForms accessible. All WinForms currently supported by mono.
- Make Moonlight accessible
- Allow UI Automation based Accessibility Technologies to run on Linux
This plan includes integration, system, and acceptance testing. Unit testing is excluded, as it is being managed by individual developers.
Integration testing to test WinForms accessibility via UIA provider and UIA/ATK bridge to AT-SPI is scheduled to be completed in 2008.
System testing will be performed in 2009. Testing will be performed using an AT management tool to test the client and provider.
Acceptance testing will be performed later in 2009 before the product is released. During system testing, the product should be tested in its entirety from the end user's point of view.
According to the project plan, test items is below:
Software risk issues
This section describes any risks resulting from lack of time and/or resources.
- frequent changes to requirements and/or design.
- loss of support from Novell/Microsoft upper management
Developers are also tracking problems they encounter:
Features to be tested
Abstraction of test items. What will be tested from the user or customer point of view.
According to Q2 2008 of the Accessibility#Roadmap roadmap, testing contents need relate with below info:
- The WinForms sheet in WinForms Controls list defines which WinForms controls will be implemented and therefore need to be tested.
- Create WinForms application samples to test against. These application samples should be written in IronPython. Our sample applications can be found at svn://anonsvn.mono-project.com/source/trunk/uia2atk /test/samples. Some C# samples (that can be translated) can be checked out via svn from http://anonsvn.mono-project.com/viewvc/trunk/winforms.
- Test WinForms applications samples using Accerciser to ensure that the samples are accessible.
- Test WinForms applications samples using Orca to ensure that the samples are accessible.
- Write automated scripts using Strongwind to verify accessibility of all WinForms controls.
Stuffs for WinForms test:
- Define what WinForms controls will be tested
- Define what WinForms application samples will be used
- Create custom WinForms applications in IronPython
- Test WinForms application samples with Accerciser to ensure accessibility
- Test WinForms application samples with Orca to ensure accessibility
- Create automated test suite in Python with StrongWind framework to ensure accessibility of all WinForms controls
- Review automated test suites with each other by using Review_Board, Please read UsersGuide before you use it.
According to Silverlight doc that we have 32 Moonlight controls will be implemented, so we should test 32 controls whether they are accessible. (there might be some differences between silverlight and moonlight, it's not sure yet)
- The Moonlight sheet in Moonlight Controls List defines what Moonlight controls should be tested.
- Moonlight test sample will be written in C# and xaml
- Moonlight test script will be written in Python with Strongwind framework
Testing for Moonlight test:
- Define what Moonlight elements will be tested
- Define what Moonlight application samples will be used
- Create or reuse Moonlight applications
- Test Moonlight application samples with Accerciser(against Firefox web browser?)
- Create automated test suite to ensure the accessibility of all Moonlight elements
- Test Moonlight application samples with Orca
- Define what functional should be test for each control according to Provider_Functional_Specification
- Test each controls to ensure provider interface is implemented
- Define what functional should be test for each control according to Bridge_Functional_Specification
- Test each controls to ensure ATK interface is showing correct information
- Test each controls to ensure AT-SPI is showing correct information
UIA Client API
UIA verify is an open source test tool like UISpy running on Windows, developer can use it to watch controls properties and patterns, it will be posted running on Linux, so we will need to make sure the behavior of UIA verify on Linux is match to Windows.
White is a thin wrapper of UIAutomationClient that is like Strongwind wrap pyatspi, developer will post it running on Linux, so we will use White framework to write tests for Client API to against Winforms, Moonlight, Gtk+ applications. The work path of using White to test application is: C# -> White -> UIAutomationClient -> DBus -> UIA -> applications(Winforms/Moonlight/GTK+)
Testing for UIAutomation Client API test:
- Design UserCases of the real application(We will use KeePass), analyze the coverage of each ControlPattern's properties and methods
- Write tests using White in C# for the real application
- Run the tests on Linux to make sure they do the same behavior as on Windows
- Design UserCases of the real application, analyze the coverage of each ControlPattern's properties and methods
- Write tests using White in C# for Moonlight apps on Windows
- Run the tests on Linux to make sure they do the same behavior as on Windows
- Test UIA verify on Linux to make sure the behavior is match with on Windows. UIA verify have provided automation tests, we can run all the applications on Linux and Windows, then to verify we get the same result of how many tests pass or fail
- Design UserCases of the real application(gedit?), analyze the coverage of each ControlPattern's properties and methods
- Write tests using White for GTK+ apps
- Run the tests on Linux to make sure Client API is worked for GTK+ app
Features not to be tested
Any features that will not be tested and why. Decisions not to test a feature should be based on priority and risk. If a feature will not be tested, define what features (class, methods, properties) will not be tested.
At this time we plan to test all features exhaustively.
A description of how and where testing will be performed and explain any issues that have a major impact on the success of testing and ultimately on the project.
The accessibility of WinForms applications will be tested using Strongwind tests and WinForms sample applications. A test harness has also been developed to facilitate the execution and logging of a suite of tests. For more information see the WinForms portion of the Testing Howto.
The accessibility of Moonlight applications will be tested using Strongwind tests and Moonlight sample applications. Moonlight application will be writted in C#. For more information see the Moonlight portion of the Testing Howto.
UIAutomation Client API Testing
- All testers shall be on the team IRC channel (#mono-a11y on irc.gimp.org) during work hours.
- Tests shall be automated whenever possible. Time constraint is not a good excuse not to automate.
- Test plan will be developed using the IEEE Std. 829-1998 Standard for Software Test Documentation.
- All bugs shall be logged in Bugzilla at the time they are found.
- Bugzilla can be accessed from http://bugzilla.novell.com with Novell account. The product category is "UI Automation," which can be found under the Mono classification.
- When a bug's status is changed to RESOlVED or VERIFIED in Bugzilla, the reporter of the bug should change the bug's status to CLOSED or reopen the bug as soon as possible. If the person who reported the bug verifies the bug, the bug can be closed without having its status changed to VERIFIED. A bug should not be closed by someone who did not report the bug unless the reporter is unavailable.
- Types of system testing include function, performance, security, load, reliability, usability, documentation testing.
- Acceptance criteria for patch acceptance: Before a patch is accepted, a QA engineer must ensure that the patch submitted from developer passes QA testing. A build engineer must ensure the patch builds properly and meets packaging standards. QA and build engineers will then create a patch acceptance report, and the patch can be included in the product.
- Testers may perform system testing on the product only after development has verified that they have completed a development milestone and the build team has created a stable release.
- WinForms samples will be created in C#, Boo, or IronPython(here we chose IronPython). Automation scripts, that test the accessibility of the WinForms apps will be created in CPython. Strongwind and Orca Regression Tests will be used for the automation scripts.
- No regularly scheduled meetings at this time
- Minor editing (grammar and spelling corrections) of this test plan can be done at any time. Any change to the test plan that changes how the product will be tested shall be approved by the QA team who will determine if the changes are large enough to require a change to the test plan identifier.
- Black box and white box testing methods are both acceptable. However, it is anticipated that black box testing will be the norm.
- Integration testing will involve the iterative testing of new developer code as it becomes available and it is integrated into the product. System testing will begin after provider and client pieces has been implemented and released.
- Smoke tests will be performed and will consist of (1) ensuring that the newest packages provided by the build team install successfully and (2) ensuring basic functionality and sanity of the newest packages.
- Exit criteria has yet to be established. This should be discussed with project management.
- At this time, Orca and Accerciser are the standard tools to be used to test the accessibility of an application. The Orca test harness and Strongwind will be used to automate accessibility tests.
- Run sample with UISpy in Windows system as reference to ensure we implement the similar accessibility.
- Run Gtk sample with Accerciser in Linux system as reference to ensure we implemnet the similar accessibility.
- Log bug into bugzilla, add bug reference like "#BUGxxxx: xxxxxxxx" into automated test suite and comment out the test case if test case is failing. Delete bug reference and uncomment test case till bug is closed.
Item pass/fail criteria
The pass/fail criteria for each of the items described in Test Items section.
Criteria for Test Components:
| Test Items
Individual test case pass/fail criterion is defined by the automated script which performs the testing. Upon failure of a test case, the script should will log the failure. For exit criteria, see Approach.
Suspension criteria and resumption requirements
- Unavailability of external dependency during execution.
- When a defect is introduced that cannot allow any further testing (i.e., a blocker bug).
- Critical path deadline is missed so that the client will not accept delivery even if all testing is completed.
- A specific holiday shuts down both development and testing.
- Lack of testing resources (e.g., testers, hardware, etc).
- When the external dependent systems become available again.
- When a fix is successfully implemented and the testing team is notified to continue testing.
- The contract is renegotiated with the client to extend delivery.
- The holiday period ends.
- Testing resources become available
A failed build would not constitute suspension as we could generally continue to use the previous build. Major or critical defects would also not constitute suspension as other areas of the system could continue to be tested.
All documents, tools, and other components that are to be developed and maintained in support of the testing effort
- Test plan (this document)
- Test cases - test cases are been included into test scripts
- Custom tools - test harness, test scripts, sample applications
- Defect reports - none at this time
- Test summary reports - dashboard web page
- Other - none at this time
Testing preparation and setup
set of tasks necessary to prepare for and perform testing
|Build UIAutomation project on Bugzilla and Testopia.||X|
|Prepare virtual machines for most recent releases of supported platforms (openSUSE, Ubuntu, Fedora)||X|
|Setup test environment on VMs||X|
|Obtain testable build||X|
|Build DashBoard for Test Summary Report||P|
X = Done P = In Progress
- Enable assistive technologies on the GNOME desktop. This is done from "Assistive Technology Preferences" from the GNOME Control Center.
- Create Novell Bugzilla account
- Install Accerciser on OS
- Install most recent Mono
- Install most recent Orca
- Install most recent Strongwind
- Install Python >=2.5
- Install Iron Python (IPCE) >=1.1
- Setup Windows os(Vista) with UISpy on VM if necessary
Hardware, software, data, interfaces, facilities, publications, other requirements that pertain to the testing effort
Testing may be done on physical and virtual machines.
All tests must be performed on the most recent official release of the following platforms:
|Product||openSUSE 11.1||openSUSE 11.2||SLED 11||Fedora 12||Ubuntu 9.10 Karmic Koala||openSUSE 11.3||Fedora 13|
|Moonlight ATK Bridge||x||x||x|
|Winforms ATK Bridge||x||x||x||x||x|
|Client API project||x||x||x||x||x|
- No specific hardware requirements at this time
- Mono - most recent release
- Accerciser - most recent package for your platform
- Orca - most recent package for your platform
- Python >=2.5
- IronPython (IPCE) - most recent package for your platform
- Strongwind - most recent release
- All testers can work wherever they are needed, however, below is where our team focuses their efforts at this time:
Test Harness: Brian
DashBoard: Brian, Neville
Strongwind Tests: Brian, Calen, Ray
Orca Tests: Brian
Smoke Tests: Brian
Sample Applications: Calen, Ray
Sample Applications: Calen, Neville
Strongwind Tests:: Calen, Neville
UIAutomation Client API
White Tests: Ray, Felicia
Staffing and training needs
- QA Automation Engineer (4)
- Solid programming experience (C#, Python are a bonus)
- QA Engineer experience
- Bugzilla experience is a bonus
- Iterative testing experience is a bonus
|Oct 19||C# Introduction||Matt|
|Oct 23||C# Introduction||Matt|
|Oct 30||C# Introduction||Matt|
Everyone in the open source community is encouraged to join our QA team!
Built around the roadmap but specific to testing with testing milestones
Based on Q2 in roadmap, our initial testing schedule is below:
|Task||Start Time||End Time||Owner|
|Design test plan||Mar.||Apr.||Brian & Calen|
|Prepare test environmental stuff(OS, VM, App samples)||Mar.||Brian & Calen|
|Set up test environment||Apr.||May.||Brian & Calen|
|Design test case and test samples||Apr.||Jun.||Brian & Calen & Ray|
|Automate smoke tests||Sep. 10 2008||Sep. 12 2008||Brian|
|Design Strongwind test, Run testcase and submit defect (release 0.9)||July||Nov||Calen & Brian|
|Design Strongwind test, Run testcase and submit defect (release 1.0 BETA)||Dec||Jan||Calen & Brian & Ray|
|Design Strongwind test, Run testcase and submit defect (release 1.0 FINAL)||Jan||March 16 2009||Calen & Brian & Ray|
|Review Strongwind test, expand test cases||April||Jun||Brian & Author|
|Build DashBoard||May||Jun||Brian & Neville|
|Finish test report||Brian & Calen|
|Task||Start Time||End Time||Owner|
|Design test plan||Aug. 6||Aug. 7||Calen & Neville|
|create application samples||Aug. 10||Aug. 31||Calen & Neville|
|Write and Review Strongwind tests for each control(5 weeks)||Sep. 1||Oct. 16(10.1-10.11 are national holiday)||Calen & Neville|
|Run tests and log bugs(3 weeks)||Oct. 19||Nov. 6||Calen & Neville|
|test one or two Real Moonlight app(1 week)||Nov. 9||Nov. 13||Calen & Neville|
|test report(add moonlight test to dashbard)(1 week)||Nov. 9||Nov. 13||Calen & Neville|
|Task||Start Time||End Time||Owner|
|Design test plan||Aug. 6||Aug. 14||Calen & Ray|
|Study how to use White to write tests||Aug. 6||Aug. 14||Ray & Felicia|
|Study C# and UIA ClientAPI(4 weeks)||Aug. 17||Sep. 11||Ray & Felicia|
|Create UIA ClientAPI UserCases and Tests for WinForms real app(KeePass)||Sep. 14||Oct. 30(10.1-10.11 are national holiday)||Ray & Felicia|
|Run UIA ClientAPI tests and log bugs for Winforms(KeePass)||Nov. 2||Nov. 9||Ray & Felicia|
|Create UIA ClientAPI UserCases and Tests for Moonlight real apps||Neville & Calen|
|Run UIA ClientAPI tests and log bugs for Moonlight real apps||Neville & Calen|
|Create UIA ClientAPI UserCases and tests for GTK+ real app(Tomboy, Tasque?)||Felicia|
|Run UIA ClientAPI tests and log bugs for GTK+ real app(Tomboy, Tasque?)||Ray & Felicia|
|Finish test report||Calen & Neville & Ray & Felicia|
Risks and contingencies
Any activity that jeopardizes the testing schedule is a planning risk
- Program release schedule delay will jeopardizes testing schedule
- Program quality of release version couldn't satisfy with acceptance criteria
- Test environmental stuff unobtainable easily
- Delays in necessary QA training (e.g., understanding what needs to be tested, writing sample applications, test tools, and automation scripts)
- Tester staff change or lack of tester resources
- Changes to the original requirements or Designs
- Frequent program design changes
- Loss of support from Novell/Microsoft upper management
Persons who declare that the software is ready to move to the next stage