Skip to content

APG Regression Testing Framework Project Description

Jump to bottom
Matt King edited this page Jun 28, 2018 · 1 revision

Project Overview

The center-piece of ARIA Authoring Practices 1.1 is a set of design patterns for web-based GUI components. Each pattern includes an “Examples” section that includes one or more canonical reference implementations of the pattern. The primary purpose of the examples is to provide assistive technology, web, and browser engineers with stable, functional references for testing their support and use of ARIA. It is thus essential that modifications to the example implementations do not unintentionally break any aspect of the ARIA implementation.

The objective of this project is to integrate a test framework and test cases into the APG build process that prevents regressions in behaviors, accessibility properties, or accessibility features prescribed by the documentation that accompanies each APG example.

Deliverables

  1. Test Framework: Open source test framework integrated into the aria-practice GitHub repository and Travis CI configuration that runs the below described test suites.

  2. Test suites: Files associated with each example that define the set of assertions that encompass the below defined test coverage.

  3. Documentation supporting framework implementation and utilization.

  4. Validation Data: Data that validates the efficacy, veracity, and completeness of the framework configuration and test suite code.

Development Process

  1. Work will be managed by a project manager, APG Task Force chair (Matt King), and reviewed by the W3C Authoring Practices Task Force.

  2. Implementation and development will be carried out in the w3c/aria-practices GitHub repository.

  3. The project will be divided into phases documented by GitHub milestones.

  4. The milestones, work breakdown, and review points for each phase will be defined by the contractor in collaboration with the project manager after a brief orientation and exploration phase.

  5. Test suites for a subset of the examples, approximately six, will be coded by various members of the Authoring Practices Task Force and reviewed by the contractor to ensure task force members adequately understand how to utilize the framework.

Design Goals

  1. Contributor Friendly: For people who develop or modify code for the APG, the framework is:

    a. Platform agnostic, e.g., is equally easy to use on Windows, Mac, or Linux.

    b. Easy initial setup, e.g., NPM package or installs by executing a script.

    c. Generates easily digestible output.

  2. Integration friendly:

    a. The framework is integrated into Travis CI.

    b. Performance is sufficient to run for every build. Or, if a specific class of tests require long run time, they are executed with a separate script that is run manually while the remainder is run for every build.

  3. Sustainability:

    a. Developing test suites for new examples relies on skills and knowledge common among front-end web engineers.

    b. The architecture utilizes packages that have a proven history of reliability and stability.

  4. Extensibility: While adding and modifying test suites and assertions should be as simple as possible, any other form of extensibility or reusability is much less important that minimizing complexity. The goal is to focus on developing the most straightforward and maintainable framework for achieving the singular objective of preventing errors in the APG.

Test Coverage

Types of Tests

  1. Role, state, property, named and description: Each APG example page includes one or more tables that describes the implementation of ARIA roles, states, and properties for the examples or examples on that page. The test suite for an example will include assertions that validate claims made within those descriptions. This may include test after page load as well as after exercising elements of the component that change states or properties.

  2. Keyboard focus: Each APG example page includes a table or tables documenting the keyboard interface for the example or examples on that page. The test suite for an example will include assertions that validate the behaviors described in those tables.

Validation of Framework Implementation and Test Suites

The project includes developing and executing a methodology for generating data that validates:

  1. Efficacy: Does it capture a wide variety of injected faults and make the source of the fault easy to locate?

  2. Veracity: Is the output accurate?

  3. Completeness: Do the tests cover all documented requirements for each example tested?

The validation data will be included in the repository and surfaced in the documentation wiki.

Documentation

The project includes providing documentation in a section of the aria-practices wiki that describes:

  1. How APG contributors install and use the framework

  2. How APG contributors code test suites for new or changed examples

  3. Framework selection decision process

  4. Framework Implementation and configuration details

  5. Framework maintenance

  6. Framework validation process

Rough Sizing

Sizing by Project Phase

Activity and estimated Hours:

  1. Explore: 4
  2. Experiment: 8
  3. Evaluate Options: 4
  4. Design/Configure: 8
  5. Coding: 300
  6. Documentation: 6
  7. Validation: 8
  8. Task Force Reviews and issue responses: 12
  9. Total: 340

Test Suite Development Breakdown

  1. Examples: 59
  2. Approximate keyboard command requirements to test: 407
  3. Approximate Role, State, and property requirements to test: 488

Calculation of coding hours assumes:

  1. 1000 requirements to test across 59 examples.
  2. Average of 5 hours per example.
  3. Total hours for test suite coding: 295
Clone this wiki locally