Acceptance testing

Acceptance testing of an aircraft catapult
Six of the primary mirrors of the James Webb Space Telescope being prepared for acceptance testing

In engineering and its various subdisciplines, acceptance testing is a test conducted to determine if the requirements of a specification or contract are met. It may involve chemical tests, physical tests, or performance tests.

In systems engineering it may involve black-box testing performed on a system (for example: a piece of software, lots of manufactured mechanical parts, or batches of chemical products) prior to its delivery.[1]

Software developers often distinguish acceptance testing by the system provider from acceptance testing by the customer (the user or client) prior to accepting transfer of ownership. In the case of software, acceptance testing performed by the customer is known as user acceptance testing (UAT), end-user testing, site (acceptance) testing, or field (acceptance) testing.

A smoke test is used as an acceptance test prior to introducing a build to the main testing process.

Overview

Testing generally involves running a suite of tests on the completed system. Each individual test, known as a case, exercises a particular operating condition of the user's environment or feature of the system, and will result in a pass or fail outcome. There is generally no degree of success or failure. The test environment is usually designed to be identical, or as close as possible, to the anticipated user's environment, including extremes of such. These test cases must each be accompanied by test case input data and/or a formal description of the operational activities to be performed. The intentions are to thoroughly elucidate the specific test case and description of the expected results.

Acceptance tests/criteria (in agile software development) are usually created by business customers and expressed in a business domain language. These are high-level tests to verify the completeness of a user story or stories 'played' during any sprint/iteration. These tests are created ideally through collaboration between business customers, business analysts, testers, and developers. It's essential that these tests include both business logic tests as well as UI validation elements (if need be). The business customers (product owners) are the primary project stakeholder of these tests. As the user stories pass their acceptance criteria, the business owners can be reassured the developers are progressing in the right direction.

Acceptance test cards are ideally created during sprint planning or iteration planning meeting, before development begins so that the developers have a clear idea of what to develop. Sometimes acceptance tests may span multiple stories (that are not implemented in the same sprint) and there are different ways to test them out during actual sprints. One popular technique is to mock external interfaces or data to mimic other stories which might not be played out during an iteration (as those stories may have been relatively lower business priority). A user story is not considered complete until the acceptance tests have passed.

Process

The acceptance test suite is run against the supplied input data or using an acceptance test script to direct the testers. Then the results obtained are compared with the expected results. If there is a correct match for every case, the test suite is said to pass. If not, the system may either be rejected or accepted on conditions previously agreed between the sponsor and the manufacturer.

The objective is to provide confidence that the delivered system meets the business requirements of both sponsors and users. The acceptance phase may also act as the final quality gateway, where any quality defects not previously detected may be uncovered.

A principal purpose of acceptance testing is that, once completed successfully, and provided certain additional (contractually agreed) acceptance criteria are met, the sponsors will then sign off on the system as satisfying the contract (previously agreed between sponsor and manufacturer), and deliver final payment.

User acceptance testing

User acceptance testing (UAT) consists of a process of verifying that a solution works for the user.[2] It is not system testing (ensuring software does not crash and meets documented requirements), but rather ensures that the solution will work for the user i.e. test the user accepts the solution (software vendors often refer to this as "Beta testing").

This testing should be undertaken by a subject-matter expert (SME), preferably the owner or client of the solution under test, and provide a summary of the findings for confirmation to proceed after trial or review. In software development, UAT as one of the final stages of a project often occurs before a client or customer accepts the new system. Users of the system perform tests in line with what would occur in real-life scenarios.[3]

It is important that the materials given to the tester be similar to the materials that the end user will have. Provide testers with real-life scenarios such as the three most common tasks or the three most difficult tasks you expect an average user will undertake. Instructions on how to complete the tasks must not be provided.

The UAT acts as a final verification of the required business functionality and proper functioning of the system, emulating real-world usage conditions on behalf of the paying client or a specific large customer. If the software works as required and without issues during normal use, one can reasonably extrapolate the same level of stability in production.[4]

User tests, usually performed by clients or by end-users, do not normally focus on identifying simple problems such as spelling errors or cosmetic problems, nor on showstopper defects, such as software crashes; testers and developers previously identify and fix these issues during earlier unit testing, integration testing, and system testing phases.

UAT should be executed against test scenarios. Test scenarios usually differ from System or Functional test cases in the sense that they represent a "player" or "user" journey. The broad nature of the test scenario ensures that the focus is on the journey and not on technical or system-specific key presses, staying away from "click-by-click" test steps to allow for a variance in users' steps through systems. Test scenarios can be broken down into logical "days", which are usually where the actor (player/customer/operator) system (backoffice, front end) changes.

In the industrial sector, a common UAT is a factory acceptance test (FAT). This test takes place before installation of the concerned equipment. Most of the time testers not only check if the equipment meets the pre-set specification, but also if the equipment is fully functional. A FAT usually includes a check of completeness, a verification against contractual requirements, a proof of functionality (either by simulation or a conventional function test) and a final inspection.[5][6]

The results of these tests give confidence to the client(s) as to how the system will perform in production. There may also be legal or contractual requirements for acceptance of the system.

Acceptance testing in extreme programming

Acceptance testing is a term used in agile software development methodologies, particularly extreme programming, referring to the functional testing of a user story by the software development team during the implementation phase.[7]

The customer specifies scenarios to test when a user story has been correctly implemented. A story can have one or many acceptance tests, whatever it takes to ensure the functionality works. Acceptance tests are black-box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release. A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created for each iteration or the development team will report zero progress.[8]

Types of acceptance testing

Typical types of acceptance testing include the following

User acceptance testing
This may include factory acceptance testing, i.e. the testing done by factory users before the product or system is moved to its destination site, after which site acceptance testing may be performed by the users at the site.
Operational acceptance testing
Also known as operational readiness testing, this refers to the checking done to a system to ensure that processes and procedures are in place to allow the system to be used and maintained. This may include checks done to back-up facilities, procedures for disaster recovery, training for end users, maintenance procedures, and security procedures.
Contract and regulation acceptance testing
In contract acceptance testing, a system is tested against acceptance criteria as documented in a contract, before the system is accepted. In regulation acceptance testing, a system is tested to ensure it meets governmental, legal and safety standards.
Alpha and beta testing
Alpha testing takes place at developers' sites, and involves testing of the operational system by internal staff, before it is released to external customers. Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback, before the system is released to other customers. The latter is often called “field testing”.

List of acceptance-testing frameworks

See also

References

  1. ↑ Black, Rex (August 2009). Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing. Hoboken, NJ: Wiley. ISBN 0-470-40415-9.
  2. ↑ Cimperman, Rob (2006). UAT Defined: A Guide to Practical User Acceptance Testing. Pearson Education. pp. Chapter 2. ISBN 9780132702621.
  3. ↑ Goethem, Brian Hambling, Pauline van (2013). User acceptance testing : a step-by-step guide. BCS Learning & Development Limited. ISBN 9781780171678.
  4. ↑ Pusuluri, Nageshwar Rao (2006). Software Testing Concepts And Tools. Dreamtech Press. p. 62. ISBN 9788177227123.
  5. ↑ "Factory Acceptance Test (FAT)". Tuv.com. Retrieved September 18, 2012.
  6. ↑ "Factory Acceptance Test". Inspection-for-industry.com. Retrieved September 18, 2012.
  7. ↑ "Introduction to Acceptance/Customer Tests as Requirements Artifacts". agilemodeling.com. Agile Modeling. Retrieved 9 December 2013.
  8. ↑ Don Wells. "Acceptance Tests". Extremeprogramming.org. Retrieved September 20, 2011.

Further reading

External links