Comparing Automated Generation of Conformance Tests With Traditional Development

This project was performed by the Conformance Testing Group of the Software and Systems Division, part of the Information Technology Laboratory (ITL) , at the National Institute of Standards and Technology (NIST), under the direction of the Department of Commerce (DoC). The goal of the project was to investigate the effectiveness of using automated software test generation methods to develop conformance tests for specifications of software standards.. Traditionally, conformance tests are developed by manually coding the test source code, and then testing that source code.
This project

  1. Used a particular automated method, the Assertion Definition Language (ADL), developed by Sun Microsystems, Inc., to generate conformance tests for a particular application specification (a subset of the POSIX C language interface).
  2. Used traditional methods to develop conformance tests for that same specification
  3. Compared the results.

Our basic observation was that the use of ADL did not reduce the time needed to develop conformance tests, and that a reasonable question exists regarding the assumption that automated tools provide a more effective means of developing conformance tests for program interfaces than do traditional approaches. We have published the results of the study as NISTIR 6114.  For a hard copy, or for more information, please email Alan Goldfine, or go to

http://www.nist.gov/itl/div897/staff/goldfine/goldfine.htm.

 

Disclaimer:  Any links on this page to web sites outside NIST are provided because they have information that may be of interest.  NIST does not necessarily endorse the views expressed or the facts presented on these sites, nor endorse any commercial products that may be advertised or available on these sites.  Also, please see http://www.nist.gov/public_affairs/disclaim.htm.

Last updated: January 11, 2000.