image of SDCT banner with links Go to ITL website Go to SDCT Home Page Go to NIST Home Page Go to SDCT Home Page Go to SDCT Products Page Go to SDCT Contact Page Go to SDCT About Page Go to SDCT Projects Page

Software Testing by Statistical Methods



Investigation and development of new methods for software testing based on stochastic processes and statistical measures in order to improve the quality of software and to provide qualitative measures for determining the probability that software correctly adheres to its specification.

History and Goals

During 1997, ITL staff from the Statistical Engineering Division (898) and the Software and Systems Division (897) proposed a collaborative effort to research the potential and practicality for applying statistical methods to software testing. This effort evolved into a new ITL Competency proposal, entitled Software Testing by Statistical Methods, that was approved by the NIST Director for initial funding in fiscal year 1998.

The Software Testing by Statistical Methods project includes both white-box testing and black-box testing to determine if a program conforms to its functional specifications. White-box testing assumes that program source code is available for inspection and metering, whereas black-box testing allows access to a program only through its defined interface. Functional specifications may be specified using either conventional techniques or by formal methods. Conventional techniques usually specify formal syntax with semantic rules written in natural language, whereas formal methods require that both syntax and semantics be specified using mathematically rigorous techniques. In any case, the overall goal of this project is to integrate statistical techniques into software testing to ensure software quality and to provide quantitative measures of stability, reliability, and conformance to specifications.

Initial priority for this project is on statistical methods potentially applicable to software defined using conventional techniques. A companion project in the Software Quality Group places more emphasis on formal methods. Conventional specifications are important because a large majority of National, International, and Internet standards are specified via non-formal methods, and a major mission of ITL is to provide conformance testing techniques for such software standards.


Name and Email    NIST Directory
Len Gallagher          directory link
Charles Hagwood     directory link
Jeff Offutt                 directory link
Lynne Rosenthal      directory link

Current Projects

    Component Integration
    Testing methods for object-oriented software, software components, and component interactions in an integrated software system.
    Contact: Len Gallagher

Collaboration with Industry

    Testing - Working with a group of Computer Graphics vendors and users to define usage models for estimating software reliability.

    Others - We would enjoy working with your company or industry group on any aspect of applying statistical methods to software testing, possibly under a cooperative research and development agreement (CRDA). For information contact any project participant.

Technical Reports and Papers

    Project Proposal - Software Testing by Statistical Methods, July 10, 1997.

    Initial Tasks - Software Testing by Statistical Methods, August 22, 1997. (HTML file)

    NISTIR 6129. David Banks, William Dashiell, Leonard Gallagher, Charles Hagwood, Raghu Kacker, Lynne Rosenthal; Software Testing by Statistical Methods - Preliminary Success Estimates for Approaches Based on Binomial Models, Coverage Designs, Mutation Testing and Usage Models, March 12, 1998.
    (PDF file) (Postscript file)

    James Yen, David Banks, P. Black, L.J. Gallagher, C.R. Hagwood, R.N. Kacker, L.S.Rosenthal; Software Testing: Protocol Comparison, March 28, 1998, Eleventh International Software Quality Week (QW' 98) conference, San Francisco.
    (PDF file)

    Charles Hagwood, Lynne Rosenthal, Raghu Kacker, Leonard Gallagher, James Yen; Reliability of Conformance Tests, August 20, 1998. Proceedings of 22nd Annual International Computer Software and Applications (COMSAC '98) conference, Vienna, Austria.

    Len Gallagher; Conformance Testing of Object-Oriented Components Specified by State/Transition Classes, April 6, 1999, Draft technical report.
    (PDF file) (Postscript file)


Privacy Policy/Security Notice
Disclaimer NIST is an agency of the U.S. Commerce Department's Technology Administration.

Created on September 21, 2001
Last Modified:
Tuesday, 03/30/2004 12:32 PM
SSD Division Webmaster: