
3.1.5 Statistics of Software Conformance Testing
Charles Hagwood Statistical Engineering Division, CAML
Lynne Rosenthal Software Diagnostics and Conformance Testing Division, ITL
The reliability of a software product is the probability that it will function properly. By functioning properly we mean, the production of answers that are not deviant from that required. In discussing the statistical measurement of software reliability, it is convenient to think of two types of situations (i) those where the reliability measurement involves the occurrence of failure over time, i.e. the time until the next failure, and (ii) the static model, where the reliability depends only on the successful performance of software given an intensive input interrogation.
In the time dependent reliability model, the raw data available to the tester is a sequence
of execution times between successive failures. As failures occur, attempts are made to correct the underlying faults in the software. Because the errors are corrected when detected and the corrections do not lead to further errors, it is reasonable to assume that the number of failures occurring in the time interval (0,t], N_{t} is a nonhomogeneous Poisson process. Thus the observed times can be regarded as realizations of random variables
,
the interarrival times of N_{t}. The problem is to compute the conditional distribution of T_{k} given that
,
and is given by
is just the probability that a failure does not occur in (t,t+x) given the last failure occurred at time t. Of course, is calculated from the distributional properties of N_{t} and the observed failure times. Note that here the error detection times will, more often than not, depend on the class of users.
In the static model or the timeindependent model the software system is subjected to an intensive test suite in order to determine if it meets specification. Conformance testing captures the technical description of a specification and measures whether an implementation (i.e., software product or system) faithfully implements the specification. The goal is to provide some level of assurance that the requirements imposed by a specification are being met by implementations claiming conformance to that specification. Conformance testing can be applied to the broad spectrum of software domains and
software specifications. Regardless of the domain or specification,
conformance testing is blackbox or functional testing. Specifically, the
internal structure and behavior of the implementation is not considered in
the testing process. The conformance suite of tests are derived solely
from the specification.
Conformance test suites are designed by carefully choosing different input values, trying to design test cases that will invoke every functional requirement in the specification, at least once. One method of accomplishing this is to partition the input space into disjoint subdomains, E_{i}, where the E_{i} represent different homogeneous parts of the program that test similar aspects of the program. The statistics involved in this collaboration is to estimated 1p, the probability that the software will not fail, based on this stratified sampling method. In most cases in conformance testing, the software is said to pass if and only if there are no failures. Given this scenario, the estimate of reliability based on classical methods produces a reliability of one, although a nontrivial confidence interval can be found. A Bayesian approach is used to derive a nontrivial estimate of the reliability. As examples we take several software implementations of the Computer Graphics Metafile (CGM), International Standard ISO/IEC 8632.
Date created: 7/20/2001 