SED navigation bar go to SED home page go to SED publications page go to NIST home page SED Home Page SED Contacts SED Projects SED Products and Publications Search SED Pages

contents     previous     next

3.1.5 Statistics of Software Conformance Testing

Charles Hagwood

Statistical Engineering Division, CAML

Lynne Rosenthal

Software Diagnostics and Conformance Testing Division, ITL

The reliability of a software product is the probability that it will function properly. By functioning properly we mean, the production of answers that are not deviant from that required. In discussing the statistical measurement of software reliability, it is convenient to think of two types of situations (i) those where the reliability measurement involves the occurrence of failure over time, i.e. the time until the next failure, and (ii) the static model, where the reliability depends only on the successful performance of software given an intensive input interrogation.

In the time dependent reliability model, the raw data available to the tester is a sequence $t_1,t_2,\ldots,t_n$ of execution times between successive failures. As failures occur, attempts are made to correct the underlying faults in the software. Because the errors are corrected when detected and the corrections do not lead to further errors, it is reasonable to assume that the number of failures occurring in the time interval (0,t], Nt is a nonhomogeneous Poisson process. Thus the observed times can be regarded as realizations of random variables $T_1,T_2,\ldots,T_n$, the interarrival times of Nt. The problem is to compute the conditional distribution of Tk given that $S_{k-1}=\sum^{k-1}_{i=1}T_i=t$, and is given by

\begin{displaymath}R(x\mid t) =P(T_k > x\mid S_{k-1}=t). \end{displaymath}

$R(x\mid t)$ is just the probability that a failure does not occur in (t,t+x) given the last failure occurred at time t. Of course, $R(x\mid t)$ is calculated from the distributional properties of Nt and the observed failure times. Note that here the error detection times $t_1,t_2,\ldots,t_n$ will, more often than not, depend on the class of users.

In the static model or the time-independent model the software system is subjected to an intensive test suite in order to determine if it meets specification. Conformance testing captures the technical description of a specification and measures whether an implementation (i.e., software product or system) faithfully implements the specification. The goal is to provide some level of assurance that the requirements imposed by a specification are being met by implementations claiming conformance to that specification. Conformance testing can be applied to the broad spectrum of software domains and software specifications. Regardless of the domain or specification, conformance testing is black-box or functional testing. Specifically, the internal structure and behavior of the implementation is not considered in the testing process. The conformance suite of tests are derived solely from the specification.

Conformance test suites are designed by carefully choosing different input values, trying to design test cases that will invoke every functional requirement in the specification, at least once. One method of accomplishing this is to partition the input space into disjoint subdomains, Ei, where the Ei represent different homogeneous parts of the program that test similar aspects of the program. The statistics involved in this collaboration is to estimated 1-p, the probability that the software will not fail, based on this stratified sampling method. In most cases in conformance testing, the software is said to pass if and only if there are no failures. Given this scenario, the estimate of reliability based on classical methods produces a reliability of one, although a nontrivial confidence interval can be found. A Bayesian approach is used to derive a nontrivial estimate of the reliability. As examples we take several software implementations of the Computer Graphics Metafile (CGM), International Standard ISO/IEC 8632.


contents     previous     next

Date created: 7/20/2001
Last updated: 7/20/2001
Please email comments on this WWW page to sedwww@nist.gov.