
3.1.3 Statistics of Software Conformance Testing
Charles Hagwood, Raghu Kacker, James Yen
Lynne Rosenthal, Leonard Gallagher, Paul Black
Conformance tests are written to determine if the implementation of a software package complies with agreed upon specifications. The goal is to provide some level of assurance that the requirements imposed by a specification are being met by implementations claiming conformance to that specification.
In this project we provide a confidence interval for the reliability of the NIST CGM (Computer Graphics Metafile) Conformance Test. The CGM International Standard ISO/IEC 8632, is a precise specification of a method of storing pictures on a computer system. The ATA (Air Traffic AssociationUnited, Delta, KLM, Northwest,etc), aerospace companies (Boeing, Aerospatiale), and vendors have agreed on a subset, i.e. a common standard for representing graphical illustrations of plane parts, wiring, etc. The ATA uses the CGM to exchange electronic graphics sent by the aircraft, engine, and equipment
manufacturers to the airlines. It is critical that the electronic graphics, exchanged in the CGM format, be processed in an identical manner by the airlines, whatever their origin. The NIST CGM conformance test suite determines whether a CGM interpreter product meets all the requirements specificed in the ATA profile of CGM. Testing implementations of the ATA profile is a means of achieving this goal and ensuring CGM software product reliability.
Ideally, one would like the test suite to include all the means of invoking the functional requirements of the specification at least once. In general, this is impossible due to the explosion of all the possible combinations of input values as well as the combination of requirements. Exhaustive testing is impossible. Therefore, it is important to sample the input space in such a way to obtain an accurate as possible estimate of the software's reliability. The method we choose is to partition the test point space into s disjoint homogeneous subspaces, E_{i}. Homogeneous in the sense that the input values in each subspace test similar aspects of the specification. We assume that test point space is finite, albeit containing a large number of elements, N. Let N_{i} denote the number of elements in E_{i}. Then the conformance test suite is made up of random samples of zero/one random variables,
,
from each of these subspaces. That is, n_{i} of the N_{i} tests points from E_{i} are selected and tested on the implementation of the software resulting in outcomes y_{i,j}. Define the reliability, 1p, as the probability that the specifications will be performed correctly. The reliability is calculated from the conformance test data.
Let p_{i} denote the probability that the implementation of the software fails to correctly perform at an arbitrary test point from E_{i}, then the reliability is given by
.
The ATA/CGM specification requires that no errors occur. Therefore in order for the implementation to conform none of the test results must fail, i.e. y_{i,j}=0. We find a nondegenerate confidence interval for 1p given these zero data.
The solution derived is based on solving the optimization problem for p_{i}
and letting the confidence interval be (1p_{0},1] where at the maximum. The solution is where is the divergence between the vectors and . The above formula provides a general confidence interval for any conformance test with fault intolerant specifications.
Date created: 7/20/2001 