M. Carroll Croarkin
James J. Filliben
Lisa M. Gill
William F. Guthrie
Eric S. Lagergren
Mark G. Vangel
Statistical Engineering Division, ITL
Janet E. Rogers
Bert W. Rust
Mathematical & Computational Sciences Division, ITL
Standard Reference Data Program, TS
With the widespread use and availability of statistical software, concerns about the numerical accuracy of such software are now greater than ever. Inevitably, numerical accuracy problems can exist with some of this software despite extensive testing. Indeed, this has been a continuing cause of concern for statisticians, see e.g. Francis, Heiberger, and Velleman (American Statistician, 1975) and Eddy and Cox (Chance, 1991). Many have cited the need for an easily-accessible repository of reference datasets. To date no such collection has been available. In response to concerns of both the statistical community and industrial users, the Statistical Engineering Division in collaboration with the Mathematical & Computational Sciences Division and Standard Reference Data Program has developed a Web-based service that provides reference datasets with certified values for a variety of statistical methods. This service is called Statistical Reference Datasets (StRD).
Currently 62 datasets with certified values are provided for assessing the accuracy of software for univariate statistics, analysis of variance, linear regression, and nonlinear regression. The collection includes both generated and ``real-world" data of varying levels of difficulty. Generated datasets are designed to challenge specific computations. These include the classic Wampler datasets for testing linear regression algorithms and the Simon & Lesage datasets for testing analysis of variance algorithms. Real-world data include challenging datasets such as the Longley data for linear regression, and more benign datasets such as the Daniel & Wood data for nonlinear regression.
Certified results for linear procedures were obtained using extended precision software to code simple algorithms for each type of computation. Carrying 500 digits through all of the computations allowed calculation of output unaffected by floating point representation errors. Certified values for nonlinear regression are the ``best-available" solutions, obtained using 64-bit precision and confirmed by at least two different algorithms and software packages using analytic derivatives.
In the coming year, the team will be publicizing the StRD web service. A special contributed paper session, ``Statistical Reference Datasets (StRD) for Assessing the Numerical Accuracy of Statistical Software," will be presented at the 1997 Joint Statistical Meetings in Anaheim, CA.
Date created: 7/20/2001