StRD: Frequently Asked Questions
NIST StRD

Frequently Asked Questions


Q1: What exactly is NIST providing with the Statistical Reference Datasets?
A1: NIST is providing a collection of datasets with associated certified values of model-dependent parameters.

Q2: What areas of statistical computing are covered by the Statistical Reference Datasets?
A2: At the current time, datasets and associated certified values are provided for four areas of statistical interest:

Q3: What led NIST to offer this service?
A3: The Statistical Engineering Division of NIST has been approached by users of statistical software packages and by scientists within its own organization about the accuracy of statistical software. This service is provided to aid these communities in evaluating the accuracy of commercial and home-grown software and the community of software developers in evaluating their algorithms.

Q4: I have a machine which is capable of carrying a large number of significant digits. Doesn't that guarantee the accuracy of results?
A4: No, for two reasons.

  1. There is always a possibility of incorrect implementation of the algorithm in the computer code; and
  2. Some algorithms, especially for nonlinear regression, can be sensitive to particular classes of problems or data.

Q5: Can NIST recommend statistical software packages which give consistently accurate results?
A5: NIST does not endorse or recommend any commercial products.

Q6: How do I know if my software package is giving accurate results?
A6: One criterion for judging the accuracy of software output is comparison with "certified values" from sources that are known to be reliable. The NIST collection of datasets serves this purpose. The certified values for the datasets in this collection should be compared with the values estimated from your software package. The NIST datasets are ordered according to level of difficulty, and for completeness, comparisons should be made over all levels of difficulty. Further details can be found in the background information sections for each statistical area.

Q7: My software gives results which disagree with the NIST certified results in the __ significant digit. Should this difference be worrisome?
A7: The answer to this question depends upon your needs and the application. NIST results are quoted to a high number of significant digits for comparison purposes, not to indicate statistical significance. Results from a given package will depend on the statistical area being tested, the individual problem and the precision used in your computations. Guidance for comparing your results with the certified values is given in the background information sections for each statistical area.

Q8: What should I do if my software gives results which are very different from the NIST results?
A8: Start by checking to see if your computer program is attempting the same calculation as that used for the certified results. It may be necessary to invoke some specific option, which is not the default, in order to reproduce the certified results. Also, you should verify that your computer program obtains correct results for the lowest difficulty datasets before worrying about the results for datasets of moderate or high difficulty. If you become convinced that the computer algorithm is at fault, you may need to contact the software developer for a resolution of the discrepancy.

Q9: How accurate are the NIST results?
A9: The NIST results for the linear regression, analysis of variance, and univariate pages are accurate to the fifteen quoted significant digits. Results for the non-linear regression pages are quoted to eleven significant digits for comparison purposes only. A description of how certified results are obtained is given as background information on the web page for each statistical area.

Q10: I am interested in the accuracy of statistics other than those certified in the NIST pages. Can you help me?
A10: NIST is not prepared at this time to evaluate statistics other than those given in these pages. However, you are invited to express your interest in further developments by sending email to StRD@nist.gov.

Q11: Will NIST in the future offer a service for testing statistical software packages?
A11: NIST has no plans to offer such a service. However, a specialized service for testing the performance of geometric fitting software for coordinate measurement systems is available through the NIST Calibration Program; this special test service is called the Algorithm Testing and Evaluation Program for Coordinate Measurement Systems.

Q12: Can we advertise that our commercially available software package is traceable to NIST certified results?
A12: There is no mechanism for establishing software traceability. The NIST Statistical Reference Datasets are provided simply as an aid to software development, and use of these materials for software testing is strictly voluntary.