Overview: Trusted, good quality software is most often taken for granted and not even noticed, but software with bugs causes software to be unreliable. Moreover, bad software often means hidden security vulnerabilities. Software quality relies on precise, testable software standards, followed by the development of conformance test suites and tools to help vendors build high-quality, interoperable implementations that meet the needs of the users. Often times, software standards are developed without rigorous planning with respect to precisely specifying requirements, defining conformance, etc. In addition, little thought is given to allotting sufficient resources to ensure that test suites, that check implementations for conformance, are developed. NIST is working with industry to address these needs by providing guidance on conformance topics, helping to develop testable specifications, and developing conformance tests and tools.

Industry Need: It has become well-known that software is not adequately tested and is rushed to the market full of bugs. A recent NIST study says that the annual cost to the country for inadequate testing is as high as 59 billion dollars. According to Business week, “More than 75% of the incidents are the direct result of bugs” and “more real testing - not ‘beta testing’ in the marketplace will yield programs less vulnerable to attack.” One of the main reasons for inadequate testing is the non-existence of specifications to test against as well as the poor quality of the specifications that are developed. Another reason is the expense of developing comprehensive test suites to test the software. Awareness of the importance and need for quality software is growing.

NIST/ITL Approcah: NIST/ITL works with industry to provide guidance on conformance as well as assist in the development of standards and conformance tests and tools. An online resource of conformance papers and presentations is available to provide information on conformance and conformance-related topics. We develop confomrance test suites, tools, and methodologies for various technologies. With the World Wide Web Consortia (W3C), we developed a set of quality assurance guidelines to foster the development of precise, better defined specifications as well as comprehensive, publicly available test suites, thus resulting in better quality software with fewer bugs. Additionally, we are collaborating with various healthcare organizations to define conformance and develop appropriate test materials.

Impact: NIST/ITL work with W3C has resulted in improved specifications, tests, and implementations. In particular, W3C specification developers have used the QA Specification Guidelines document to create and better specify their requirements. The NIST expertise and W3C Specification Guidelines have been applied to other efforts, including HL7 Messaging and EHR Functional Model, telemedicine guidelines, and guidelines for voting systems. Interoperability Specification.