What is this thing called
Conformance?
Keyword: conformance, conformance testing, DOM test suite, validation, XML test suite, XSL test suite
Introduction
What
does it mean when software conforms to a standard or specification? How do you know that the software conforms?
And just as important, why is this important?
According
to a study by the Gartner Group[1],
the private sector spends over $2.2 trillion on information technology (IT)
annually. Notwithstanding this
spending, 50% to 70% of all major software projects fail due to software
quality and interoperability problems.
Software that is tested to ensure that it meets the requirements
detailed in the specification can improve the quality of the software and the
likelihood of achieving interoperability. This type of testing is called
conformance testing. Determining
whether a product faithfully implements a standard or specification is
essential to creating robust, interoperable solutions. This bulletin presents an overview of
conformance and addresses conformance to the World Wide Web Consortium (W3C)
Recommendations for the Extensible Markup Language (XML) family of
specifications.
Unscrambling the Terminology
ISO/IEC
Guide 2[2]
defines conformance (also
referred to as conformity) as the fulfillment of a product, process, or service
of specified requirements. These
requirements are specified in a standard or specification as either part of a
conformance clause or in the body of the specification. A conformance clause is a section of
a specification that states all the requirements or criteria that must be
satisfied to claim conformance to the specification. Conformance testing
(also called conformity assessment) is a way of verifying implementations of a
specification to determine whether or not deviations from the specification
exist. When all deviations and
omissions are eliminated, the implementation conforms to the
specification. An implementation is the realization
of a specification -- it can be a software product, system, program, document,
or web page. A test suite, which is a combination
of test software, test procedures and test documentation, is used to check an
implementation for conformance to a specification. The test software, consisting of a set of test files
(i.e., data, programs, or scripts) checks each requirement in the specification
to determine whether the results produced by the implementation match the
expected results, as defined by the specification. The test procedures define the
administrative as well as the technical process for testing an
implementation. The test
documentation describes how the testing is to be done and the directions
for the tester to follow. Additionally,
the documentation should be detailed enough so that testing of a given
implementation can be repeated with no change in test results.
Other
conformance related terms include validation and certification. Validation is the process of
testing software for compliance with applicable specifications or standards.
Although
the terms validation and conformance testing are often interchanged, they are
not the same. The validation process consists of the steps necessary to perform
the conformance testing by using an official test suite in a prescribed
manner. Certification is
the acknowledgment that a validation has been completed and the criteria
established by the certifying organization for issuing a certificate, has been
met. Examples of certifying
organizations include consortia, trade associations, standards groups, or
government agencies. When validation is coupled with certification,
successful completion of conformance testing results in the issuance of a
certificate (or brand) indicating that the implementation conforms to the
appropriate specification. Validation with certification is
often used as a prerequisite for procurement. It is important to note that
certification cannot exist without validation, but validation can exist without
certification. Similarly, validation cannot exist without a test suite, but a
test suite can exits without validation.
Degree of Formality
The level and formality of the conformance testing is determined by the buyer, an organization acting on behalf of a community of buyers, or by regulation (e.g., safety, health, national security concerns). Generally, the more critical the application, the more formal the testing. For example, some testing programs may require a very formal testing and certification approach using an independent (i.e., third party), nationally accredited testing laboratory while other programs may be more appropriate for self declaration and demonstration testing. Typically, the decision to establish a testing program is based on weighing the benefits achieved by obtaining conformance versus the costs of creating and running a validation and certification program.
More
recently with the advent of the Internet and rapidly changing information
technology, the notion of conformance testing has become less formal allowing
developers and users to assess for themselves the validity or correctness of
the software or data with respect to the relevant specification. They can then
make any necessary corrections to allow the implementation to conform. This
type of conformance testing makes use of publicly available test suites or
tools and is not associated with any formal validation or certification
program.
Benefits of conformance
testing
Conformance
test suites increases the probability that software products that claim to
conform to a specification are implemented correctly. Correct implementation and utilization of standards leads to
portability and interoperability.
Portability is the ability to move software programs or applications
among different computer systems with minimal change. Interoperability is defined as the capability of two or more
systems to exchange and use information.
Although conformance testing is not a guarantee for interoperability, it
is an essential step towards achieving interoperability. Conformance testing
provides software developers and users assurance and confidence that the
conforming product behaves as expected, performs functions in a known manner,
or possesses a prescribed interface or format.
In
order to test that a specification is faithfully adhered to, one must carefully
examine the specification and compare the implementation's result against the
syntax and semantics contained in the specification. Often times, errors or ambiguities in the specification are
discovered. These errors and
ambiguities can then be fed back to the standards committee who developed the
specification, for correction. For
software developers, using conformance test suites early-on in the development
process can help improve the quality of their implementations by identifying
areas in which they conform as well as those areas in which they do not. In the marketplace, conformance test suites
and consequently conformance testing provides a vehicle for exchanging
information between buyer and seller.
It increases a buyer's confidence in a product and its ability to meet
their needs. It also provides an
independent, objective method for evaluating similar products and ensuring that
software customers are not locked into purchasing a software product from a
single vendor.
Testing Methodologies
Depending
on the type of specification (e.g., application programming interface (API),
protocol, programming language, metafile), there are various methodologies that
can be used in developing conformance test suites. However, these methodologies all follow the same basic strategy -
falsification testing; and the same basic process - analyze the specification
for test assertions or test requirements, write a test purpose for each test,
and create and document a test case which realizes each test purpose.
Ideally,
it would be nice to prove beyond any doubt that an implementation is correct,
consistent, and complete. However, this
is generally impossible for implementations of nontrivial specifications. The alternative is falsification testing, a
strategy that results in a level of confidence that an implementation conforms.
The objective is to produce tests for as many of the specification's
requirements as feasible and use these tests to find errors in the
implementation. Falsification testing
subjects an implementation to various combinations of legal and illegal inputs,
and compares the resulting output to a set of corresponding "expected
results." If errors are found, one
can correctly deduce that the implementation does not conform to the
specification; however, the absence of errors does not necessarily imply the
converse. Falsification testing can
only demonstrate non-conformance.
Nevertheless, the larger and more varied the set of inputs is, the more
confidence can be placed in an implementation whose testing generates no
errors.
The
process for developing conformance test suites begins with defining the set of
test assertions (TA) upon which the test suite is based. Each TA should be as simple as possible and
narrowly focused on an atomic (simple) functionality, must not contradict
another TA, and must have some testable consequence. Using this set of TAs, a set of tests that will exercise each of
the requirements is developed. Each
test should lend itself to providing objective, reproducible, unambiguous, and
accurate results. Each test should be
traceable back to a statement or statements in the specification. Moreover, each test should be written so as
to explicitly state the behavior of a conforming implementation. Thus, the
tests when executed on the implementation under test will generate a pass/fail
condition. Failure to pass a test
implies failure to conform.
Conformance Test Suites for
XML Technologies
There are many organizations, including private sector companies, standards group, consortia, and government agencies that develop conformance tests and/or tools. Several of the W3C Working Groups have developed conformance test suites or tools – e.g., SVG, CSS, and HTML. The National Institute of Standards and Technology (NIST) and the Organization for the Advancement of Structured Information Standards (OASIS) are addressing XML-related interoperability issues by jointly developing test suites that are intended to complement the W3C Recommendations. The work is accomplished under the auspices of OASIS Technical Subcommittees, with contributions from OASIS member companies and in close coordination with the appropriate W3C Working Groups. Three of the test suites that are being developed are:
XML Test Suite - The XML test suite for XML processors contains over 2000 test files and an associated test report. Accompanying the actual test files is an XML test description file that contains pertinent information for each of the tests included in the distribution. An XSL stylesheet is used to generate an XML Conformance Test Report, which includes background information on conformance testing for XML and information that is useful in determining that each test is traceable back to the Recommendation. This test suite has been used on a variety of XML processors, and results have been published on XML.com. XML Test Suite is available from the NIST web site, http://xw2k.nist.gov/xml/xml-test-suite.html.
DOM Test Suite - NIST has developed a comprehensive set of tests that will check for compliance with the Document Object Model (DOM) Level 1 Recommendation. Two sets of test suites were built, one for DOM with an ECMAScript binding and the other with a Java™ binding. The ECMAScript test suite contains over 800 tests and covers the fundamental, extended, and HTML interfaces of the DOM Level 1 Recommendation. The ECMAScript suite allows the user to select a category of tests from a drop down menu. The tests are then executed and the results displayed. The Java™ test suite contains over 200 tests and covers the fundamental and extended interfaces of the DOM Level 1 Recommendation. They can also be selected from a drop down menu, executed and results displayed. Alternatively, the Java™ test suite can be downloaded and run locally.
The
test suites are available at http://xw2k.nist.gov/xml/dom-test-suite.html.
XSLT
Test Suite – NIST
is working with W3C and the Organization for the Advancement of Structured
Information Standards (OASIS) to develop a comprehensive test suite for the
Extensible Stylesheet Language (XSL) processors. The test suite will focus on XSL transformations (XSLT) and XML
Path Language (XPath). The XSLT test
suite will allow users to customize the test suite to filter out optional
category tests. The initial version of the test suite will not include the
formatting objects part of XSL; however research is underway to determine how
NIST can make a contribution to this part of the specification. The final test suite is expected to be
complete by March 2001.
NIST – http://www.nist.gov/xml/
OASIS – http://www.oasis-open.org/committees
W3C – http://www.w3c.org
Disclaimer
Any mention of commercial products or reference to commercial organizations is for information only; it does not imply recommendation or endorsement by NIST nor does it imply that the products mentioned are necessarily the best available for the purpose.
™Java and all Java based marks are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States and other countries.