Conformance Testing and
Certification Model for Software Specifications
Lisa Carnahan, Lynne Rosenthal, Mark Skall
Software Diagnostic and Conformance Testing Division
The use of conformity assessment as a means by which buyers and sellers can communicate requirements and conformance will increase as information technology systems and applications grow more complex. Models for conformance testing and certification programs are necessary to understand principles and issues that are essential for successful conformity assessment programs. This paper presents one such model by identifying key roles, activities and products involved in any conformance testing and certification program. The authors have successfully used this model in helping private-sector organizations establish their certification programs.
As the pervasiveness of information technology
increases, so does the importance of ensuring the quality of products (i.e.,
software and systems). Conformity
assessment is defined in ISO/IEC Guide 2: 1996 as "any activity concerned
with determining directly or indirectly that relevant requirements are
fulfilled." In the marketplace, conformity assessment provides a vehicle
for exchanging information between buyer and seller. It increases a buyer’s (and/or user’s) confidence in a product
and its ability to meet their needs. It
provides an independent, objective method for evaluating products and not becoming
locked-into a single vendor. For
sellers (and developers), conformity assessment can help to substantiate claims
that a product meets the given specification.
Often conformity assessment is accomplished by
conformance testing. Conformance testing
is a means of measuring whether a product faithfully implements a
specification. The level and formality
of the testing are determined by the market – the requirements of the buyer
directly or an organization acting on behalf of a community of buyers, or by
regulation (e.g., safety, health, national security concerns). For example, some programs may require a
very formal testing and certification approach consisting of independent (i.e.,
third party), nationally accredited testing laboratories while others may be
more appropriate for self declaration and demonstration testing.
The Information Technology Laboratory (ITL), within
the National Institute of Standards and Technology (NIST), develops conformance tests for
forward-looking, publicly available standards and specifications. The tests are used by companies in the
private sector who build implementations which purport to conform to these
specifications. The intent of these
tests is twofold: 1) to be used by the implementors early on in development to
improve the quality of their implementation; and 2) to be used by industry
associations wishing to administer a testing and certification service. ITL focuses on the technical task of
developing test suites; leaving the regulatory aspects of testing and
certification to the private sector.
However, ITL is very active in helping industry associations set up
testing and certification programs to use the ITL-developed tests.
The sections below describe a generic model for
establishing a conformance testing and certification program. It describes the processes and procedures
for establishing, administering a testing program. While much has changed
regarding conformity assessment given the growth and changes in the software
industry, the conformance and certification model has not. Examples are used to describe how the model
is applied to support the changes in the software industry.
It is well recognized that conformance testing and
certification is a way to ensure that “standard-based” products are
implemented. The advantage afforded by
testing and certification are fairly obvious: quality products, competitive
markets with more choices, commodity pricing, and less opportunity to become
“locked in” to a particular vendor.
Moreover, a testing and certification program based on well understood
and sound principles will be acceptable and credible to its community of
users.
The conformance testing and certification model
described herein contains the fundamental roles,
activities, and products that are
necessary in administering and operating a testing and certification program
(see Figure 1). By adjusting and modifying the various activities, roles and
products, the model can be applied and used in establishing any testing and
certification program. Figure 2 highlights the interactions between the roles
and activities. The model allows for
roles, activities and/or products to be consolidated or further partitioned.
Roles |
Activities |
Products |
Buyer Seller |
Require Certification Test IUT |
Specification Implementation Under Test (IUT) |
Test Laboratory (TL) Certificate Issuer Control Board (CB) |
Recognize Test Method Recognize Tester Validate Results |
Test Method Test Report Certification Program Policy |
|
Answer Programmatic Queries |
Testing Laboratory Criteria |
|
Answer Test Method Queries Resolve Test Method Disputes Validate Conformance Issue Certificate |
Certificate of Conformance |
Figure 1: Roles, Activities and Products
While actual testing and certification can be carried
out by various organizations, it is essential that there be a centralized
sponsor or owner of the testing and certification program. The sponsor has a
fundamental interest in ensuring the success of the program.
Typically, the sponsor establishes and maintains the
conformance testing and certification program.
It assumes responsibility for insuring that the components of the
program are in place and becomes the centralized source for information about
the program. The sponsor may be
composed of one or more organizations.
Examples of sponsors are consortia, trade associations, standards
groups, or a government agency. More
often than not, the sponsor of the program is also the Certificate Issuer.
Figure 2: Interactions Among Roles and Activities
To execute the activities of the model, five roles
are defined. In the realization of
this model, some roles may be combined and performed by a single organization
or further distributed among several organizations.
·
Buyer requires conformance to the
Specification.
·
Seller builds the product with the
intent of meeting the conformance requirement of the purchaser. Products that
undergo testing are called Implementation Under Test (IUT)
·
Test Laboratory (TL) performs the operational
testing of the IUT .
·
Certificate Issuer (CI), issues a Certificate of
Conformance for IUTs that have successfully completed the testing process.
·
Control Board (CB), resolves dispute and answers
queries on behalf of the CI.
Buyer
The Buyer requires that a product be tested for
conformance. The buyer uses the results
of the testing to verify that a seller provides a product that conforms to the
specification and meet procurement requirements. In general, the buyer is the impetus for sellers to undergo
conformance testing. Specifically, if
buyers don’t demand that a product be tested and show evidence of that testing,
it is most likely that sellers will not undertake having their products tested.
The Seller or developer uses the conformance tests
and undergoes testing to demonstrate that the product adheres to the
specification and thus, meets established conformance requirements. Additionally, developers may use the tests
to debug their products prior to market.
Test Laboratory
The Test Laboratory (TL) conducts the conformance
testing using the prescribed test method. The testing is performed on the
seller/developer’s product. A TL can be
an organization or individual. A TL can
be accredited from a formal accreditation organization such as NIST’s National
Voluntary Laboratory Accreditation Program (NVLAP) or recognized by the buyer,
seller, and certificate issuer, as qualified to perform the testing.
The Certificate Issuer (CI) is responsible for
issuing certificates for conforming products. The decision to issue a
certificate is based on the testing results and established criteria for
issuing certificates.
The Control Board (CB) is an impartial body of
experts who function on behalf of the CI.
The CB is responsible for resolving queries and disputes related to the
testing process.
The activities comprising the model can be
categorized into one of four areas:
·
Recognition of competent testing
laboratories;
·
Testing with an approved test method;
and,
·
Testing process;
·
Resolution of Queries and Disputes.
Recognition of
Competent Testing Laboratory
A Testing Laboratory (TL) is an entity that provides
services to measure, examine, test, or otherwise assess conformance of an
implementation with its specification.
Within the buyer/seller model, a TL can be either a first-party, (the
seller performs the testing), second-party (the buyer performs the testing), or
third-party[1] (an independent organization
performs the testing) testing organization.
All three types of testing are used in the software industry. Often
there will be multiple TLs for a conformance testing and certification program.
The Certificate Issuer (CI) as well as Sellers and
other interested parties, must have confidence in the competency of the TL.
Competence is based on three concepts,
1.
the
ability to apply the test method correctly,
2.
the
ability to repeat a given test and generate the same results, and
3.
the
ability to operate the TL in a manner that maintains objectivity and neutrality
(obviously, first and second party testing organizations are not neutral).
The CI defines competence through requirements and
criteria. The CI can then apply the
criteria to a TL, determine its level of competency and, if appropriate, recognize
the TL as competent to perform testing.
This practical approach to identifying and recognizing qualified testing
organizations is appropriate when costs, time and efforts do not warrant
seeking accreditation from a formal accreditation organization.
If a more formal and rigorous approach is
appropriate, there exists many accreditation bodies exist that are capable of
performing this function. The National
Voluntary Laboratory Accreditation Program (NVLAP) is a NIST organization that
accredits testing organizations based on the requirements of ISO Guide 25[2] and additional subject-matter
requirements. NVLAP is responsible for
accrediting testing organizations to perform POSIX and Cryptographic Module
testing.
The purpose of the recognition criteria or
accreditation is to assure that TLs are capable and competent to meet the needs
of the testing and certification program.
The basic activities to make this determination include:
·
proficiency
testing – demonstration of a TL’s competency to successfully perform the
conformance testing using the test method,
·
on-site
assessment – visit by a technical expert to determine compliance with the
recognition criteria and ensure the TL is a legally identifiable organization
with staff and resource to discharge their duties.
· quality assurance – documentation and practices to ensure technical integrity of testing and analyses and adherence to quality practices appropriate to the testing and certification program.
Additional attributes required of a third-party TL
include that it:
·
ensure
that its personnel are free from any commercial, financial and other pressures
which might adversely affect the quality of their work,
·
ensure
that the protection of sellers’ confidential information and proprietary rights
are protected,
·
ensure
that sellers are served with impartiality and integrity,
·
maintain
a functional record keeping system for each seller testing process, and
·
have
the adequate facilities and equipment to fulfill the requirements of a TL.
Testing with an Approved Test Method
For a Certificate of Conformance to be meaningful,
all implementations must be tested in the same manner. Testing reflects the essence of technical
requirements of specifications and measures whether a product faithfully
implements the specification. A test method is a defined technical
procedure for performing a test. A test is the technical operation that
consists of the determination of one or more characteristics of a given
product, process or service according to a specified procedure. [ISO/IEC Guide
2] A test suite is the collection of
tests. Critical to the success of any
conformance testing and certification program is an appropriate and adequate
test method.
An adequate test method is one that provides test
results that give enough information for the CI to be satisfied that
conformance can be measured. An
adequate test method meets the requirement of rigor. An appropriate test method is one that, while adequate, does not
place undue requirements on the IUT and is cost justifiable. If the test method is too expensive to
employ then it will not be used. The
definition of adequate and appropriate is left to the CI to determine.
The Testing Process is described in a conformance testing and certification policy and procedures document. The document identifies the administrative as well as testing processes.
The testing process initiates with a seller (or anyone desiring to be tested) contracting with the TL to have an implementation tested for conformance. The seller and TL negotiate the scope of testing, the cost of testing, and the timeliness of testing. For a given seller, the TL must not be in a position to benefit nor suffer (beyond the testing fees) from the resulting pass or failure of the implementation under test (IUT).
Using the approved Test Method, The TL tests
the IUT for conformance and reports the results in a Test Report. The TL
forwards the Test Report and an indication of pass/fail to the CI. If the IUT successfully completes all the
tests and meets the criteria for issuing certificates, the CI issues a Certificate of Conformance to the
seller. Typically, the CI maintains a
list and makes available to the public, a register containing a listing of
products that have received certificates of conformance.
Resolution of
Queries and Disputes
Queries and disputes involving the test method,
procedures, test results, and program administration are directed to the
Control Board (CB). The purpose of the
CB is to resolve these issues and communicate the decision to all parties
involved. The CB acts on behalf of the CI. A query or dispute can be initiated
by a seller, TL or entity (e.g., developer) at any point in the testing
process. Queries and disputes should contain a statement of the problem,
rationale for dispute, and desired resolution.
All matters to be resolved by the CB should be determined by consensus
or as determined by documented CB policy and procedures.
Additional activities that may be under the auspices
of the CB include:
·
maintain
liaison with appropriate standards bodies and test laboratories,
·
participate
in the assessment of TL’s seeking recognition status,
·
recommend
changes to new versions of the test method or test laboratory recognition
criteria,
·
serve
as technical advisor to the CI and TLs
·
maintain
the test suite, and
·
control
changes to the conformance testing process.
The following products are used in the model:
·
Certification
Program Policy;
·
Testing
Laboratory Criteria;
·
Specification;
·
Implementation
Under Test (IUT);
·
Test
Method;
·
Test
Report; and
·
Certificate
of Conformance.
Certification
Program Policy
The Certification Program Policy (CPP) defines the
certification system. ISO/IEC Guide 2
defines a certification system as a “system having its own rules of procedure
and management for carrying out conformity certifications. The CPP addresses the following:
·
responsibilities
of the CI;
·
responsibilities
of the TLs;
·
responsibilities
of the seller (the IUT owner);
·
policy
and procedures for test laboratory recognition;
·
policy
and procedures for the testing process;
·
policy
and procedures for handling queries and disputes;
·
complete
definition of the certificate of conformance.
Testing
Laboratory Criteria
Testing Laboratory Criteria serves three purposes. The first purpose is to define the
competence and quality-related requirements that a testing laboratory must
possess to be designated as a recognized testing laboratory. The second purpose
is to describe the manner in which the laboratory will be assessed against the
requirements. The third purpose is to
show those who want to use the testing laboratory (e.g., sellers) , or those
who want to accept the conformance certificate as evidence of conformance
(e.g., buyers) the rigor under which the testing laboratory operates.
Specification
First and foremost to conformance testing and
certification is the specification.
This paper delineates ‘standards-based’ software specification from
other types of specification. This is
because not all specifications can be objectively tested for conformance. We recognize that not all ‘standards-based’
specifications can be objectively tested. However objective measurement (not
necessarily conformance testing per se) is usually a goal in these
specification development efforts.
If the specification can not be objectively tested,
then a alternate approach to conformance testing should be used to measure
whether a produce faithfully implements the specification. This is because an
accepted test method cannot be developed, thus repeatability and
reproducibility cannot be ensured.
Implementation
Under Test
The implementation under test (IUT) is the object
that is being tested for conformance.
For software specifications it is the software that has ‘implemented’
the specification. For any certification program, the scope of the IUT must be
defined and delineated from the rest of the supporting software and hardware of
the total system (referred to as the system under test). In many current certification programs the
hardware that is used by the software must also be defined. The software and supporting hardware
constitute the IUT and are listed in both the test report and certificate of
conformance.
Test Method
The test method must be adequate and appropriate
within the conformance testing and certification program in which it is used. Beyond these properties, test methods (and
thus the tests) should be objective, have adequate coverage, and correctly
implement the specification. In trying
to meet these requirements, those using and applying the test method should not
make the common mistake of allowing the test method to become the
‘specification’. This means that
sellers (builders of IUTs) will build the IUT to pass the conformance tests,
rather than building to the specification.
An objective test method allows for test results to
be reproducable by the same testing laboratory and to be repeatable by a
different laboratory. Initially some test methods do not quite
achieve a sufficient level of objectivity.
However objectivity should be something that is always strived for in
the development and ongoing refinement of a test method.
Test Report
A test report contains the results of the testing
effort, along with any additional information required by the CI. The test report should provide enough
information that, if necessary, the testing effort could be duplicated. The testing report should contain:
·
a
complete description of the IUT;
·
the
name of the testing laboratory;
·
the
signature of a testing laboratory official;
·
the
date that the testing occurred;
·
the
name and version number of the test method (and test suite);
·
the
results of the test method;
·
an
unambiguous statement indicating pass or fail.
Certificate of
Conformance
The certificate of conformance is typically a
summation of the test report. Since it is often used in the procurement
process, it includes information most pertinent between the buyer and the
seller.
The certificate includes statements made by the
CI. These statements articulate what
the CI is asserting as being conformant.
Typically these statements indicate that “this IUT was tested in this
environment, on this day, using this test method: the test results produced
were consistent with expected test results”.
The certificate also includes the signature of a CI official.
Cryptographic
Module Validation Program
The Cryptographic Module Validation[3] (CMV) Program was established
to provide independent testing to Security
Requirements to Cryptographic Modules, Federal Information Processing Standard,
140-1,(FIPS 140-1). When applied
appropriately, FIPS 140-1 can help provide strong protection for equipment that
provides security services such as encryption, authentication, and digital
signature generation and verification.
FIPS 140-1 was developed by a joint industry/government working group.
The requirement for certification is specified in the
FIPS 140-1 standard. Therefor, this
program is based on a regulatory requirement.
However there is currently an effort underway in the ANSI X9 area to
adopt FIPS 140-1. Thus the requirement
for certified products will be driven from the private sector.
The Security Technology Group at National Institute
of Standards and Technology (NIST) serves as the sponsor and Certificate Issuer
(CI) for the CMV Program. As such, the
certifications are considered to be second-party (i.e., NIST is acting on
behalf of user of FIPS 140-1, primarily federal agencies.) The CI requires that
testing laboratories be accredited by NVLAP under its Cryptographic Module
Testing Program. The CMV Program
requires that testing laboratories be independent from clients (i.e.,
third-party testing.)
The test method was developed NIST and was vetted by
the industry. The test method, policies
and all other program documents are publicly available.[4]
The validation certificate, signed by a CMV Program
official, contains the following:
·
the
name of the cryptographic module;
·
an
indication of pass/fail for each of the requirement areas specificed in the
standard;
·
the
accredited testing laboratory with its NVLAP identification code;
·
a
statement defining the scope of the validation;
·
the
date of the validation.
ATA Computer
Graphics Metafile (CGM) Conformance Testing Program
The Air Transport Association (ATA)[5] CGM Program was originally
established and operated by NIST to support the ATA 2100 Specification, Graphics Exchange (a.k.a. ATA CGM
profile). The testing program is a
critical component of the ATA’s program to represent maintenance manuals in
digital form and move to completely on-line maintenance manuals. Testing is done to ensure that the fidelity
and quality of the digital information is sufficient to satisfy the airline
companies’ safety and quality concerns.
The program is a means whereby a seller of a CGM implementation can
formally demonstrate conformance to the ATA CGM profile.
NIST is currently working with the ATA in its
assumption of the testing program. The
ATA CGM Conformance Testing Program will consist of recognized Testing Laboratories
to conduct the testing and a Control Board to handle disputes and serve as an
advisor to the ATA. The ATA will act as
the sponsor and administrator of the program.
The ATA or an ATA designate will issue certificates of conformance. The roles, activities, and products as
described in the generic model apply here with little modification. The Control Board takes on the additional
activity of assessing the testing laboratories according to pre-established
criteria. Additionally, the ATA
Technical Information Communication Committee’s Graphics Working Group[6] serves as a technical advisor
to both the ATA and the control board.
The test method consists of a NIST developed test
suite and test procedures. The test
method has been accepted and used by the community. It is publicly available along with other program documents[7].
The IEEE established a validation service for the
POSIX (Portable Operating System Interface).
The IEEE validation service[8] uses Accredited POSIX testing
laboratories, issue certification of validated test results, and maintains a
register of accredited laboratories and successfully tested products. The laboratories are accredited by the NVLAP
under its POSIX program.
The requirement for testing is buyer driven. Initially, federal agencies in their
requests for procurement (RFP) of POSIX systems required certificates of
validation prior to purchase. However,
the benefits of POSIX testing and its acceptance in the industry has resulted
in sellers requesting to be tested as a matter of course, rather than a
procurement requirement.
The test suite was produced in a joint effort between
NIST and several computer vendors. The
original testing policy and procedures produced by NIST have been adopted by
the IEEE.
This model describing the conformance testing and
certification process has been used many times over in certification programs
for standards-based software specifications. The examples above illustrate just
a few of these programs. It will
continue to be used as a communication mechanism between buyers and
sellers.
Test method developers must continue to develop test
methods that have adequate coverage with regard to the specification; are well
defined in terms measurement (i.e., what does each test case prove); and be
adequate and appropriate as defined by the Certificate Issuer.
As the industry moves toward component based
software, the challenge will be to develop test methods and associated
certification programs that can provide meaningful measurement in this
environment.
Breitenberg, Maureen, The ABC’s of the U.S. Conformity Assessment
System, NISTIR 6014, April 1997.
Breitenberg, Maureen, The U.S. Certification System from a
Government Perspective, NISTIR 6077, October, 1997.
Carnahan, Lisa, Developing Federal Standards and
Accreditations for Data Protection Products, Proceeding of SPIE Conference, October, 1995.
Dashiell, William H., L. Arnold
Johnson and Lynne S. Rosenthal, Overview
of Model for United States Geological Survey Recognition of Spatial Data
Transfer Standard Certification System, NIST IR 6124, May 1998.
Horlick Jeffrey, and Lisa
Carnahan, Cryptographic Module Testing,
Handbook 150-17, April, 1995.
ISO/IEC Guide 2: 1996, Standardization and Related Activities:
General Vocabulary
ISO/IEC Guide 25: 1990, General Requirements for the Competence of
Calibration and Testing Laboratories.
NIST, Derived Test Requirements for FIPS 140-1, Security Requirements for
Cryptographic Modules, March, 1995.
NIST, Procedures and Requirements, NIST Handbook 150, March 1994.
NIST, Security Requirements for Cryptographic Modules, FIPS 140-1,
January, 1994.
[1] The use of the term 'independent'
is ambiguous and thus applied differently in the testing community. Some programs may determine independence
based on corporate structures; while others may apply financial-interest
related measures. It is incumbent for
the Certificate Issuer to define the term 'independent' for its community of
interest.
[2] The requirements of ISO Guide
25, General Requirements for the Competence of Calibration and Testing
Laboratories, are based on the ISO 9000 standards. Accreditation based on ISO Guide 25 provides the basis for many
of the international recognition agreements for recognizing test results.
[3] The CMV Program uses the term
‘validation’ rather than ‘certification’.
Beyond legal reasons for not using ‘certification’; the program
developers chose not to use this term because of its many uses within the
cryptographic and computer security communities (e.g., ‘certification authority
for digital signatures’ and ‘certification and accreditation of systems’).
[4] CMV Program information can be found at http://csrc.nist.gov/cryptval/.
[5] Information about the ATA can
be found at http://www.air-transport.org/.
[6] The Graphics Working Group is
the organization responsible for creating and maintaining the ATA 2100
Specification, Graphics Exchange specification.
[7] ATA CGM Program information
can be found at http://www.itl.nist.gov/div897/ctg/graphics/cgm.htm.
[8] IEEE POSIX Validation Program
information can be found at http://standards.ieee.org/regauth/posix/index.html.