WPC ڠ`O?zui7+=$ j .D07pYh`4U㔸YmdY9B8qwDZ}F )pYMsВ;k=H*6[0Y[TCzc'v6 BZ҄J=e)OMa!oq`u< /s獒.*®$})-],o2boPBxd,ʼn_i6;5©6qQE6GsN Ysε@IdՖ;z؁GŲC|ځΛ_J=RHĊ&(~[K2`p>;N٪ҸB~658Ltz@_+NPyQ_9 u&'pB -|1R.L  % 0 ` N B *8 w@b 0 f 0 U<JY 0 0L>a 0| 0UNUN*UHx 0| R<4 0 0 0DI B  0AUN 0D9 0}& 0N< 0| h D3G 0K^ 0T 0h 0|e BUxN R7dUN*b* C* + h++4++ 0|+ AM5,j,UN. h:/ h@/@/ AMY/f/a/U:/r///h0h0~000U:000# 0b1 R11O2O2 AQ33333333f3f3f3f3f3f3f3fn3n3 0@p3a3 0J33334 0|4 0DP5P5P5P55f5a55 D/5 m555556 0 e666666637 : : A:g; D+< = "=@8-=-=-=-=-=-=-=-=-=-=o4e=e=e=e=e=pEQ $R#*X, XX*  ݛ ( Figure  1   ݀Logicalrelationshipamongmetrologyconcepts  foruseinstandardizationinmeasurements..%(8$*X, XX* c:\wpwin60\template\report\report1.wpt&0 d d {{[DD`>$Level 1 Heading %p%%.% H' % %p   %p%%.% H' % %p(K>:$ %p%%.% g% %p  q %p%%.% g% %p'@FA2Humanst521 Cn BT Bold(s39$ !*X, XX*   .P%.%>b$Large Circle0 TABLE A(T3R$ !*X, XX*   F>>G,TitleReport TitleXXXX\XXXXXX\XXxhH  Z6Times New Roman Regular(hH  Z 6Times New Roman Regular+hH  0Times New Roman Bold(T3R$ !*X, XX*   1[ReportTitle]%/  WS"  3|xDK>$Level 3 Heading %p%%.% X X% %p  q %p%%.% X X% %p   (T(33*$ !*X, XX*      0  (#$  0   d(Vm$0   (hH  Z 6Times New Roman Regular($$  0  (T3 $ !*X, XX*     0  . e    >$"Small Circle"0  (T3R$ !*X, XX*   'dxd  R**X, XX*   ݛ.XY. X%.%Preparedfor  MEL/ITLManagementbyMEL/ITLTaskGrouponMetrologyforInformationTechnology(IT)#.%2% X.XY.T#( $ Figure  1  (,!$0  0` (#(#  (@$0  0` (#(#0 ` (#` (# ` (T$0  0` (#(#0 ` (#` (#0 (# (# Level 1Level 2Level 3Level 4Level 5P Pd(ON  Z 6Times New Roman Regular5d (#pP  - mR"*X, XX*  ݛ.&+&%.%*wABddd Xdd Xdd X((w,jdd , dd , dd , dd +  &  & #.%2%+&.&F#. 1 %.%2MeasuringPhysicalQuantities#.%2%1 . #  dd [length,mass,time,electriccurrent,thermodynamictemperature,luminousintensity,pH,hardness,RichterScale,...] 1'"" 1 .$ . Units 7-" 7 Standards 7-" 7 AppliedUses/Practices .$" . Definitionand rr Specification %HH %ISO1000[meter,...] %rr  %ISO261,ISO262,ISO724,ISO965[metricscrewthreads],ISO7,ISO228[pipethreads] %  %NFPA70[nationalelectricalcode] %rr  % Methodsof  Testing %^^ %primaryreference[atomicclock,cesiumlaser],standardreferencematerial,standardreferencedata,calibration %  %primaryreference[standardreferencethread,scratchstandard,gage],calibration,conformancetesting %4 4  %inspection,calibration,referencematerial,referencedata,conformancetesting,interoperabilitytesting %4 4  % Commercial t t  Products %J J  %measurementinstrument[laserinterferometer,tapemeasure] %J J  %buildingcomponents[pipe,nut,bolt,screw] %J J  %structure[building,bridge] 2(t t  2 . 1 %.%2MeasuringDigitalInformationTechnologySystemsQuantities#.%2%1 . m#   ! [time,information,mathematicaloperations,...] 1'""" 1 .$XX# . Units 7-XX$" 7 Standards 7-XX%" 7 AppliedUses/Practices .$XX&" . Definitionand ' Specification %( %ISO2382[bit,byte,word,error,fault,...],ISO1000[second,...] %* %ISO646[ASCII],ISO2382[floatingpointrep],ISO/IEC9899[C] %, %ISO10303[STEP],IETFRFC1610[TCP/IP],ISO99451[POSIX] %. % Methodsof / Testing %0 %calibration,conformancetesting %1 %conformancetesting,interoperabilitytesting,referencedata,referenceimplementation %4 %inspection,conformancetesting,interoperabilitytesting,referencedata,referenceimplementation %7 % Commercial 8 Products %9 %performanceanalyzer,logictester %: %Ccompiler,printer,monitor,microprocessor %< %operatingsystem,networkingsoftware,router,computerassistedmanufacturingdevice 2(? 22(&..@"  2(hH  Z 6Times New Roman Regular  d)*2A    v 9R&*X, XX*   ݛThispageintentionallyleftblank7(T3R$ !*X, XX*   ,$(&A<< CReport - Traditional /, '  0 .   (hH  Z 6Times New Roman RegulardssS$hvA<< BhTABLE A,  AZ"Arial Regular h ] \R3' Letter\         ,cAZ"Arial RegularHP LaserJet 4 PlusHPPCL5MS,,,,,,0nLPage H$VdTEMP1 'Level 1Level 2Level 3Level 4Level 5lm$nvC<<C 2c0Quick 220 TABLE A(P$ Table  1  Level 1Level 2Level 3Level 4Level 5(T3R$ !*X, XX*   ($$   1  TABLE B   --' dxdU1, 2, 3,Level 1Level 2Level 3Level 4Level 54#I^2Quick 1.|  .0 |  3*$*X, XX*    *    ݛ.&+&%.%SIunitsofmeasureareveryusefulandwellestablishedformeasuringmany  physicalquantities.11   C @  ؀However,somephysicalquantitiesaremoreusefully  measuredinnonSIunits,suchasahardnessscale   C ,12pH   C ,13andRichterscale   C .14In \ fact,theSIspecificall#.%2%+&.&|#ystatesthatitdoesnottreatconventionalscales,resultsof 2 conventionaltests,currencies,norinformationcontent.HereconventionaltestsmeanssuchmeasurementsasofpHwhicharecarriedoutunderaconventiondifferentfromSI.  ,$( I&mage <=8C HKKKK ] $R#*X, XX*  ݛ.XY. X%.%  ,  X          4  `    #.%2% X.XY.G#.-%.%2Figure2#.%2%.-# 'dxd  C:\WINDOWS\BLACKT~1.BMPWPC1hyB95139'"!UI cqd9 Z6Times New Roman RegularcXXX        @attribute/quantity'.""C x x G pj 5cC G'"#X+cqd9 Z6Times New Roman RegularcXXX        @ကunit'."$u{{J< s fJ'"% + cqd9 Z6Times New Roman RegularcXXX        @primaryreference'."&u{{J< s fJ'"'/+cqd9 Z6Times New Roman RegularcXXX        @secondaryreferences'."(u{{J< s fJ'")B}+ cqd9 Z6Times New Roman RegularcXXX        @measuredvalues"*| V\cqd9 Z6Times New Roman RegularcXXX        @Methodsofrealization"+T cqd9 Z6Times New Roman RegularcXXX        @Methodsofcalibrationandtesting", t(cqd9 Z6Times New Roman RegularcXXX        @Methodsofmeasurement"/) Icqd9 Z6Times New Roman Regularc        @"0cqd9 Z6Times New Roman RegularcXXX        @Realization"3K Icqd9 Z6Times New Roman Regularc        @"4 Icqd9 Z6Times New Roman Regularc        @"5Ccqd9 Z6Times New Roman RegularcXXX        @Dissemination"6pcqd9 Z6Times New Roman RegularcXXX        @Measurement"7 Icqd9 Z6Times New Roman Regularc        @"8Jcqd9 Z6Times New Roman RegularcXXX        @DefinitionWPWin 6.0/OLE 1.0 Prefix Information MarkerWPDraw30.Drawing4ࡱ>  Root Entry`.@p OOle 7WPG204CompObjn'F! WP 7 [0] `.@pPresentations 7 Drawing Embed SourceWPDraw30.Drawing9qHPoWPC1hyB95139'"!UI cqd9 Z6Times New Roman RegularcXXX        @attribute/quantity'.""C x x G pj 5cC G'"#X+cqd9 Z6Times New Roman RegularcXXX        @ကunit'."$u{{J< s fJ'"% + cqd9 Z6Times New Roman RegularcXXX        @primaryreference'."&u{{J< s fJ'"'/+cqd9 Z6Times New Roman RegularcXXX        @secondaryreferences'."(u{{J< s fJ'")B}+ cqd9 Z6Times New Roman RegularcXXX        @measuredvalues"*| V\cqd9 Z6Times New Roman RegularcXXX        @Methodsofrealization"+T cqd9 Z6Times New Roman RegularcXXX        @Methodsofcalibrationandtesting", t(cqd9 Z6Times New Roman RegularcXXX        @Methodsofmeasurement"/) Icqd9 Z6Times New Roman Regularc        @"0cqd9 Z6Times New Roman RegularcXXX        @Realization"3K Icqd9 Z6Times New Roman Regularc        @"4 Icqd9 Z6Times New Roman Regularc        @"5Ccqd9 Z6Times New Roman RegularcXXX        @Dissemination"6pcqd9 Z6Times New Roman RegularcXXX        @Measurement"7 Icqd9 Z6Times New Roman Regularc        @"8Jcqd9 Z6Times New Roman RegularcXXX        @DefinitionBOlePartOlePres000"6 C  6" """DDDUUUwww@3f3333f333ff3fffff3f3f̙3f3333f3333333333f3333333f3f33ff3f3f3f3333f3333333f3̙333333f333ff3ffffff3f33f3ff3f3f3ffff3fffffffffff3fffffff3fff̙ffff3fffff3f3333f333ff3fffff3f3f̙3f3f3333f333ff3fffff̙̙3̙f̙̙̙3f̙3f3f3333f333ff3fffff3f3f̙3f45  QTimes New Roman0-- .  "Systemn--"2 &attribute/quantity&-- "-$55b^{_b5--- .  --2 unit---$--- .  --!2 primary reference ---$@=Z>@--- .  --%2 zsecondary referencesz ---$--- .  --2 measured values -- .  --2 Methods of---2 realization(-- .  --2 ' Methods of',--2 8calibration and8C--2 ItestingI-- .  --2  Methods of,--2  measurement  9-Times New Roman-- .  --WP MathExtendedA- 2 s9-- .  --2 Realization T--- .  --- 2 ps9p--- .  --- 2 s9-- .  --2 C Dissemination  Ch-- .  --2  Measurement c--- .  --- 2 s9-- .  --2 a Definition aM- METAFILEPICT"n"6 C  6" """DDDUUUwww@3f3333f333ff3fffff3f3f̙3f3333f3333333333f3333333f3f33ff3f3f3f3333f3333333f3̙333333f333ff3ffffff3f33f3ff3f3f3ffff3fffffffffff3fffffff3fff̙ffff3fffff3f3333f333ff3fffff3f3f̙3f3f3333f333ff3fffff̙̙3̙f̙̙̙3f̙3f3f3333f333ff3fffff3f3f̙3f45  QTimes New Roman0-- .  "Systemn--"2 &attribute/quantity&-- "-$55b^{_b5--- .  --2 unit---$--- .  --!2 primary reference ---$@=Z>@--- .  --%2 zsecondary referencesz ---$--- .  --2 measured values -- .  --2 Methods of---2 realization(-- .  --2 ' Methods of',--2 8calibration and8C--2 ItestingI-- .  --2  Methods of,--2  measurement  9-Times New Roman-- .  --WP MathExtendedA- 2 s9-- .  --2 Realization T--- .  --- 2 ps9p--- .  --- 2 s9-- .  --2 C Dissemination  Ch-- .  --2  Measurement c--- .  --- 2 s9-- .  --2 a Definition aM- !*X, XX*   ݛ    <  h          D  p          L  L .sW%.% NISTIR6025 #.%2%.sWC# \  .e%.%2#.%2%.e# *q ddd Xdd Xdd X (# (#q,p +  -  h   G%2%%.%2m %%2  G  %  Metrologyfor |   InformationTechnology(IT)%oGQ݌&0  - &Ќ.m  m.%2% .m  % / .XY. X%.%2April28,1997May1997%/V j '#.%2% X.XY.c#   '8 p      . %.%2 '  .TableofContents#.   '##.%2% .#  4  Vm  VmG0     .XY. X%.%2Preface#""J(#.(#(##1VmGc݌  Ќ  Vm  Vm0     Introduction#""J(#.(#(##2Vm9݌ r  Ќ  )!  )!0  0(#(#    Scope#""J(#.(#(##2)!݌ ^  Ќ  )!  )!0  0(#(#    Definitions#""J(#.` ` (#(##2)!݌ J  Ќ    Vm  Vm{ 0     EstablishingaConceptualBasisforITMetrology#""J(#.dd;(#(##4Vm{ ݌ " r  Ќ  )!  )!\ 0  0(#(#    PrinciplesofPhysicalMetrology#""J(#.,(#(##4)!\ w ݌ ^  Ќ  )!  )!? 0  0(#(#    PrinciplesofITMetrology#`"`"I(#.44&(#(##10)!? Z ݌ J  Ќ  MethodsofTestingforDigitalITSystemsQuantities`"`"I(#.\\>15 "  )!  )! 0  0(#(#    )! ݌ (#(# Ќ  StatusandOpportunitiesforITMetrology`"`"I(#.418   RolesforNISTinITMetrology`"`"I(#.(21  Conclusions`"`"I(#.21 Z Vm  Vm0     AnnexA:References#`"`"I(#.f f (#(##22Vm/݌ 2 Ќ  AnnexB:GlossaryofAbbreviations`"`"I(#.,25  Z AnnexC:ExamplesofPresentITMetrologyatNIST`"`"I(#.``;27 2 #.%2% X.XY.# ;      g %,%%.%2 a% %,    P  Preface TԌ  Ќ. a.%2%.  72%%d%%d7.XY. X%.%2InMay1996,NISTmanagementrequestedawhitepaperonmetrologyfor > informationtechnology(IT).AtaskgroupwasformedtodevelopthiswhitepaperwithrepresentativesfromtheManufacturingEngineeringLaboratory(MEL),theInformationTechnologyLaboratory(ITL),andTechnologyServices(TS).ThetaskgroupmembershadawidespectrumofexperiencesandperspectivesontestingandmeasuringphysicalandITquantities.ThetaskgroupbelievedthatitscollectiveexperienceandknowledgewereprobablysufficienttoinvestigatetheunderlyingquestionofthenatureofITmetrology.#.%2% X.XY.p#.XY. X%.%2ԀDuringthecourseofitswork,thetask    groupdidnotfindanypreviousworkaddressingtheoverallsubjectofmetrologyforIT.Thetaskgroupfoundittobebothexcitingandchallengingtopossiblybefirstinwhatshouldbeacontinuingareaofstudy.Aftersomespiriteddeliberations,thetaskgroupwasabletoreachconsensusonitswhitepaper.Also,asaresultofitsdeliberations,thetaskgroupdecidedthatthiswhitepapershouldsuggestpossibleanswersratherthanassertdefinitiveconclusions.Inthisspirit,thewhitepapersuggests:ascopeandaconceptualbasisforITmetrology;ataxonomyforITmethodsoftesting;statusofITtestingandmeasurement;opportunitiestoadvanceITmetrology;overallrolesforNIST;andrecapitulatestheimportanceofITmetrologytotheU.S.Thetaskgroupisveryappreciativeofhavinghadtheopportunitytoproducethiswhitepaper.Thetaskgrouphopesthatthiswhitepaperwillprovidefoodforthoughtforourintendedaudience:NISTmanagementandtechnicalstaffandourcolleagueselsewherewhoareinvolvedinvariousaspectsoftestingandmeasuringIT.#.%2% X.XY.#.XY. X%.%2TaskGroupMembers: "  LisaCarnahan(ITL)GaryCarver(MEL)MarthaGray(ITL)MikeHogan(ITL),ConvenerTheodoreHopp(MEL)JeffreyHorlick(TS)GordonLyon(ITL) ElenaMessina(MEL)#.%2% X.XY.# *B&) %%%.%2 %,%%% a% %, a a      Introduction Ԍ   . a...%2%..%2%%.%2  % Q  :%%%.%2 %,%%% ' % %, '   '     :  Scope : iԌ d Ќ.   '.  ..%2% ..%2%%.%2  ݀ R .XY. X%.%2Thescopeofthiswhitepaperisthetestingormeasuringofdigital ( x informationtechnology(IT)systemsattributesorproperties;theuseofdigitalITsystemsintestingandmeasuring;andtheunderlyingmathematical,computational,andstatisticalsciencesusedintestingandmeasuring.ThispapersuggestsaconceptualbasisforITmetrology;reviewsITtestingmethods,thestatusofITmetrology,andopportunitiesforadvancingITmetrology;andnotespossiblerolesforNIST.#.%2% X.XY.#<%%%.%2<XX%<%OnegoalofthiswhitepaperistoapplytheconceptsofmetrologytoITsystems.AnothergoalistorelatemeasurementsinITtoestablishedconceptsoftraceability.#.XY. XX<X⦼##.%2% X.XY.# %,%%.%2  8 % %,  l  Definitions/ %,%X% %p  8!/%%% %,%%%%%%%% %,%%% XX% %,    {!   PX X#.- P #t.-InformationTechnology(IT)#.-t"# P.-# XX P\"# {!>" &"Ԍ f Ќ XXX X XXX X XXX X XXX X %,%X X  #.%2%% %,"#.XY. X%.%2InformationTechnology(IT)isarelativelyrecentlycoinedtermfor > referringtoseveralindustrysectorswhoseboundariesareincreasinglyfuzzy:computing,telecommunications,andentertainment.Ageneric,functionaldefinitionofITisthestorage,processing,transfer,display,management,organization,andretrievalofinformation.ITcanbecharacterizedasincreasinglydigital.ITsystemsaretypicallyablendofhardwareandsoftware.Thehardwarecanbecharacterizedasincreasinglycomplexanddifficulttomanufacture.Thesoftwarecanbecharacterizedasincreasinglycomplexanddifficulttodevelopwhileeasytoreplicate.ExamplesofITsystemsare:computers,computernetworks,telephones,telephonenetworks,televisions,andcablenetworks.ITsystemsareubiquitous,impactingallbusinesses(manufacturing,healthcare,education,etc.)whichmeansincreasinglycomplexdigitalITsystemsareeverywhereandneedtobetestedforavarietyofreasons.#.%2% X.XY.I$#  %&!" .XY. X%.%2TheNISTLaboratoryMissionistopromotetheU.S.economyandpublic  welfarethroughtechnicalleadershipandparticipationinthedevelopmentofthenationsmeasurementandstandardsinfrastructure.Fromthisperspective,theNISTInformationTechnologyLaboratory(ITL)hasdefinedITas:0 Informationtechnologyisthebodyofmethodsandtoolsbywhichcommunicationsandcomputingtechnologiesareappliedtoacquireandtransformdata,andtopresentanddisseminateinformationtoincreasetheeffectivenessofthemodernenterprise.  (# (# X)X X.XY. XXXX) XXX X     +   PX Xt PMetrology# Pt+## XX P+# ++ v+Ԍ $  Ќ.XY. XX X.XY. X X.XY..XY. X X.XY.  Thedefinitionoftheterm metrologyintheInternationalVocabularyof   BasicandGeneralTermsinMetrology(theVIM)1is:    .f X.XY.L; .fmetrology #.f L;-##.XY. X.f-#   LX"X X.XY.L; "XLXscienceofmeasurement#LX"X L;͋.##.XY. X"XLXl.# 4 TheVIMfurthernotesthatmetrologyincludesallaspectsboththeoreticalandpracticalwithreferencetomeasurements,whatevertheiruncertainty,andinwhateverfieldsofscienceortechnologytheyoccur.Metrologyforphysicalandchemicalpropertieshasadvancedoverthelast200years,keepingpacewithtechnologyandindustrialadvancements.MetrologyforITsystemsisinitsinfancy.MeasurementofITsystemsoftwareconsistsofascertainingortestingforlogical/mathematicalstatesorfunctionalityinanITsystem.ITsystemhardwareisrelativelyeasytomeasure(exceptthatcomplexityofVLSIcausesitstestingtoremainincomplete,justlikesoftware),becauseitreliesuponmatureandsophisticatedphysicalandchemicalmeasurementscience.+.XY. X% %p X.XY.+#.XY. X X.XY."2#  #"  XX X.XY. aX X    D2 u  O[6Oa![2  EstablishingaConceptualBasisforITMetrology2g3 2Ԍ  Ќ. a.XY. X.   : XX X.XY. ' X X     :>4  PrinciplesofPhysicalMetrology  :>44 4Ԍ *z Ќ.   '.XY. X .  +<XX%%.% X.XY.+InordertoexplainITmetrology,itisnecessarytoexaminethelogical h basisofmetrology.Manyoftheclassicalconceptsofmetrologyhavetheirrootsinphysics,buttheyhavebeensuccessfullyappliedtootherareasofscienceandtechnology./.XY. X%%.%X<Xp5/ 8     (# (#+<XX%%.% X.XY.+:&K;7z| `h@Ef` 8 8'!Amodelofthelogicalrelationshipbetweenstandards,measurement,andquantitiesisshowninFigure1.Thisfigureshowsthelogicalchainbetweenaconceptualizedpropertyandthemeasuredvalueofthatproperty,withinasystemofstandardsandtraceability.ThefollowingexamineseachofthecomponentsofFigure1.Theterm standard,whileperhapsunavoidable,mustbeusedcarefully.InEnglish,ithastworelevantmeanings:asaspecification(whatiscalled normein,T')   French)andasthereferencerealizationoftheunitofaquantity(whatiscalled taloninFrench)./.XY. X%%.%X<X7/+<XX%%.% X.XY.+TheVIMdefinitionforthelattertermis:   #.XY. XX<Xt:#CXS$X X.XY.C&$XCXS(measurement)standard   talon  H #.f&C;#L; .f8 #.f L;;#L; .fmaterialmeasure,measuringinstrument,referencematerialor   measuringsystemintendedtodefine,realize,conserveorreproduceaunitoroneormorevaluesofaquantitytoserveasareference#LX"X L;0;##.XY. X"XLX<#+<XX%%.% X.XY.+ԀJ  ! ! Thetwomeaningsareverydifferent.Forinstance,theASCIIcodeisastandardinthefirstsense,butnotinthesecond.Unfortunately,thereisatendencytousethetermwithoutregardtothesenseinwhichitwillbeunderstood.#.XY. XX<XS=#<XX X.XY. J  ItisimportanttounderstandthatFigure1isadiagramoflogicalrelationships,notofchronologicaldevelopment.Historically,many(ifnotmost)quantitiesbeganasqualitativecomparisons(forexample, warmerand colder),followedbytheinventionofaformallydefinedquantity(e.g., temperature),andfinallywiththedevelopmentofunits,scales,andasystemofstandards.ITismuchmoreintheearlierpartofthisevolutionaryprocessthanaremorematurefieldssuchasphysicsorchemistry.#.XY. XX<X⦭>#<XX X.XY.#.XY. XX<XA#<XX X.XY. Quantities  J FromthetopofFigure1,theVIMdefinitionoftheterm quantityis: #.XY. XX<X_A#CXS$X X.XY.C&$XCXSquantity   Z 8 attributeofaphenomenon,bodyorsubstancethatmaybedistinguishedqualitativelyanddeterminedquantitatively#CXS$X&CUB##.XY. X$XCXS6B#<XX X.XY.b  ! ! Thisappearsclear.However,itisnecessarytoexaminetheoperativeelementsofthisdefinitioninordertoapplyittoIT.Thefirstrequirementisthatitisnecessarytodealwithanattribute(ofanITsystem).Inotherwords,theremustbeaspecific,distinctpropertytomeasure.2󀀀Itiscritical $: $ tounderstandtheimpactofthisseeminglyobviouspoint.Thereareexamplesof measurementsbeingdoneforwhichnoquantitycanbeclearlyidentified(e.g., flavor, feel, consumerconfidence).Forthese,itmaybedifficulttoapplyconceptsoftraceabilityandstandards.Notallqualitativelydistinctattributesaresubjecttomeasurement,however.Anattributemaybestrictlyqualitative(forexample,whethera z+&+ computerprogramisawordprocessororapaintingisbeautiful).Tobesubjecttomeasurement,itmustbepossibletodetermineanattributequantitatively.Apropertyisaquantityifitallowsalinearorderingofsystemsaccordingtothatproperty.3Inotherwords,apropertypisa  quantityifonecanalwayssayoftwosystemspossessingpthatthetwo p areequalinporthatonesystemislessthantheotherinp.Assigning `  numberstopropertiesisnotenough.Thenumbersmustbemeaningfulintermsofanorderingrelationshipamongobjectspossessingthatproperty.Thisrequirementeliminatesmanytaxonomicrelationshipsfromthepossibilityofquantitativetreatment. UnitsandScales  P  Theexistenceofaquantityisanecessary,butnotasufficient,requirementfortheexistenceofameasurement.Inordertomakemeasurements,itisalsonecessarytobeabletoassignnumberstoquantities.Ellisproposesthefollowingdefinitionsforameasurement:4  8   1) 0 ! !Measurementistheassignmentofnumeralstothingsaccording  toanydeterminative,nondegeneraterule. !! 8   2) 0 ! !Wehaveascaleofmeasurementifandonlyifwehavesucha p rule. !! Thisspecificationisquiteopenended,sincetheruleofassignmentisarbitrary.Forthemeasurementofaspecificquantity,however,headdsadditionalrequirementstotheeffectthatthenumeralsobtainedbymeasurementareconsistentwiththeorderingdeterminedbythequantity.Otherauthoritiesaremorespecificabouttherequirementsofmeasurement.Theiraimistodefinemeasurementinawaythatconformstointuitivenotions.Tothisend,thefollowingrequirementsareusuallyputforth:5   8   1) 0 ! !Thereisaruleforassigningadistinguishedvalue(usually "  zero)tothequantity; !! 8   2) 0 ! !Thereisaspecified,reproduciblestateofobjectsforwhicha $" second,distinguishedvalue(usuallyone)ofthequantityshouldbeassigned(thatis,thereshouldbeaunit);andp&!$!! 8   3) 0 ! !Thereisascale,ofmultiplesandsubmultiplesoftheunit,for `'"% whichthereisarulestatingtheempiricalconditionsunderwhichtwointervalsbetweenmeasuredvaluesareequal.(Forexample,acentimeteristhesameintervaloflengtheverywherealongaruler.) !!  ,`'* ЇThereis,however,thepossibilityofanothertypeofmeasurement.6For  thesemeasurements,therequirementoforderingcanbereplacedbyalooserrequirementofequality.Thisissupplementedbytwoadditionalrules:thatoftheunit(number2above)andanewrequirementthatquantitiesbeadditive.Thismeansthatwhentwoobjectspossessingaquantityarecombined(inawelldefinedway),thecombinedobjectpossessesthequantityinamagnitudethatistheexactsumofthemagnitudesofthequantityinthecomponents.Thus,forinstance,acombinedobjecthasamassequaltothesumofthemassesofitscomponents.(Notallquantitiesareadditive:whenequalamountsofwateratagiventemperaturearecombined,theresultantwaterwillnothaveatemperaturethatisthesumofthetemperaturesoftheindividualamounts.)TheVIMdefinesavalueofaquantityasa magnitudeofaparticularquantitygenerallyexpressedasaunitofmeasurementmultipliedbyanumber.However,itallowsthepossibilitythataquantitymightnotbeexpressibleasaunitofmeasurementmultipliedbyanumber.Inthatevent,itmaybeexpressedbyreferencetoaconventionalreferencescaleand/ortoameasurementprocedure.Theprocessofdefiningquantities,units,andscalesisoneofestablishingaconsensus.Generally,thereisacertainlevelofarbitrarinessinthisprocess,andothersystemscouldhaveservedequallywell.ThisiscertainlytrueoftheSIsystemofunits.Havingsaidthat,thereisalsoagreatdealofempiricaltruthconstrainingthedevelopmentofasystem.Tobepracticable,asystemofquantitiesandunitsmustbebothinternallyconsistentandconsistentwithrealityasweexperienceit.Likewise,thestartingpointisnevertheunit;itisalwaysnecessarytostartwithadefinitionofthequantitytobemeasured.(Thus,forinstance,sayingthatthe bitisaunitofmeasureinITisnotvalidwithoutspecifyingwhatquantityisbeingmeasured.Thebit,forinstance,canbeusedtomeasureopticalresolvingpower,7,8probablynotwhatmostcomputerscientists "  associatewiththeterm.) RealizationandReferences  % # Definitionsofquantityandunitarenotenoughtoprovideameansofmeasurement.Measurementis,inessence,thecomparisonofanobjectnottotheunitofthequantitybeingmeasured,buttoaphysicalrealizationoftheunit.AsstatedbyEllis:9 0*%(  ,`'* ![  ! !  8 Thethingtobemeasuredismatched,inrespecttothequantityconcerned,byaseriesofoperationswiththemembersofasetofstandards,ortheirequivalents.  ! ! TheVIMdefinesanumberoftypesofstandards.Thereisusuallyone,distinguishedstandard: #.XY. XX<XYC#LX"X X.XY.L; "XLXprimarystandard  @  8 standardthatisdesignatedorwidelyacknowledgedashavingthehighestmetrologicalqualitiesandwhosevalueisacceptedwithoutreferencetootherstandardsofthesamequantity#LX"X L;7^##.XY. X"XLX^#<XX X.XY.@  ! ! #.XY. XX<X⦆_#<XX X.XY.Therealizationofaunitusuallytakestheformfirstofaprimarystandard. >  Thisisaphysicalobjectorphenomenondeemedtoembodytheunitofthequantityinquestion.IntheSIsystem,onlytheunitofmass(thekilogram),isdefinedintermsofanartifact.Allotherunitsaredefinedintermsofscientificprinciplesandtherealizationoftheunitisatechnologicalchallenge.#.XY. XX<X_#<XX X.XY.  #.XY. XX<Xa#<XX X.XY.Secondarystandardsarestandardswhosevaluesareassignedbycomparisonwithaprimarystandardofthesamequantity.Secondarystandardsareusedwhenitisimpracticalforallmeasurementstobemadebydirectcomparisontotheprimarystandard. MeasuredValues  .~ Ameasuredvalueisthenumericalresultobtainedfromtheapplicationofameasurementmethodtoanobject,possessingaquantity.Onecharacteristicofameasuredvalueofinteresttothetaskgroupistraceability.Muchoftraderequirestraceablemeasurements.TheVIMdefinitionis: #.XY. XX<X"b#LX"X X.XY.#.XY. X"XLXݿd#LX"X X.XY. L; "XLXtraceability#.f L;e#L; .f  "! 8 #.f L;{e#L; .fpropertyoftheresultofameasurementorthevalueofastandard .$~# wherebyitcanberelatedtostatedreferences,usuallynationalorinternationalstandards,throughanunbrokenchainofcomparisonsallhavingstateduncertainties  ! ! #LX"X L;-e##.XY. X"XLXe#<XX X.XY.ThisdefinitionisintendedtobeappliedwithinasystemofmeasurementsthatconformstoFigure1.AchallengefacingNISTistoapplythedefinitionoftraceabilitytoassessmentsofITproductcharacteristics.Itisnecessarytoeitherputintoplaceametrologysystemthatisconsistentwiththeexistingstructure,ortoextendthestructuretoincludeITproducts. ,(- 6Oa  6 6  Number,Counting,andProbability   Itisworthbrieflyexaminingthelogicalstatusofcountingandofprobabilityinthephilosophyofmetrology.#.XY. XX<XZg#<XX X.XY.ԀHistorically,somequestions  havebeenposedaboutcountingandprobabilitywhicharesomewhatironicalsincesomanyphysicalmeasurementsarebasedupontheseconcepts.Theprocessofcountingposesdifficultiesforphilosophers:iscountingobjectsameasurementprocedure?Inonesense,itseemstobe.Certainly,numberisaquantityinthesensethatitsatisfiesthepreviousdefinitionsofaquantity.Whatseemslackingisthearbitrarinessofascaleofmeasurement;thereseemsnothingwhichcorrespondstochoosingaunit.AsEllisstates, Ifwemustspeakofcountingasameasuringprocedure,itisuniqueamongallmeasuringprocedures.Carnapclaimsthatmeasurement goesbeyondcountinginthatitgivesvaluesthatcanbeexpressedbyirrationalnumbers,henceenablingtheapplicationofcalculusandotherpowerfulmathematicaltools.However,manyphysicalphenomena(suchascharge)areinessencediscrete.Despitetheirdiscretenature,advancedmathematicaltoolsareusedtoanalyzequantitativerelationshipsamongthem,measuringthem,andtreatingmeasuredvaluesashavinguncertainty.Ifdiscretequantitiesareessentiallydifferentfromcontinuousones,thelogicalbasisofthedistinctionhasnotbeenclearlyputforth.Probabilitypresentsdifferent,butequallyseriouschallengestophilosophersofmeasurement.Istheassessmentofprobabilityameasurement?Inthesenseofprobabilityas relativefrequencyoras subjectiveprobabilitythereseemstobeagreementthatthisisindeedmeasurement,sincetheoutcomedependsontheactualstateoftheworld.However,probabilityisunderstoodinanothersense:as degreeofconfirmation.Carnap10claimsthatthetermprobabilityisambiguous,involvingtwo $" distinctkinds(whichmaybecalledempiricalandlogical).Moreimportantly,heclaimsthatassessmentoflogicalprobabilityisnotmeasurement.Ellis,however,arguesthatthedistinctionbetweenkindsofprobabilityisbasedonreasoningthatcanbeappliedtoeveryotherquantityconcept.Hisconclusionisthat,justasthedistinctionbetweenempiricalandlogicaltemperature,length,etc.areunimportant,soisthedistinctionbetweenempiricalandlogicalprobability.Allsuchassessmentsshouldbeconsideredmeasurements. ,`'* Ї/.XY. X%%.%X<Xj/ : XX X.XY. ' X X     :s  PrinciplesofITMetrology :s:t "tԌ  Ќ.   '.XY. X .  +.XY. X%%.% X.XY.+AfterreviewingthelogicalrelationshipsbetweenmetrologyconceptsillustratedinFigure1,thetaskgroupbelievesthattheseconceptsandtheconceptoftraceabilityapplytometrologyforIT.However,itisimportanttorecognizetwoaspectswhichdelineateordistinguishITmetrologyfromphysicalmetrology.First,usefulITquantitiesarenotrealizablesolelybyuseofaphysicaldimensioningsystem;suchasSI.*inguishedst #  *      Secondly,existing v  methodsforcalculatingexpressionsofuncertaintyinphysicalmetrologycannotbeeasilyoralwaysappliedinITmetrology.Thereappearstobenorecognized,establisheddimensioningsystemorquantitiesrelevanttoITmetrology.OfthesevenbaseunitsinSI,onlythe secondfortime,appearsessentialforITmetrology.Possibly,theonlyotherbaseunitnecessaryforITmetrologyisthe bitforinformation.ThereisnoequivalentinITmetrologytotheISO1000(andISO31)forSIinphysicalmetrology.Possiblydevelopingsuchanequivalentwouldbeuseful,maybenot.OneadvantageinITmetrologyappearstobethat,whateverbaseandderivedunitsareused,thetechnologicalchallengeposedinrealizingSIunitsdoesnotexist.Inotherwordsanyonecandefineandestablisha bitofinformationwithoutuseofameasurementdevice.Possiblyallthatisneededtodefinethequantityofinformationisreferencetoaclassicwork,suchasMathematicalTheoryof J CommunicationbyShannonandWeaver.15Suchworkprecededthe 6 present,dramaticdeploymentofdigitalITsystemsbutstillmaysufficientlycharacterizeinformationasaquantityandbitasaunitofmeasure.TheVIMdefinitionoftraceabilityrequiresevaluationofuncertainty.ForITmetrology,uncertaintycanbedifficulttodefine,muchlesstoquantify.StatisticalmethodsoftreatingrepeatabilityandaccuracyinphysicalmetrologydontclearlyapplytothemanylogicalmeasurementsassociatedwithIT.Whentestresultsarerepresentedbypass/failinsteadofquantitativeresultsorwhentestresultscannotexhaustivelytesttoanITstandard(i.e.,numberofpossibletestsaretoolargetoeconomicallyor Z$" quicklycomplete),itappearsthatmethodsforestablishingalevelofconfidencearemoreusefulforestablishingtraceabilityinITmetrology.#.XY. X X.XY.t#Ԁ  Figure2illustratesandcomparestheconceptsofmeasuringphysicalquantitiesandmeasuringdigitalinformationtechnologysystemsquantities.Figure2includesandexpandsuponthemetrologicalconceptsillustratedbyFigure1.Theconceptof definition fromFigure1mapsinto 8  the specification rowinFigure2.Theconceptsof realization , ( x  dissemination ,and measurement fromFigure1mapintothe methodsof  h testing rowinFigure2.Figure2addsathirdrowfor commercial  X  products toillustratehowcommercialproductsdependupon  H  measurements.Therefore,thethreerowsinFigure2areintendedtoshowhowspecifications,whichmayemployphysicalordigitalinformationsystemsquantities,areimplementedcorrectlyincommercialproductsbyuseofappropriatemethodsoftesting.ThethreecolumnsinFigure2(fromlefttoright)areintendedtoshowhowspecifications,methodsoftesting,andcommercialproductscanbecomeincreasinglycomplex.Theconformanceofimplementations(commercialproducts)withrespecttothespecificationmaybeestablishedthroughtraceabilitycalculationsorlevelofconfidenceassertions.   p  O  e    GOaIhJOaрU0QA=~|7( p @E7|* ((7|*   D2 D]TRY3' Letter3' 3' Letter3'T D  Inanefforttodevelopataxonomyformethodsoftesting,thefollowingkeydefinitionsinFigure e 3werecollected.Wheredefinitionscouldnotbefound,thetaskgroupdevelopeditsowndefinition.FromFigure3,thetaskgrouphasdevelopedataxonomyoftestingormeasuring:  "  "x"0    calibration"x݌(#(# Ќ       4 referencematerial     "  "g"0    inspection"g݌ (#(# Ќ    "  " "0    referencedata" $݌ t(#(# Ќ    "  ""0    conformancetesting"ʊ݌ `(#(# Ќ       4 referenceimplementation  L    "  ""0    interoperabilitytesting"Nj݌ 8 (#(# Ќ       4 referenceimplementation $    .:4 X.XY.Թ  KeyDefinitions#.XY. X4.:#  4  *Udgddp  (#(#U, dd ,dd ,dd +   D   Term 2!   2  Definition 4!   4  Source/%#    /*wstdd dd dd dd dg(#(#w, dd ,dd ,dd +   ) calibration ( Setofoperationsthatestablish,underspecifiedconditions,therelationshipbetweenvaluesofquantitiesindicatedbyameasuringinstrumentormeasuringsystem,orvaluesrepresentedbyamaterialmeasureorareferencematerial,andthe ) correspondingvaluesrealizedbystandards   VIM  ( conformity 0 Fulfilmentbyaproduct,processorserviceofspecifiedrequirements.   ISO/IECGuide2  0 conformityevaluation   Systematicexaminationoftheextenttowhichaproduct,processorservicefulfillsspecifiedrequirements.  t# ISO/IECGuide2  $ conformitytesting Y!% Conformityevaluationbymeansoftesting  Y!& ISO/IECGuide2  Y!' inspection "t( Conformityevaluationbyobservationand "t) judgementaccompaniedasappropriatebymeasurement,testingorgauging.  $L + ISO/IECGuide2  "t, interoperabilitytesting 1&!- Thetestingofoneimplementation(product,system)withanothertoestablishthattheycanworktogetherproperly.   (#0 TaskGroup  1&!1 meansoftesting )$%2 Hardwareand/orsoftware,andtheproceduresforitsuse,includingtheexecutabletestsuiteitself,usedtocarryoutthetestingrequired.  M,'6 ISO/IEC96461  )$%7 u]  measurement 'd ' Setofoperationshavingtheobjectofdeterminingavalueofaquantity. '   'VIM '   'referencedata  ,  Inphysicalmetrology,referencedataisquantitativeinformation,relatedtoameasurablephysicalorchemicalpropertyofasubstanceorsystemofsubstancesofknowncompositionandstructure,whichiscriticallyevaluatedastoitsreliability.Ininformationtechnology,referencedataisanydatausedasastandardofevaluationforvariousattributesofperformance.   TaskGroup   , reference   implementation u Implementationwhoseattributesandbehavioraresufficientlydefinedbystandard(s),testedbycertifiabletestmethod(s),andtraceabletostandard(s)thattheimplementationmaybeusedfortheassessmentofameasurementmethodortheassignmentoftestmethodvalues.  %  TaskGroup   ! referencematerial D" Materialorsubstanceoneormoreofwhosepropertyvaluesaresufficientlyhomogeneousandwellestablishedtobeusedforthecalibrationofanapparatus,the y& assessmentofameasurementmethod,orforassigningvaluestomaterials.  Y!( VIM  D) test "x* Technicaloperationthatconsistsofthedeterminationofoneormorecharacteristicsofagivenproduct,processorserviceaccordingtoaspecifiedprocedure.  %H!. ISO/IECGuide2  "x/ testing 1'"0 Actionofcarryingoutoneormoretests.  1'"1 ISO/IECGuide2  1'"2 traceability 'd 'Propertyoftheresultofameasurementorthevalueofastandardwherebyitcanberelatedtostatedreferences,usuallynationalorinternationalstandards,throughanunbrokenchainofcomparisonsallhaving stateduncertainties. 'y  'VIM"d ".h X.Xr Figure3 #.Xr X.ht#  :    Allofthesemethodsoftestingormeasuring(calibration,inspection,referencedata,conformancetesting,interoperabilitytesting)areapplicabletoeitherphysicalordigitalITsystemsmetrology.ManyofthetermsinFigure3aredefinedinbasicmetrologyorconformityassessmentdocuments(VIM1,ISO/IECGuide216).Somewhatsurprisingly,thetaskgroupwas e  unabletofindsuitableexistingdefinitionsforinteroperabilitytesting,referencedata,andreferenceimplementation.SuitabledefinitionsforthesetestingmethodsweredevelopedbythetaskgroupinordertoallowforacompletediscussionaboutallofthemethodsoftestingpresentlybeingusedfordigitalITsystemsquantities.ItisinterestingtonotethattheVIMdefines measurement butnot test or testing andthatthe  ISO/IECGuide2defines test and testing butnot measurement .Tothetaskgroup,   measurement and testing appeartobedefinedsothatthesetermsareeitherconceptually  equivalentor,atleast,veryclosetoequivalent.Therefore testingandmeasurementareoftencombinedinthiswhitepapernottodelineatebuttoemphasizetheirroughequivalence.Thetaskgroupalsoacknowledgesthat,insomefields,adistinctionbetweenthesetermsismadebyconsideringtestingtobeameasurementtogetherwithacomparisontoaspecification. XX X.Xr  X X    ϥ  MethodsofTestingforDigitalITSystemsQuantitiesϥJ 2Ԍ   Ќ.f   .Xr X.f   +.Xr X% %p X.Xr+Ofthefivemethodsoftestingidentifiedintheprevioussectioncalibration,conformancetesting,interoperabilitytesting,referencedata,andinspection,allbutcalibrationareinwidespreaduseasmethodsfortestingfordigitalITsystemsquantities.Conformanceandinteroperabilitytestingoftenmakeuseoftheconceptofreferenceimplementations.ThefollowingprovidesabriefreviewandstatusonmethodsoftestingfordigitalITsystemsquantities. Calibration  ' #& Theconceptofcalibrationiswellunderstoodinthephysicalmetrologycommunity.Calibrationmeansthatthemeasurementofthevalueofthepropertiesisrelatedtomeasurementsonprimarystandardsusuallyprovidedbytheprimarynationallaboratory.Therelationiscalledtraceability. 5,'+ Thepurposeofcalibrationandtraceabilityistoensurethatallmeasurementsaremadewiththesamesizedunitsofmeasurementtotheappropriatelevelofuncertaintysothattheresultsarereliablycomparablefromtimetotimeandplacetoplace.Thedefinitionoftraceabilityistheabilitytorelateindividualmeasurementresultsthroughanunbrokenchainofcomparisonsleadingtooneormoreofthefollowingsources:nationalprimarystandards,intrinsicstandards,commercialstandards,ratios,andcomparisontoawidelyusedstandardwhichisclearlyspecifiedandmutuallyagreeabletoallpartiesconcerned.IntheopensystemssubcommunityofIT,ISO/IECTR1323317؀states Sincemeasurement  p  traceabilityandcalibrationarenotgenerallydirectlyrelevanttosoftwareandprotocoltesting,thetitleofclause9inthisinterpretationhasbeenchangedtoValidationandtraceability.Thisreportconcludesthatvalidationistosoftwareandprotocoltesttoolsascalibrationistomeasurementequipment. ConformanceTesting  u TheITmethodoftestingwiththegreatestamountofexperience,widespreaduse,anddevelopmentofmethodologyisconformancetestingofdigitalITsystems.Testingmethodologieshavebeendevelopedforoperatingsysteminterfaces18,computergraphics19, 5 documentinterchangeformats20,computernetworks21,andprogramminglanguageprocessors22. % Additionally,aboutfifteenyearsago,ITstandardsdevelopersbegantorealizethatstandardsfordigitalITsystemswerebecomingquitecomplexanddependentuponbothphysicalmetrologyandnonphysicalmetrology.Consequently,assessingconformityofhardware/softwareimplementationsisnowoninherentlycomplexandsomewhatambiguousprocess.Thereareonlyaveryfewdocumentswhichaddresssuchconformityissues23,24. p Mostofthetestingmethodologydocumentscitedaboveusethesameconcepts,ifnotthesamenomenclature.ITstandardsarealmostalwaysdevelopedandspecifiedinanaturallanguage,English,whichisinherentlyambiguous.Sometimesthespecificationsareoriginallydevelopedortranslatedintoamoreunambiguouslanguagecalledaformaldescriptiontechnique(FDT).SincethespecificationsinITstandardsareoftenverycomplex,aswellasambiguous,mosttestingmethodologydocumentsrequirethedevelopmentofasetoftestcasescenarios(e.g.,abstracttestsuites,testassertions,testcases)whichmustbetested.Thestandardsdevelopingactivityusuallydevelopsthestandard,theFDTspecification,thetestingmethodology,andthetestcasescenarios.Executabletestcodewhichteststhetestcasescenariosisdevelopedbyoneormoreorganizationswhichmayresultinmorethanoneconformancetestingproductbeingavailable.However,ifarigoroustestingmethodologydocumenthasbeenadheredto,itshouldbepossibletoestablishwhethereachconformancetestingproductisaqualityproductandanequivalentproduct.Sometimesanexecutabletestcodeandtheparticularhardware/softwareplatformitrunsonbecomeacceptedasareferenceimplementationforconformancetesting.Itshouldbenotedthat,onoccasion,awidelysuccessfulcommercialITproductbecomesboththedefactostandardandthereferenceimplementationagainstwhichothercommercialproductsaremeasured. ,P(+ GOa G G  <Xr X X.XrInIT,anexampleofaprimarystandardmightbeareferenceimplementationofafunction e (assumingthatsuchanimplementationisameasurementstandardtobeginwith).#.Xr X X<Xr#/.Xr X% %p X.Xr$/+.Xr X% %p X.Xr+<Xr X X.XrItispossible U tohavemultipleprimarystandards(or,dependingononesviewpoint,noprimarystandard).Forinstance,areferenceimplementationofanalgorithmmayberunningontwo(nominallyidentical)machines.Thisraisesissuesbecausethebehaviorofthetworunningsystemsmaydiffer;mechanismsmustbeestablishedforintercomparisonofprimarystandards.#.Xr X X<Xr#/.Xr X% %p X.Xr/+.Xr X% %p X.Xr+    InteroperabilityTesting    Nointeroperabilitytestingmethodologieshavebeenestablishedcomparabletoexistingconformancetestingmethodologies.Interoperabilitytestingusuallytakesoneofthreeapproachestoascertainingtheinteroperabilityofimplementations(i.e.,commercialproducts).Thefirstistotestallpairsofproducts.TypicallyanITmarketcanbeverycompetitivewithmanyproductsanditcanquicklybecometootimeconsumingandexpensivetotestallofthecombinations.Thisleadstothesecondapproachoftestingonlypartofthecombinationsandassumingtheuntestedcombinationswillalsointerwork.Thethirdapproachistoestablishareferenceimplementationandtestallproductsagainstthereferenceimplementation.#.Xr X X.Xr# ReferenceData  E TheuseofreferencedataisveryimportantinbothphysicalandITmetrology.Whenthetaskgroupcouldnotfindanyexistingdefinitionforreferencedata.ThetaskgroupturnedtoNISTexpertsforsuggestions,andasaresult,Figure3hasseparatedefinitionsforreferencedataasappliedtophysicalandITmetrology.ForIT,referencedataisusedtomeasurevariousaspectsofperformanceofdigitalITsystems. Inspection  ` Inspection,asamethodoftesting,isaconceptthatappliesequallywelltoeitherphysicalorITmetrology.TherehasbeenatleastoneattempttodocumentaninspectionmethodologyforoneareaofIT,theevaluationofsoftwareproducts.25   Inspectionofcomplexstructures,forinstancebuildings,inphysicalmetrologyhasalegacyofmanydecadesofexperience.WhileinspectionofdigitalITsystemsisarelativelynewareacomparedtobuildinginspections,thereisoneadvantageinITmetrology.Intheareaofsoftwareproducts,eachcopyofaproductcanreasonablybeassumedtobeidenticalandinspectionofonecopyisthereforesufficienttoknowsomethingaboutallcopies.Thepass/faildecisionbasedoninspectionisusuallymoresubjectivethanobjective.Thisforcestwonecessaryconditions.Thefirstconditionisthattheinspector(thepersonperformingtheinspection)isqualifiedtomakeasubjectivedecision.Thesecondconditionisthatthesurroundingenvironmentbeasdefinedandconsistentwithsimilarinspectionsaspossible.Forexample,todeterminethatanapplicationproducesacorrectcolorforviewinganinspectioncouldbeperformed.Theconditionsthatwouldbedefinedfortheinspectioncouldbetheroom ,P(+ lighting,thehardware/softwareplatformoftheapplication,themonitortypeusedfortheinspection,andtheexpertiseoftheinspector. XX X.Xr  X X    +  StatusandOpportunitiesforITMetrology+ Ԍ 5 Ќ.f   .Xr X.f   ThestateofITmetrologyisbestillustratedbycomparingittothestateofphysicalmetrology.Manyofthedefinitionsandgeneraltermsformetrology1,standardization16,andrequirementsfor  @ calibrationandtestinglaboratories؀(ISO/IECGuide25)26applyequallywelltophysicalandIT  0 metrology.ITmetrologyhassomeconceptsandtermsforwhichnowellestablisheddefinitionsexist(e.g.,referencedata,interoperabilitytesting,referenceimplementation).Also,someITtestersbelievethattherequirementsinISO/IECGuide25forcalibrationandtestinglaboratoriesrequireextensiveinterpretationforITtestingandhavespentconsiderabletimeandresourcesindevelopingsuchaninterpretation17؀.؀OtherITtestersbelievethatISO/IECGuide25؀issufficient, E  withoutextensiveinterpretation,forITtesting.Forphysicalmetrologythereareatleastseveraldecadesofpapersrefiningmetrologicalconceptssuchastraceability.27,28,29,30󀀀Thereisnocomparableliteraturefordeterminingthelevelof  confidenceinITtestresultswhichmightservethesamepurposeasestablishingtraceabilityinphysicalmetrology.NISTstaffmembershavebeenmajorparticipantsintheadvancementofphysicalmetrology.TheITequivalentofphysicalmeasurementuncertaintymaybestraightforwardor,formorecomplexsoftware,agenuinefrontierforITmetrology.Threeexamplescanillustratethespectrumofdifficultyindealingwithuncertaintyinsoftwaremeasurements.Inthefirstcase,asoftwarestandardmaybeunambiguousandthecombinations/permutationstobetestedarefiniteandpossibletoexhaustivelytest(e.g.,128charactersinsevenbitASCII).Inthesecondcase,asoftwarestandardmaybeunambiguous(e.g.,anencryptionalgorithmsuchasDES)andthecombinations/permutationstobetestedareverylargeandnotfeasible/possibletoexhaustivelytest(e.g.,DEShasmorethan10**36possibletests).Inthethirdcase,asoftwarestandardmaybesomewhatambiguous(e.g.,thesyntaxandsemanticsforaprogramminglanguage,suchasC)andthecombinations/permutationstobetestedareverylargeandnotfeasible/possibletoexhaustivelytest(e.g.,possibleCcodeisinfinite).Intheabovefirstcase,uncertaintyisclearlymoremeasurablethantheabovethirdcase.Recently,therehasbeenseveralcontributiononcomputerssystemsinmetrologyandtheneedforanempiricalsciencefortheperformanceofalgorithms.31,32,33,34Again,NISTstaffmembers %p!# havecontributedtothisliteraturewhichisofpotentialvaluetoadvancingbothphysicalandITmetrology.ThereisalargeamountofliteratureonITmetricsandmeasurement.Arecentsearchonamajorsearchengineonthewebnettedover150thousandentrieson software+metric.Mostofthisliteraturediscussesapplyingexistingmetricsforquality,size,complexity,orperformanceandrefiningthesemeasures.Thereisverylittlediscussiononfundamentalmeasurementstrategies e,(*  [Ih  I I  forIT.ThetaskgroupknowsofnojournalsdevotedtoITmetrologyasthereareforphysicalmetrology(e.g.,CALLABTheInternationalJournalofMetrology).Therearenewsletters,##35 U journals,andbooksonsoftwareengineeringandtestingtechniqueswhichincludediscussionsofmetricsandmeasurements.Atleastonestandardforsoftwaremeasurementisbeingdeveloped.36 5 Therearealsoconference,symposia,37andongoingresearch38inthearea.Mostofthese % publicationsandactivitieshaveoccurredinthelastthirtyyearssincetheITfieldisfairlyyoung. Opportunities    FromtheliteraturereviewedanddiscussionsheldbythetaskgroupitisapparentthattherearenumerousareaswithopportunitiestoadvancethestateofITmetrology.Someareasarealreadybeingworkeduponbyindustry.Otherareashaveseenrelativelylittlestudyanddevelopmenttodate.Innoparticularorder,thetaskgroupsuggeststhefollowingareareaswithopportunitiesforadvancingITmetrology:1.0  LevelofconfidenceintestresultsToday,thequalityofaninformationtechnologyproduct u orcomponentisassuredwithoutrigorousmetricsfortheconfidencefactor.Forinstance,commercialproducersofsoftwaremayuseacombinationofthefollowingtodecidethataproductis goodenoughtorelease: (#(# 0  0(#(#asufficientpercentageoftestcasesrunsuccessfully%(#(# 0  0(#(#  0(#(#executingatestsuitewhilerunningacodecoverageanalyzertogatherstatisticsabout  whatcodehasbeenexercised (#(# 0  0(#(#classificationofdefectsintodifferentseveritycategories,andanalysisofnumbersand  trendswithineachcategory (#(# fc  0  fc 20(#(#  betatesting:allowingrealuserstorunaproductforacertainperiodoftimeand p reportingproblems;analyzingtheseverityandtrendsforreportedproblemsfc 2݌ (#(# Ќ  fc  0  fcW20(#(#  analyzingthenumberofreportedproblemsinaperiodoftime;whenthenumber P stabilizesorisbelowacertainthresholdforaperiodoftime,itisconsidered goodenough.fcW~݌ (#(# Ќ  0  Althoughcodecoverageandtrendanalysisareinitialstepstowardsamorerigorousdefinitionofcertaintyofaproductsquality,thereisstillmuchworkthatisneededindefiningthemathematicalfoundationsandmethodsforassessingtheuncertaintyinqualitydeterminations. (#(# 0  ITmetrologywouldprofitfromthedevelopmentofanequivalentsetofconceptstocalibration,traceability,anduncertaintywhicharesoimportantinphysicalmetrology.Whereuncertaintyiscalculatedbystatisticalmethodsforphysicaltestresults,thelevelofconfidencecanbecalculated.BeingabletoanalyticallyderivealevelofconfidenceforITtestresultswouldadvanceITmetrology. (#(# 2.0  InteroperabilitytestingIfimplementationAandimplementationBinterworkandif +`'* implementationBandimplementationCinterwork,whataretheprospectsofimplementationsAandCinterworking? (#(#  .0*- JOa J J  3.0  AutomaticgenerationoftestcodeDevelopingtestcodeforITconformancetestingcanbe e moretimeconsumingandmoreexpensivethandevelopingthestandardoraproductwhichimplementsthestandard.Thereareseveraleffortsinspecifyingmoreformallythestandardorspecificationandgeneratingtestcodefromthisformalization.OneexampleistheAssertionDefinitionLanguage(ADL)effortmanagedbyX/Open,withfundingfromMITIbasedonongoingresearchatSun.39,40,41Thereisotherongoingresearchbasedonmodeling,   finitestatemachines,combinatoriallogic,andotherformallanguagessuchasZ. (#(# 4.0  NeedforITdimensioningordescriptionsystem(s)Thegeneralconceptoffundamentaland   derivedunitsforITmetrologyhasbeenraisedinthispaper.Isthereaneedtoexpanduponthisconcept? (#(# 0  Ageneralvocabularyneedstobedevelopedtodescribecomponentswhichcompriseinformationsystems.Thisentailsdevelopingarich,standardizedterminologytocapturethefunctionalityandcapabilitiesofasoftwarecomponent,inadditiontotheinterfacespecifications.Thiscouldbeconsideredanalogoustothesituationoneseescurrentlyinthemicroelectronicshardwareworld,whereacircuitdesignerchooseschipsandchipsetsforaboarddesignbaseduponpublishedspecificationsdetailingperformancecharacteristics.Thisispossibleforhardwaresystemsbecausespecificationsexistthatcomprehensivelydefinetheperformanceofhardwarecomponents. (#(# 0  Thedefinitionoftheseformalspecificationsinastandardized,rigorouswaywillenabledesignersandsystemsintegratorstoselectsoftwarecomponentswithconfidenceregardingthecomponentscapabilitiesandhowitwillintegrateintothesystembeingbuilt.Furthermore,automatedcompositionofsystemsbasedonspecificationswillbepossibleoncethesetypesofdefinitionsexistandarewidelydeployedinacertifiableway. (#(# 5.0  SoftwaremetricsTheneedtomorerigorouslymeasureandtestsoftwareasitisdeveloped P isbeingexploredbyindustry.Assoftwareproductsbecomeincreasingcomplex,soundsoftwaremetricswillbeneeded. (#(# 6.0  AlgorithmtestingAsresearchersdevelopnewalgorithms,somemeansofmeasuringthe u! performanceofthesealgorithmsforcomparisonpurposesisneeded.Thereexistsomemeasuresofperformancetoday,suchasWhetstones,Dhrystones,etc.whicharebenchmarkingprogramstargetedatspecificaspectsofacomputerscapabilities.Amoregeneralcapabilityforestablishingtheperformanceofalgorithmsinasimilarfashionshouldbedeveloped.Forexample,planningorschedulingalgorithmscouldberunagainststandard (#(# 0  datasetsorscenarios(artifacts?).Thereareseveralchallenges,including:determinationofatheoreticalfoundationformeasuringtheperformanceofalgorithms,andmeansofensuringthatimplementationdependentperformanceresultsaremeaningful. (#(#  ,P(+  XX X.Xr  X X      RolesforNISTinITMetrologyx `Ԍ e Ќ.f   .Xr X.f   ThetaskgroupdevelopedFigure2toillustrateaconceptualbasisforphysicalandITmetrology.Figure2alsoservesasaframeworkfordiscussingNISTsroles.AsakeynationalmeasurementlaboratoryforU.S.industry,thetaskgroupbelievesNISTalreadyservesinmanymeasurementrolesforallthreecolumnsinFigure2formeasuringbothphysicalquantitiesanddigitalITsystemsquantities.ForthetestingofdigitalITsystems,NISThasbeenveryactiveinthetestingofcomplexspecifications.Inthisarea(i.e.,therightsideofFigure2)NISThasasuccessfulhistoryofprovidingkeytestingsupport.Forphysicalmetrology,NISTclearlyhasprovidedkeymeasurementsupportforfundamentaltocomplexspecifications(i.e.,fromlefttorightsideinFigure2).ThereisalsoasubstantivehistoryofworkbyNISTinthemathematical,computational,andstatisticalscienceswhichsupportallofthecolumnsinFigure2.Inotherwords,NISTsrolesinmetrology(past,present,andfuture)are,appropriately,theentirematrixofFigure2.ItshouldbenotedthatNISTsITmetrologymandatewillalwaysbeboundedbyavailableresources.Forinstance,iftheITindustryweretolooktoNISTforassistanceindevelopingallofitsconformancetestingneeds,theassociateddevelopmentcostscouldoverwhelmtheentireNISTmeasurementbudget.NISTwillhavetocontinuetoprioritizeitsprogramofworkinITmetrologyaspartofitsoverallmetrologyprograminsupportofU.S.industry.XX X.Xr XXXX  X X    ;  Conclusions; Ԍ 0 Ќ.f   .f .f .Xr X.f   ITmetrologyisavalidbranchofmetrology.Thetaskgroupstartedwiththisasanassumptionandendedwiththisasabelief.ITmetrologydiffersfromphysicalmetrologyinseveralwaysincluding;theSIdimensioningsystemisnotasrelevant;lessanalyticalmethodsexisttoquantifyuncertainty;andtheareaisrelativelynewcomparedtophysicalmetrology.AllofthismeansthatITmetrologyhasitsownuniquesetofchallenges,opportunities,andpriorities.ITandITmetrologywillbeakeytoU.S.competitivenessandinternationalcommerceinthetwentyfirstcentury.AdvancingITmetrologyandsupportingspecificpriorityITtestingandmeasurementneedsofU.S.industryshouldbekeygoalsforNIST.Thispaperhasattemptedtoproposeconcepts,provideinformation,andposequestionswhichmighthelptoestablishaframeofreferenceforNISTstaffandmanagementastheyconsiderhowtoadvanceITmetrologyandsupportU.S.industrysITtestingandmeasurementneeds.  U(#%  XX X.Xr  X X      AnnexA:References"  Ԍ e Ќ.f   .Xr X.f   }I^  E%e , e, , XX, XE}I^|1  .0 '   InternationalVocabularyofBasicandGeneralTermsinMetrology.International  @   OrganizationforStandardization:Geneva.1993.}I^<|ی '(#'(# Ќ  0 '  2. ' StephanKorner, ClassificationTheory,EncyclopediaBritannica:Macropaedia,15th @!  @! ed.,  P 1977.AccordingtoKorner,weorganizeourunderstandingoftheworldinthreeways:objectsandtheirattributes;objectsandtheirparts;andrelationshipsbetweendistinctclassesofobjects. '(#'(# |}I^  }I^9|3  .0 '   BrianEllis,BasicConceptsofMeasurement.CambridgeUniversityPress:Cambridge, @!  @!  e  England.1966.}I^9T|ی '(#'(# Ќ  4.0 ' Ellis,op.cit.,p.41.5 '(#'(# |}I^  }I^|5  .0 '   RudolfCarnap,PhilosophicalFoundationsofPhysics.BasicBooks:NewYork.1996.}I^|ی'(#'(# Ќ  6. ' KarelBerka,Measurement:ItsConcepts,TheoriesandProblems.D.ReidelPublishing:   ' Dordrecht,Holland.1983.7. ' DonaldM.MacKay,Information,Mechanism,andMeaning.TheM.I.T.Press:  `  ' Cambridge.1969.8. ' TomStonier,InformationandtheInternalStructureoftheUniverse.SpringerVerlag: 0  ' NewYork.1990.9. ' Ellis,op.cit.,p.155. e 10. ' RudolfL.Carnap,LogicalFoundationsofProbability.UniversityofChicagoPress: E  ' Chicago.1950.0 '  11. ' ISO1000:1992,SIUnitsandRecommendationsfortheUseofTheirMultiplesandof " CertainotherUnits(ISO31:1992,QuantitiesandUnits). '(#'(# 0 '  12. ' ISO6508:1986,MetallicmaterialsHardnesstestRockwelltest(scalesABCDEF $ " GHK). '(#'(# 13. ' OIMLPublicationR54,pHScaleforAqueousSolutions,1981. 'P#% 14. ' C.F.Richter,ElementarySeismology,W.H.Freeman&Co.,SanFrancisco,1958. )0%' 15. ' C.E.ShannonandW.Weaver,MathematicalTheoryofCommunication,Universityof u+')  ' IllinoisPress,Urbana,1949. e,(* Ї0 '  16.0''(#'(#ISO/IECGuide2:1996,StandardizationandrelatedactivitiesGeneralvocabulary.e'(#'(# 0 '  17. ' ISO/IECTR13233:1995,InformationTechnologyInterpretationofAccreditation  @  E RequirementsinISO/IECGuide25AccreditationofInformationTechnologyand  @  5 TelecommunicationsTestingLaboratoriesforSoftwareandProtocolTestingServices. '(#'(# 18. ' ISO/IEC14515,InformationTechnologyProgramminglanguages,theirenvironments,    ' andsystemsoftwareinterfacesPortableOperatingSystemInterface(POSIX)Test ' methodsformeasuringcompliancetoPOSIX.(multiplepartstandard)0 '  19. ' ISO/IEC10641:1993,InformationTechnologyComputergraphicsandimageprocessing  `  Conformancetestingofimplementationsofgraphicsstandards. '(#'(# 20. ' ISO/IECTR10183,InformationTechnologyTextandOfficeSystemsOffice  0   ' DocumentArchitecture(ODA)andinterchangeformatTechnicalReportonISO8613 ' implementationtesting.(multipleparttechnicalreport)21. ' ISO/IEC9646,InformationTechnologyOpenSystemsInterconnectionConformance U  ' testingmethodologyandframework.(multiplepartstandard)0 '  22. ' ISOTR9547:1988,ProgrammingLanguageprocessorsTestmethodsGuidelinesfor % theirdevelopmentandacceptability. '(#'(# 23. ' ECMATR/18,TheMeaningofConformancetoStandards,June1983.  |}I^  }I^7|24  .0 '   ݀ISO/IECTR10034:1990,Guidelinesforthepreparationofconformityclausesin p programinglanguagestandards.}I^7R|ی '(#'(# Ќ  0 '  25. ' ISO/IEC14598,InformationTechnologyEvaluationofsoftwareproduct.(multiplepart @ standard) '(#'(# 0 '   '(#'(# 26.0 ' ISO/IECGuide25:1990,Generalrequirementsforthecompetenceofcalibrationandtesting u! laboratories. '(#'(# 27. ' W.A.Wildhack,DraftProposalforaPolicyonTraceabilityforIBS,NCSLWorkshopon E$"  ' MeasurementAgreement,January1962. 5% # 28. ' JohnA.Simpson,FoundationsofMetrology,JournalofResearchofNBS,January1981. '"% 0 '  29.ErnestL.GarnerandStanleyD.Raspberry,WhatsNewinTraceability,Journalof Testing ($' andEvaluation,November1993.)%('(#'(# 30. ' CharlesD.EhrlichandStanleyD.Raspberry,MetrologicalTimelinessinTraceability, +`'*  ' MeasurementScienceConference,forpresentationataJanuary1997conference. ,P(+ Ї31. ' TheodoreH.Hopp,ComputationalMetrology,ManufacturingReview,December1993. e 32.0 ' ComputerSystemsinMetrology,RecommendedPracticeRP13,NationalConferenceof E StandardsLaboratories,February1996. '(#'(# 33.0 ' TheodoreH.HoppandMarkS.Levenson,PerformanceMeasuresforGeometricFittingin   theNISTAlgorithmTestingandEvaluationProgramforCoordinateMeasurementSystems,   JournalofResearchoftheNationalInstituteofStandardsandTechnology,SeptemberOctober1995. '(#'(# 34.0 ' J.N.Hooker,Needed:AnEmpiricalScienceofAlgorithms,OperationsResearch,March  `  April1994. '(#'(# 35.0 ' TestingTechniques,ANewsletterDevotedtotheTechnologyofSoftwareTesting,  0  SoftwareResearchInc. '(#'(# 0 '  36. ' ISO/IEC14143,InformationTechnology,SoftwareMeasurement.(multiplepartstandard)e'(#'(# 37.0 ' Metrics97,FourthInternationalSymposiumonSoftwareMetrics,(tobeheldNovember E 1997). '(#'(# 38.0 ' MarthaM.Gray,ApplicabilityofMetricstoLargeScaleInfrastructure,(Tobepublished).'(#'(# 39.0 ' SriramSankarandRogerHayes,SpecifyingandTestingSoftwareComponentsusingADL.'(#'(# 40.0 ' ShaneP.McCarron,TheAPIDefinitionLanguageProjectABriefIntroduction,X/Open p CompanyLtd.,July1993. '(#'(# 41.0 ' JosephL.HungateandMarthaM.Gray,AutomatedTestingTechnologiesWorkshop, @ Section3ofConferenceReport,JournalofResearchoftheNationalInstituteofStandardsandTechnology,NovemberDecember1995. '(#'(#   e"   XX X.Xr  X X    !  AnnexB:GlossaryofAbbreviations!! !Ԍ e Ќ.f   .Xr X.f   ADL:0  04(#(#0` 4(#4(#AssertionDefinitionLanguagep` (#` (# AP:0 ' 0'(#'(#04(#(#0` 4(#4(#ApplicationProtocol P` (#` (# ASCII:0 4 0` 4(#4(#AmericanStandardCodeforInformationInterchange 0` (#` (# ATEPCMS:0 ` AlgorithmTestingandEvaluationProgramCoordinateMeasuringSystem.u  ` (#` (# ATS:0  04(#(#0` 4(#4(#AbstractTestSuiteU ` (#` (# DES:0  04(#(#0` 4(#4(#DataEncryptionStandard5 ` (#` (# DSA:0  04(#(#0` 4(#4(#DigitalSignatureAlgorithm` (#` (# DSS:0  04(#(#0` 4(#4(#DigitalSignatureStandard` (#` (# DSSVS:0 4 0` 4(#4(#DigitalSignatureStandardValidationSystemp` (#` (# FDT:0  04(#(#0` 4(#4(#FormalDescriptionTechniqueP` (#` (# IEC:0  04(#(#0` 4(#4(#InternationalElectrotechnicalCommission0` (#` (# IETF:0  04(#(#0` 4(#4(#InternetEngineeringTaskForceu` (#` (# ISO:0  04(#(#0` 4(#4(#InternationalOrganizationforStandardizationU` (#` (# IT:0 ' 0'(#'(#04(#(#0` 4(#4(#InformationTechnology5 ` (#` (# ITI:0 ' 0'(#'(#04(#(#0` 4(#4(#IndustrialTechnologyInstitute"` (#` (# ITL:0  04(#(#0` 4(#4(#InformationTechnologyLaboratory(NIST)#!` (#` (# MEL:0  04(#(#0` 4(#4(#ManufacturingEngineeringLaboratory(NIST)%p!#` (#` (# MITI:0  04(#(#0` 4(#4(#MinistryofInternationalTradeandIndustry'P#%` (#` (# NFPA:0 4 0` 4(#4(#NationalFireProtectionAssociation)0%'` (#` (# NIST:0  04(#(#0` 4(#4(#NationalInstituteofStandardsandTechnologyu+')` (#` (#  e,(* pH:0 ' 0'(#'(#04(#(#0` 4(#4(#Thenegativelogarithmofthehydrogenionconcentrationinsolution.e` (#` (# POSIX:0 4 0` 4(#4(#PortableOperatingSystemInterfaceE` (#` (# RFC:0  04(#(#0` 4(#4(#RequestForComments%` (#` (# SHS:0  04(#(#0` 4(#4(#SecureHashStandard ` (#` (# SI:0 ' 0'(#'(#04(#(#0` 4(#4(#InternationalSystemofUnits(themodernmetricsystem) ` (#` (# STEP:0  04(#(#0` 4(#4(#StandardfortheExchangeofProductModelData ` ` (#` (# TCP/IP:0 4 0` 4(#4(#TransmissionControlProtocol/InternetProtocol@ ` (#` (# TS:0 ' 0'(#'(#04(#(#0` 4(#4(#TechnologyServices(NIST) ` (#` (# VIM:0  04(#(#0` 4(#4(#InternationalVocabularyofBasicandGeneralTermsinMetrologye` (#` (# VLSI:0  04(#(#0` 4(#4(#VeryLargeScaleIntegrationE` (#` (#    Xr X X.XrXX XXrXXXXXXXX XXXX  X X        c1  AnnexC:ExamplesofPresentITMetrologyatNISTc1Z2 B2Ԍ e Ќf   f f f f f f Xr Xf Xr X XXr  ThefollowingexampleshelpedthetaskgrouptosortthroughandunderstandthebasictestingconceptsbehindtheongoingITtestingactivitiesatNIST.Therefore,theyarelistedhereasillustrativeexamplesandnotasarepresentativesamplingorasacompletesummaryofpresentITtestingactivitiesatNIST. Case1:0 4 TestingDES,DSS,SHAimplementations  04(#4(# NISThasdevelopedconformancetestsforFIPS186,DigitalSignatureStandardandFIPS180-1,SecureHashStandard.Thetests,calledtheDSSValidationSystem(DSSVS)aredescribedinDRAFTDigitalSignatureStandard(DSS)andSecureHashStandard(SHS):RequirementsandProcedures.TheSHSisusedforcalculatingamessagedigestthatcanbeusedwiththeDSS.Thecalculationtransformsanymessageoflength264bitstoa160-bitoutput.SincetheoutputsofeachSHAtransformationbecomestheinputsofthenextSHAtransformation,thefinalmessagedigestisafunctionofeachbitofthemessage.Anychangetoamessageintransitwill,withaveryhighprobability,resultinadifferentmessagedigest.UsingblackboxtestmethodstheDSSVStestsforconformancetotheSHSusingthreetests:messagesofvaryinglength,selectedlongmessages,andpseudorandomlygeneratedmessages.FIPS186specifiesaDSAforgeneratingandverifyingdigitalsignaturesondatathathasbeencondensedintoamessagedigestusingtheSHA.ThedigitalsignatureitselfisapairoflargenumbersthatarecomputedondatausingtheDSAandasetofparameterssuchthatitcanbeusedtoverifytheidentityofmessage'sclaimedsenderandtheintegrityofthemessageitself.Signaturegenerationmakesuseoftheprivatekey,whichisalargenumber,togeneratethedigitalsignature.Signatureverificationmakeuseofapublickeythatisrelatedtotheprivatekeyusedtogeneratethesignature.TheDSSVSusesblackboxtestmethodsforconformancetotheDSSinthreeareas:primenumbergeneration,generationofpublic/privatekeypair,andsignaturegeneration/verification. Case2:0 4 AlgorithmTestingandEvaluationProgramforCoordinateMeasuringSystems #  (ATEPCMS) #!4(#4(# NISTisnowofferinganewSpecialTestService,theAlgorithmTestingandEvaluationProgramforCoordinateMeasuringSystems(ATEPCMS).ThisnewSpecialTestServiceisofferedundertheOfficeofMeasurementServicesCalibrationProgram.ATEPCMSevaluatestheperformanceofdataanalysissoftwareusedincoordinatemeasuringsystems(CMSs).Testedsoftwareistreatedasafilterthattransformspointcoordinatedataintofeatureparametersaccordingtoadefinedtransferfunction.NISTevaluatestheaccuracyofthefilterunderconditionstypicalofthosefoundinindustrialpractice.NISTindependently e,(* comparestheoutputofthesoftwareundertesttopredeterminedcorrespondingreferencevalues.NISTusesorthogonaldistanceleastsquaresalgorithmsandsupportsthefollowinggeometrytypes:circle,line,plane,sphere,cylinder,cone,andtorus.IntheSpecialTests,thereportedmeasurementuncertaintyisdeterminedbytheeffectsofcomputationalroundoffandconvergencesettingsusedtogeneratethereferencefits,thepropagationoftheseeffectsthroughthecomparisonalgorithms,andsamplinguncertaintyduetothenumberofdatasetsusedtoperformthetest. Case3:0 4 STEPConformanceTesting  p 4(#4(# STEPisaninternationalstandard(ISO10303)designedtoletcompanieseffectivelyexchangeengineeringinformationbothinternallyandwiththeircustomersandsuppliers.Experiencewithcomplexstandardshasshownthatvendorclaimsofcompliancewithastandardarenotreliable.Forthisreason,theSTEPstandardprovidestestingmethodsandtoolssupporttheobjectivemeasurementofsoftwareimplementationsthatwillultimatelyaidinachievingconfermentandinteroperablesystems.STEPisimplementedthroughaseriesofstandardspecificationscalledApplicationProtocols(APs).ForeachAP,anAbstractTestSuite(ATS)isdevelopedthatcontainstestpurposesgeneratedfromtheAP,verdictcriteriaandinputspecifications.TheATSisrealizedintoanexecutabletestcasebytestinglabsthatwillbeusedtoquantifytheconformanceofanimplementationundertest.NISThasteamedwithIndustrialTechnologyInstitute(ITI)toprovideameansbywhichSTEPproductscanbeobjectivelymeasuredagainstthestandard.Thisisbeingdonebydevelopingasetofvalueaddedsoftwaretoolsforusebyvendorsduringproductdevelopment.ThesetoolsmustbeextensibletoaccommodatetheexpandingseriesofSTEPApplicationProtocols.Thisisbeingaccomplishedbyamodularsystemwithtwoelements:atestsystemwhichintegratesvarioustestingtoolsandadministerstheactualtests,andasetoftoolsforgeneratingatestsuitesforeachAPwhichareusedinthetestingprocess.Thisuniqueapproachoffersmanyadvantagesovertraditionalconformancetesting.ConformancetestingisgenerallychallengedbyU.S.vendorsasnotbeingcosteffective.Underthisapproach,vendorscangainconfidencethattheirproductcansuccessfullypasstesting,theyhaveaccesstothetoolstoimprovethequalityoftheirproducts,andtheygainfromtheexpandedmarketthatuserconfidenceinatestedproductbrings.Thesametoolscanalsobeemployedbyenduserstoassesstheabilityoftheseproductstointeroperateinanindustrialcontext,furtherexpandingthemarketforstandardsbasedproducts.Thesetoolsarebeingusedinthedevelopmentofearlypilotimplementationsofthestandard.