Elizabeth B. Lennon, Editor
Information Technology Laboratory
National Institute of Standards and Technology
IT security metrics provide a
practical approach to measuring information security. Evaluating security at
the system level, IT security metrics
are tools that facilitate decision making and accountability through
collection, analysis, and reporting of relevant performance data. Based on IT
security performance goals and objectives, IT security metrics are quantifiable,
feasible to measure, and repeatable. They provide relevant trends over time and
are useful in tracking performance and directing resources to initiate
performance improvement actions.
This ITL Bulletin summarizes the recently published NIST Special
Publication (SP) 800-55, Security Metrics Guide for Information Technology
Systems, by Marianne Swanson, Nadya Bartol, John Sabato, Joan Hash, and
Laurie Graffo. NIST SP 800-55 provides guidance for IT managers and security
professionals at all levels, inside and outside of government. The document
describes the development and implementation of an IT security metrics program
and provides examples of metrics based on the critical elements and security
controls and techniques contained in NIST SP 800-26, Security
Self-Assessment Guide for Information Technology Systems. Both documents
are available at http://csrc.nist.gov/publications/nistpubs/index.html.
Regulatory, financial, and organizational reasons drive the requirement to measure IT security performance. For federal agencies, a number of existing laws, rules, and regulations cite IT performance measurement in general, and IT security performance measurement in particular, as a requirement. These laws include the Clinger-Cohen Act, Government Performance and Results Act (GPRA), Government Paperwork Elimination Act (GPEA), and Federal Information Security Management Act (FISMA). In the financial arena, organizations that measure successes and failures of past and current security investments can use metrics to justify and direct future security investments. From an organizational point of view, metrics improve accountability to stakeholders, ensure an appropriate level of mission support, determine IT security program effectiveness, and improve customer confidence.
The IT security metrics
development process consists of two major activities:
The process steps need not be
sequential. Rather, the process provides a framework for thinking about metrics
and facilitates the identification of metrics to be developed for each system.
The type of metric depends on where the system is within its life cycle and the
maturity of the IT system security program. The framework facilitates tailoring
metrics to a specific organization and to the different stakeholder groups in
each organization.
Identify Stakeholders and Interests. Anyone within an organization is
an IT security stakeholder, though some functions have a greater stake than others:
CIO, program manager/system owner, security program manager, resource manager,
and training/human resources personnel. Metrics-related roles and
responsibilities are dispersed throughout an organization. Each stakeholder
needs a set of metrics that provides a view of the organization’s IT security
performance within their needs, for a total of no more than 5-10 metrics per
stakeholder. Many IT security metrics can be created to measure each aspect of
the organization’s IT security. Selecting the most critical elements of the
organization’s IT security program during metrics prioritization will make the
program manageable and successful.
Define Goals and Objectives. IT security performance goals
and objectives are expressed in the form of high-level policies and
requirements in many laws, regulations, policies, and guidance that describe
the dimensions of an effective IT security program. These include the
Clinger-Cohen Act, Presidential Decision Directives, Federal Information
Security Management Act (FISMA), OMB Circular A-130, Appendix III, and NIST
Federal Information Processing Standards (FIPS) and Special Publications. IT
security performance goals identify the desired results of system security
program implementation, while IT security performance objectives enable the
accomplishment of goals. IT security metrics monitor the accomplishment of
goals and objectives.
Review
Current IT Security Policies, Guidance, and Procedures. Organizations must describe control objectives and
techniques that lead to accomplishing performance goals and objectives.
Resources include the organization’s policies and procedures, the Federal
Agency Security Practices Website (http://csrc.nist.gov/fasp),
and NIST SP 800-26, Self-Assessment Guide for IT Systems (http://csrc.nist.gov/publications),
which provides many control objectives and techniques for IT systems.
Review
the System Security Program Implementation. Organizations must ensure that
processes and procedures are in place, existing capabilities are documented,
areas for improvement are noted, existing metrics are identified, and existing
data sources are available that can be used to derive metrics data. These may
be documented in the following sources (and others): system security plans, OMB
Plan of Actions and Milestones reports, the latest GAO and IG findings,
tracking of security-related activities, and risk assessments and penetration
testing results.
Establish
Level of Implementation. The focus of the metrics program depends on the IT security
program maturity within an organization. Most organizations are new to
measuring IT security with performance metrics. They will begin by measuring
the implementation level of established security standards, policies, and
procedures.
Quantify
Program Results. As an organization’s security program implementation increases
and performance data becomes readily available, metrics will focus on program
efficiency and effectiveness. Examples include the timeliness of security
service delivery and operational results experienced by security program
implementation.
Assess
Business/Mission Impact. Business impact can be measured through correlation analysis
once an organization’s processes are self-regenerating and measurement data
gathering is transparent. Examples include business value gained or lost, or an
acceptable loss estimate.
The selection of metrics is critical to the success of the
program. Selected metrics must use data that can realistically be obtained from
existing processes and data repositories, and must measure processes that
already exist and are relatively stable. Use output from standard security activities to quantify IT
security performance. Potential sources include, but are not limited to,
incident handling reports, testing results, network management logs and
records, audit logs, network and system billing records, configuration
management, contingency planning, training records, and certification and
accreditation. (NIST SP 800-55, Appendix A,
provides sample security metrics.) When selecting data sources, keep in mind that IT security metrics
data collection must be as automated and non-intrusive as possible.
The universe of possible metrics, based on existing policies
and procedures, will be quite large. Metrics must be prioritized to ensure that
the final set selected for initial implementation facilitates improvement of
high-priority security control implementation (as defined by an audit or risk
assessment). Based on current
priorities, use no more than 10-20 metrics at a time.
This ensures that an IT security metrics program will be manageable.
Selected metrics should be useful and relevant. Not all data are useful, and collecting irrelevant data could cause stakeholders to lose confidence in the IT security metrics approach. To ensure the acceptable quality of data, standardize data collection methods and data repositories. Define standard data-reporting formats for events throughout the organization, and store reports in a data repository.
Once metrics are selected, obtain
organizational acceptance. Validate metrics with the organization’s
stakeholders at headquarters and in the field. Metrics should also be vetted through
appropriate approval channels. Lastly, phase out old metrics and phase in new
metrics when performance targets are reached or requirements change.
The iterative process of implementation consists of six phases, which, when fully executed, will ensure continuous use of IT security metrics for security control performance monitoring and improvement.
Prepare for Data Collection. Key activities of this first phase include identifying, defining, developing, and selecting the IT security metrics. After the metrics have been identified, specific implementation steps should be defined on how to collect, analyze, and report the metrics. These steps should be documented in the Metrics Program Implementation Plan.
Collect Data and Analyze Results. Phase 2 of the process involves collecting metrics data, consolidating collected data in the prescribed format conducive to data analysis and reporting (e.g., a database or spreadsheet), analyze data and identify gaps between actual and desired performance, and discover areas needing improvement.
Develop Business Case. Phase 4 addresses the budgeting cycle required for obtaining resources required for implementing remediation actions identified in Phase 3. The steps involved in developing a business case are based on industry practices and mandated guidance. Steps include developing a cost model, performing a sensitivity analysis, developing the business case, and preparing a budget submission. The results of the prior three phases will be included in the business case as supporting evidence.
Obtain Resources. Phase 5 includes allocating the budget, prioritizing available resources, and assigning resources.
Apply Corrective Actions. Phase 6 of the process involves implementing corrective actions in technical, management, and operational areas of security controls. After corrective actions are applied, the cycle completes itself and restarts with a subsequent data collection and analysis. Iterative data collection, analysis, and reporting will track progress of corrective actions, measure improvement, and identify areas for further improvement. The iterative nature of the cycle ensures that the progress is monitored and the corrective actions are affecting system security control implementation in an intended way.
In summary, IT security metrics provide a practical approach
to measuring the effectiveness of an IT security program within organizations,
large and small. The results of a robust IT metrics
program provide useful data for organizations to allocate information security
resources and prepare performance-related reports. NIST SP 800-55 and related
documents are available at our website http://csrc.nist.gov/publications.
Disclaimer
Any mention of commercial products or reference to commercial organizations is
for information only; it does not imply recommendation or endorsement by NIST
nor does it imply that the products mentioned are necessarily the best
available for the purpose.