Information Access Division Banner









Benchmarking Pilot Study of Lung CT

Change Measurement Algorithms & CAD Tools


Organized by the

National Institute of Standards and Technology (NIST)

Report Pilot Results by May 16 July 16, 2008



The development of biomedical imaging change analysis algorithms and computer assisted diagnosis (CAD) tools capable of reliably measuring small changes in volume of lesions found using CT or other imaging modalities would be of great value in assessing response to therapy and in clinical cancer studies. At a 2006 NIST Workshop Imaging as a Biomarker: Standards for Change Measurements in Therapy, a broad group of industry leaders identified the need for improved measurements of the physical quantities underlying biomedical imagery, for better protocols, and for reliable methods for assessing algorithm/CAD performance. NIST has experience in these areas.


With regard to benchmarking, NIST’s Information Technology Laboratory has experience with performance evaluation of software and algorithms for text retrieval, biometrics, motion image quality, face recognition, speech, and text retrieval. The evaluation of biomedical change analysis algorithms presents very real challenges in developing suitable assessment methods. We expect the use of benchmarking to contribute to algorithm improvement.


In comparison with previous benchmarking work at NIST, the distinctive element of the current study is the lack of available ground truth for much of the clinical imagery data. In the present case, lung CT derived from the medical diagnosis of lung cancer and the assessment of the response to therapy is available, but in many cases the size and the change in size of lung nodules and lesions are not reliably known. The Reference Image Database to Evaluate Response (or RIDER) White Paper discusses approaches to estimating the uncertainty associated with the measurement of change. In the Pilot study we will examine such methods and their applicability to the algorithms and CAD tools submitted to Biochange 2008.


While the Pilot study contains a small number of CT pairs, it is our expectation that even a small data set will provide insight into measuring performance of multiple algorithms and CAD tools for volume change in lung CT. Because we lack access to reliable ground truth, the study will consider the effectiveness of various performance measures and methodologies near the zero-change point. The study will include cases with no clinical change, in order to estimate the local variance and bias of each change detector. The Pilot should provide guidance in conducting future larger studies.



Develop methodology & performance measures to validate change analysis using publicly available lung CT data.


Benchmarking elements

·        Participants provide algorithms or CAD tools that measure change in lung nodules.

·        NIST gathers pairs of CT scans from various sources: RIDER (Reference Image Database to Evaluate Response) clinical data from NCI, phantom collections from FDA, and possibly other clinical data. The Pilot includes pairs of scans with “No Change” conditions.

·        With industry participation, NIST defines evaluation methodology and provides code samples for data access. The design is intended to assure low barriers to entry. The Experiment Description is available in XML format and also as a spreadsheet. The evaluation has two stages, Training and Validation



How to Participate in Biochange 2008 Pilot

·        Download and read the Biochange2008 Protocol and submit a Statement of Interest to NIST.

·        Download the CT scan data from the NIST FTP site and from the National Cancer Institute Archive (NCIA).

·        Train your algorithm or clinical CAD tool using the Training Experiment Description and Training Data.

·        Run your change analysis algorithm or commercial CAD tool in your lab on the Validation data set.

·        Report your change results in one of the required formats and provide the signed Participation Agreement to NIST by May 16 July 16, 2008. Optionally, Participants may submit their Training stage data by April 21 for preliminary analysis and discussion at the Lung Cancer Workshop.

·        NIST will analyze the results as described in the Biochange2008 Project Plan. NIST will provide Participants with individual analysis of their results. We will publish the results of the evaluation, without identifying individual scores by Participant.



Dr. Charles Fenimore, National Institute of Standards and Technology (NIST), phone 301-975-2428.

Privacy Policy/Security Notice/ Accessibility Statement


NIST is an agency of the
U.S. Department of Commerce
Site created on March 21, 2008
Last updated: Friday, May 1, 2008