There will not be a dry run evaluation meeting this year. However, all first time TDT participants and past TDT participants developing new systems must run the dry run evaluation. Dry run evaluations have proven invaluable in past evaluations to help debug systems and to ensure all developers are generating TDT-compliant system output.

Participants should only submit system runs for the primary evaluation conditions as per the evaluation plan. The intent of the dry run is not to grade system performance, but rather to familiarize new participants with the evaluation infrastructure and system requirements.

Dry Run Evaluation Resources

The dry run will use the latest versions of the TDT2 and TDT3 corpora which are available from the LDC. The latest release of TDT2 is called "Version 4" (ISBN 1-58563-183-3), and latest release of TDT3 is called "Version 2" (ISBN 1-58563-193-0). (The TDT3 corpora will be used for this Fall's evaluation, so acquiring the data now will save time later).

You can access the data through the LDC's TDT Project web page: http://morph.ldc.upenn.edu/Projects/TDT. The TDT2 release CD contains the topic annotations, however, the TDT3 release CD does not, they are available at the URL ftp://ftp.ldc.upenn.edu/pub/ldc/public_html/tdt/tdt3_em_topic_tbls_v1_0.tgz

Per the evaluation spec., some of the evaluation conditions require system-generated story boundaries. IBM has graciously provided segmentation system output to NIST for the TDT2 and TDT3 corpora. NIST has converted these segmentation system output files into TDT boundaries files for community use. You can find the data at the URL ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/AutoBoundary_20000918.tgz.

The development test corpora are different for each of the tasks. The Story Segmentation, Topic Detection and First Story Detection tasks use the TDT2 corpus and the Topic Tracking and Link Detection evaluations use the TDT3 corpora. Therefore, you must use the correct set of index files for each task.

NIST has prepared index files, (documented in the TDT 2001 Evaluation Plan) for the dry run. They are located at the URLs ftp://jaguar.ncsl.nist.gov/tdt/tdt2001/dryrun2001/dr2001_indexfiles_lnk_trk_v1.tgz, for the Link Detection and Topic Tracking tasks and ftp://jaguar.ncsl.nist.gov/tdt/tdt2001/dryrun2001/dr2001_indexfiles_seg_det_fsd_v1.tgz for the Story Segmentation, Topic Detection and First Story Detection tasks.

To submit results from dry run evaluations, follow the TDT 2001 Dry Run Submission Instructions. The schedule lists September 1, 2001 as the due date to submit results for the dry run evaluation. Unlike the formal evaluation deadlines, this date is flexible if more time is needed.