NIST Speech Group Website

Information Technology Lab, Information Access Division

NIST: National Institute of Standards and Technology

Multimodal Information Group Home

Benchmark Tests

1999 TREC-8 Spoken Document Retrieval Track.

This page contains information and links to files for the 1999 TREC-8 Spoken Document Retrieval (SDR) Track. Note that it will be updated periodically as new materials and information become available. Members of the SDR email list will be notified of updates.

BACKGROUND

This website is dedicated to the 1999 TREC-8 Spoken Document Retrieval (SDR) Track which implements an evaluation of retrieval of broadcast news excerpts using a combination of automatic speech recognition and information retrieval technologies.

INSTRUCTIONS AND DOCUMENTATION

The 1999 TREC-8 SDR Evaluation Specification Version 1.2 is the core document for the SDR Track and contains detailed information regarding participation, implementation, and schedule. If you intend to participate in the SDR Track, read this document first!

SCHEDULE

The following is the schedule for the SDR Track :

Site registration	ASAP
SPH and NDX available (recognition task begins)	03 May 1999
NDXs, LTTs, SRTs, topics available (R1, B, S retrieval tasks begin)	19 Jul 1999
SRTs due at NIST for scoring/CR sharing (recognition complete)	16 Aug 1999 9am EDT
R1, B, S search results due at NIST	30 Aug 1999 9am EDT
SRTs for CR condition available (CR retrieval tasks begin)	07 Sep 1999
CR search results due at NIST	28 Sep 1999 9am EDT
Relevance judgements released by NIST	05 Oct 1999
Scored Retrieval Results released by NIST	07 Oct 1999
Conference workbook papers to NIST	27 Oct 1999 (estimated)
TREC-8 Conference	17-19 Nov 1999

TRAINING RESOURCES

No particular training collection is specified or provided for this track. However, below are some resources available from the LDC for recognition and retrieval training. Please see the Evaluation Specification for rules governing the use of these materials.

Text Resources

An LDC compilation of text resources is available for recognition and retrieval training.

Speech Resources

1998 Hub-4 training data may be used for SDR training.

IR resources from Previous Tests

Topics and assessments used in previous SDR tests can be used for training.

TEST RESOURCES

Speech recognition task

The Evaluation Specification provides the rules and instructions for implementing the SDR track. The following resources are provided for sites implementing the speech recognition portion of the full SDR task:

Getting started
File Formats
Audio recordings and index files (required for Full SDR task)
Scoring
Contacts

Information retrieval task

The Evaluation Specification provides the rules and instructions for implementing the SDR track. The following resources are provided for sites implementing the retrieval portion of the SDR task:

DATA LICENSING

Note that the Broadcast News recordings and transcriptions used as the spoken document collection in the 1999 TREC-8 SDR Track are licensed through the Linguistic Data Consortium (LDC) and are subject to usage restrictions. Contact the LDC for license agreement information. See the Data Licensing and Costs section in the Evaluation Specification for more details.

CONTACT INFORMATION

If you would like to sign up for the SDR track or any others TREC tracks, please register per the instructions on the TREC website.

If you have questions regarding the SDR data and protocols, contact speech_webmaster[at]nist.gov.

[ Home ]

Page Created: August 17, 2007
Last Updated: November 4, 2008

Multimodal Information Group is part of IAD and ITL
NIST is an agency of the U.S. Department of Commerce