The 1997 Hub-5E Evaluation Plan for Recognition of Conversational Speech over the Telephone, in English

Version 4.5, 04-Mar-1997

 

Introduction

The 1997 Hub-5E evaluation is part of an ongoing series of periodic evaluations conducted by NIST. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. They are intended to be of interest to all researchers working on the general problem of conversational speech recognition. To this end the evaluation was designed to be simple, to focus on core speech technology issues, to be fully supported, and to be accessible.


The Hub-5E evaluation, conducted in the Spring, complements another related evaluation which is conducted in the Fall. The Fall evaluation focuses primarily on the recognition of multiple languages and on issues related to porting recognition technology to new languages, to system generality, and to language commonalties and universals. This evaluation is dedicated to the advancement of speech recognition technology for English, and specifically for General American English.


The 1997 Hub-5E evaluation will be conducted in March. (Data will go out on March 10th, and results are due back by March 31st.) A follow-up workshop for evaluation participants will be held during mid May (May 13th _ 15th) to discuss research findings. Participation in the evaluation is solicited for all sites that find the task worthy and the evaluation of interest. For more information, and to register a desire to participate in the evaluation, please contact Dr. Alvin Martin at NIST.


Technical Objective

The Hub-5E evaluation focuses on the task of transcribing conversational speech into text. This task is posed in the context of conversational telephone speech in General American English. The evaluation is designed to foster research progress, with the goals of:

  1. exploring promising new ideas in the recognition of conversational speech,
  2. developing advanced technology incorporating these ideas, and
  3. measuring the performance of this technology.


The Task

The task is to transcribe conversational speech. The speech to be transcribed is presented as a set of conversations collected over the telephone. Each conversation is represented as a "4-wire" recording, that is with two distinct sides, one from each end of the telephone circuit. Each side is recorded and stored as a standard telephone codec signal (8 kHz sampling, 8-bit mu-law encoding).

Each conversation is represented as a sequence of "turns", where each turn is the period of time when one speaker is speaking. Each successive turn results from a reversal of speaking and listening roles for the conversation participants. The transcription task is to produce the correct transcription for each of the specified turns. The beginning and ending times 1 of each of these turns will be supplied as side information to the system under test . This information, stored in a single PEM file 2 , will determine the test material.


Speech Data

Submission of Results


Schedule


Foot Notes

  1. These turn time marks will be specified in seconds (to the nearest millisecond) and will completely encompass the turn. Thus alternate turns will overlap if the speakers talk over each other.
    Back to text.

  2. The PEM ("partitioned evaluation map") file format is given in the SCLITE documentation available through NIST's web page (http://www.nist.gov/itl/div894/894.01/software.htm). Each record contains 5 fields: <filename>, <channel ("A" or "B")>, <speaker ("unknown">, <begin time> and <end time>.
    Back to text.


  3. The American Heritage Dictionary of the English Language, Book and CD ROM. Published October 1994 by Houghton Mifflin. ISBN 0395711460.
    Back to text.

  4. STM stands for "segment time marked". The STM file identifies time intervals along with the transcription for those intervals. At the time this document was prepared, the STM file format is documented in NIST's SCLITE scoring software distribution available via NIST's web page (http://www.nist.gov/speech/tools/tools.htm).
    Back to text.

  5. Mississippi State's echo cancellation software is available via ftp access here: fir_echo_canceller
    Back to text.

  6. SCLITE software is available via NIST's web page (http://www.nist.gov/speech/tools/tools.htm).
    Back to text.