Version 4.5, 04-Mar-1997

Introduction

The 1997 Hub-5E evaluation is part of an ongoing series of periodic evaluations conducted by NIST. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. They are intended to be of interest to all researchers working on the general problem of conversational speech recognition. To this end the evaluation was designed to be simple, to focus on core speech technology issues, to be fully supported, and to be accessible.

The Hub-5E evaluation, conducted in the Spring, complements another related evaluation which is conducted in the Fall. The Fall evaluation focuses primarily on the recognition of multiple languages and on issues related to porting recognition technology to new languages, to system generality, and to language commonalties and universals. This evaluation is dedicated to the advancement of speech recognition technology for English, and specifically for General American English.

The 1997 Hub-5E evaluation will be conducted in March. (Data will go out on March 10th, and results are due back by March 31st.) A follow-up workshop for evaluation participants will be held during mid May (May 13th _ 15th) to discuss research findings. Participation in the evaluation is solicited for all sites that find the task worthy and the evaluation of interest. For more information, and to register a desire to participate in the evaluation, please contact Dr. Alvin Martin at NIST.

The Hub-5E evaluation focuses on the task of transcribing conversational speech into text. This task is posed in the context of conversational telephone speech in General American English. The evaluation is designed to foster research progress, with the goals of:

1. exploring promising new ideas in the recognition of conversational speech,
2. developing advanced technology incorporating these ideas, and
3. measuring the performance of this technology.

The Task

The task is to transcribe conversational speech. The speech to be transcribed is presented as a set of conversations collected over the telephone. Each conversation is represented as a "4-wire" recording, that is with two distinct sides, one from each end of the telephone circuit. Each side is recorded and stored as a standard telephone codec signal (8 kHz sampling, 8-bit mu-law encoding).

Each conversation is represented as a sequence of "turns", where each turn is the period of time when one speaker is speaking. Each successive turn results from a reversal of speaking and listening roles for the conversation participants. The transcription task is to produce the correct transcription for each of the specified turns. The beginning and ending times ¹ of each of these turns will be supplied as side information to the system under test . This information, stored in a single PEM file ² , will determine the test material.

Transcription Conventions

The American Heritage Dictionary (AHD) ³ will serve as the standard reference for word spellings. Words that don't occur in the AHD will be spelled using the most common accepted spelling.


Hesitation sounds, referred to as "non-lexemes", will be represented with a leading "%" character. Although these sounds are transcribed in a variety of ways due to highly variable phonetic quality, they are all considered to be functionally equivalent from a linguistic perspective.

Training Data


All of the SwitchBoard corpus (SwitchBoard-1), and all of the Call_Home English training corpus (the first 100 conversations), may be used for training. Also the 20 Call_Home English conversation that were designated as the Spring 1996 development set, may be used for training. These corpora are available from the LDC. Additional data may also be used for training, provided that the data are made publicly available at the time of reporting results.


Development Data (the DevSet)


The April 1996 Call_Home English EvalSet (20 conversations) will serve as the DevSet. Segment time marks and corresponding SNOR transcriptions for these data will be provided in standard STM ⁴ (segment time marked) format. The file names for these 20 conversations are

1. en_4792	6. en_6047	11. en_6313	16. en_6479
2. en_4801	7. en_6071	12. en_6408	17. en_6521
3. en_4829	8. en_6179	13. en_6447	18. en_6625
4. en_5872	9. en_6265	14. en_6456	19. en_6785
5. en_5888	10. en_6298	15. en_6467	20. en_6825



In addition two smaller subsets of the DevSet will be defined to facilitate exchange of research results between sites. These two subsets are defined so as to maintain reasonable statistical validity while reducing the amount of speech data to be processed:


Subset 1 - to maximize speaker sampling


This subset is the first 30 seconds of speech (from the 5 minute evaluation excerpt) for each conversation side of all conversations. The actual elapsed time will be at least 30 seconds duration, in order to capture 30 seconds of speech. (In some cases the elapsed time may reach the limit of 5 minutes if the speaker mostly listens and doesn't say much.) Segment time marks and corresponding transcriptions for subset 1 will be provided in standard STM format.


Subset 2 - for development of adaptive algorithms requiring more speech


This subset is simply 7 of the 20 DevSet conversations. These conversations are selected to be a representative sampling of the speaker's sex and word error rate. Segment time marks and corresponding transcriptions for subset 2 will be provided in standard STM. The file names of the selected conversations are

1. en_4792	3. en_6179	5. en_6521	7. en_6825
2. en_5872	4. en_6456	6. en_6625

Evaluation Data (the EvalSet)


The EvalSet will comprise 20 conversations each from Call_Home English and Switchboard-2, for a total of 40 conversations. Whole conversations will be supplied, but recognition will be scored only for a 5 minute excerpt chosen from each conversation. Speaker turn segmentation information for these 5 minute excerpts will be supplied to guide the recognition system. This segmentation information will be supplied in NIST's PEM file format.

While the Evaluation data will come from two different sources (namely Call_Home and Switchboard-2), the identity of the source is not to be provided to the system under test. The system must either recognize the speech irrespective of the source or must automatically determine the source from examination of the speech signal.

The Switchboard-2 conversations will be provided after the application of an echo cancellation algorithm. This algorithm is being used to maximize the separation of the two sides of each conversation. The echo cancellation software and documentation are available from Mississippi State's ISIP laboratory ⁵ via anonymous ftp.

Version 2.5 of the echo cancellation software will be used. A perl wrapper script to process SPHERE-headered speech files will also be provided.

Echo cancellation will not be performed on the Call_Home English data. The reason is that the Call_Home data were collected from a telephone system that already had (nonlinear) echo suppression incorporated in it.


The Evaluation


Each system will be evaluated by measuring that system's word error rate (WER). Each system will also be evaluated in terms of its ability to predict recognition errors. System performance will be evaluated over an ensemble of conversations. These conversations will be chosen to represent a statistical sampling of conditions of evaluation interest. For the 1997 evaluation these conditions will include sex, geographical distribution and age.

The Reference Transcription


The reference transcriptions are intended to be as accurate as possible, but there will necessarily be some ambiguous cases and outright errors. In view of the existing high error rates of automatic recognizers on this type of data, it is not considered cost effective to generate multiple independent human transcriptions of the data or to have a formal adjudication procedure following the evaluation submissions.


The reference transcription for each turn will be limited to a single sequence of words. This word sequence will represent the transcriber's best judgment of what the speaker said.




Word fragments will be represented by an initial part of a word with a hyphen at the end. Correct recognition will consist of either ignoring the fragment, or producing a word of which the fragment is an initial part.




The reference transcription will contain no hyphenated words. Each hyphenated word will be separated into its separate constituent words.

The WER Metric


Word error rate is defined as the sum of the number of words in error divided by the number of words in the reference transcription. The words in error are of three types, namely substitution errors, deletion errors, and insertion errors. Identification of these errors results from the process of mapping the words in the reference transcription onto the word in the system output transcription. This mapping is performed using NIST's SCLITE software package ⁶ .

• A substitution error results when the spellings of the reference word and the corresponding system output word differ.
  
• A deletion error results when the reference word has no corresponding system output word.
  
• An insertion error results when a system output word has no corresponding reference word.
  

Scoring will be performed by aligning the system output transcription with the reference transcription and then computing the word error rate. Alignment will be performed independently for each turn, using NIST's SCLITE scoring software. The system output transcription will be processed to match the form of the reference transcription. Hyphenated words will be separated into their separate constituent words.

Some variant spellings of the same word exist in the transcriptions. These words, with or without hyphens, will be mapped onto a single preferred word spelling without hyphens. The set of all such mappings is:




Input	Output
mhm	uhhuh
mmhm	uhhuh
mm-hm	uhhuh
mm-huh	uhhuh
huh-uh	uhuh




For scoring purposes, all hesitation sounds will be considered to be equivalent. Thus all reference transcription words beginning with "%", the hesitation sound flag, along with the conventional set of hesitation sounds, will be mapped to "%hesitation".

The system output transcriptions should use any of the hesitation sounds (without "%") when a hesitation is hypothesized. The set of hesitation sounds for the current evaluation is defined to be:
"uh", "um", "eh", "mm", "hm", "ah", "huh", "ha", "er", "oof", "hee", "ach", "eee" and "ew".

The Confidence Score


Along with each word output by a system, a confidence measure is also required. This confidence measure is the system's estimate of the probability that the word is correct. While this might be merely a constant probability, independent of the input, certain applications and operating conditions may derive significant benefit from a more informative estimate that is sensitive to the input signal. This benefit will be evaluated by computing the mutual information (cross entropy) between the correctness of the system's output word and the confidence measure output for it, normalized by maximum cross entropy:

Results must be submitted to NIST by March 31, 1997 at 7:00 pm. EST using the following steps:
1. system output file creation,
2. directory structure creation,
3. system documentation, including execution times, and system output inclusion
4. transmission protocol to NIST.

Step 1: System output File creation

The time-marked hypothesis words for each test will be placed in a single file, called "<TEST_SET>.ctm". The CTM (Conversation Time-Mark) file format, is a concatenation of time marks for each word in each side of a conversation. Each word token must have a conversation id, channel identifier [A | B], start time, duration, case-insensitive word text, and a confidence score. The start time must be is seconds and relative to the beginning of the waveform file. The conversation id's for this evaluation will be of the form:

CONV_ID::= <SWB2_ID> | <CALLHOME_ID>

where,
SWB2_ID ::= sw_DDDDD (where DDDDD is a five digit conversation code)
CALLHOME_ID ::= en_DDDD (where DDDD is a four digit conversation code)


The file must be sorted by the first three columns: the first and the second in ASCII order, and the third by a numeric order. The UNIX sort command: "sort +0 -1 +1 -2 +2nb -3" will sort the words into appropriate order.

Lines beginning with ';;' are considered comments and are ignored. Blank lines are also ignored.

Included below is an example:

;;
;; Comments follow ';;'
;;
;; The Blank lines are ignored

;;
en_7654 A 11.34 0.2 YES -6.763
en_7654 A 12.00 0.34 YOU -12.384530
en_7654 A 13.30 0.5 CAN 2.806418
en_7654 A 17.50 0.2 AS 0.537922
:
en_7654 B 1.34 0.2 I -6.763
en_7654 B 2.00 0.34 CAN -12.384530
en_7654 B 3.40 0.5 ADD 2.806418
en_7654 B 7.00 0.2 AS 0.537922
:


Step 2: Directory Structure Creation

Create a directory identifying your site ('SITE') from the following list which will serve as the root directory for all your submissions:

• bbn
• dragon
• ibm
• sri
• .
• .
• .

You should place all of your recognition test results in this directory. When scored results are sent back to you and subsequently published, this directory name will be used to identify your organization.

For each test system, create a sub-directory under your 'SITE' directory identifying the system's name or key attribute. The sub-directory name is to consist of a free-form system identification string 'SYSID' chosen by you. Place all files pertaining to the tests run using a particular system in the same SYSID directory.

The following is the BNF directory structure format for Hub-5E hypothesis recognition results:

<SITE>/<SYSID>/<FILES>

where,

SITE ::= bbn | dragon | ibm | sri | . . .
SYSID ::= (short system description ID, preferably <= 8 characters)
FILES ::=
sys-desc.txt
(system description, described below, including reference to paper if applicable)

<TEST_SET>.ctm
(file containing time-marked hypothesis word strings created in Step 1)

where,

TEST_SET ::= english

Step 3: System Documentation, including execution times, and System Output Inclusion

For each test you run, a brief description of the system (the algorithms) used to produce the results must be submitted along with the results, for each system evaluated. (It is permissible for a single site to submit multiple systems for evaluation. In this case, however, the submitting site must identify one system as the "primary" system prior to performing the evaluation.)

The format for the system description is as follows:


SITE/SYSTEM NAME
TEST DESIGNATION


1. Primary Test System Description:
2. Acoustic Training:
3. Grammar Training:
4. Recognition Lexicon Description:
5. Differences for each Contrastive Test: (if any contrastive test were run.)
6. New Conditions for This Evaluation:
7. Execution Time:
   Sites must report the CPU execution time that was required to process the test data, as if the test were run on a single CPU. Sites must also describe the CPU and the amount of memory used.
8. References:

Your system description file should be placed in the 'SYSID' sub-directory which it pertains to and must be called, "sys-desc.txt".

Likewise, the time-marked hypothesis file, created in step 1, should be placed in the 'SYSID' sub-directory which it pertains to, and must be called, "<TEST_SET>.ctm". For this evaluation, the value for <TEST_SET> will be "english".

Step 4: Test Results Submission Protocol

Once you have structured all of your recognition results according to the above format, you can then submit them to NIST. Due to international e-mail file size restrictions, test sites are permitted to submit results to NIST using either email or anonymous ftp. Continental US sites may use either method, but international sites must use the 'ftp' method. The following instructions assume that you are using the UNIX operating system. If you do not have access to UNIX utilities or ftp, please contact NIST to make alternate arrangements.

E-mail method:

First change directory to the directory immediately above the <SITE> directory. Next, type the following:
tar -cvf - ./<SITE> | compress | uuencode <SITE>-<SUBM_ID>.tar.Z | \
mail -s "March 97 Hub-5E test results <SITE>-<SUBM_ID>" \
alvin.martin@nist.gov

where,

<SITE>
is the name of the directory created in Step 2 to identify your site.
<SUBM_ID>
The submission number (e.g. your first submission would be numbered '1', your second, '2', etc.)

Ftp method:

First change directory to the directory immediately above the <SITE> directory. Next, type the following command.
tar -cvf - ./<SITE> | compress > <SITE>-<SUBM_ID>.tar.Z

where,

<SITE> is the name of the directory created in Step 2 to identify your site. <SUBM_ID> The submission number (e.g. your first submission would be numbered '1', your second, '2', etc.)

This command creates a single file containing all of your results. Next, ftp to jaguar.ncsl.nist.gov giving the username 'anonymous' and your e-mail address as the password. After you are logged in, issue the following set of commands, (the prompt will be 'ftp>'):

• ftp> cd /pub/benchmark/mar97_hub-5e
• ftp> binary
• ftp> put <SITE>-<SUBM_ID>.tar.Z
• ftp> quit

You've now submitted your recognition results to NIST. The last thing you need to do is send an e-mail message to Alvin Martin at 'alvin.martin@nist.gov' notifying NIST of your submission. Please include the name of your submission file in the message.

Note:

If you choose to submit your results in multiple shipments, please submit ONLY one set of results for a given test system/condition unless you've made other arrangements with NIST. Otherwise, NIST will programmatically ignore duplicate files.

DevSet Release	Autumn 1996
Commitment Deadline	March 3, 1997
EvalSet Release	March 10, 1997
Results Deadline	March 31, 1997 at 7:00 pm. EST
Workshop	May 13-15, 1997 Maritime Institute of Technology and Graduate Studies Lithicum, Maryland

1. These turn time marks will be specified in seconds (to the nearest millisecond) and will completely encompass the turn. Thus alternate turns will overlap if the speakers talk over each other.
   Back to text.
   
   
2. The PEM ("partitioned evaluation map") file format is given in the SCLITE documentation available through NIST's web page (http://www.nist.gov/itl/div894/894.01/software.htm). Each record contains 5 fields: <filename>, <channel ("A" or "B")>, <speaker ("unknown">, <begin time> and <end time>.
   Back to text.



3. The American Heritage Dictionary of the English Language, Book and CD ROM. Published October 1994 by Houghton Mifflin. ISBN 0395711460.
   Back to text.
   
   
4. STM stands for "segment time marked". The STM file identifies time intervals along with the transcription for those intervals. At the time this document was prepared, the STM file format is documented in NIST's SCLITE scoring software distribution available via NIST's web page (http://www.nist.gov/speech/tools/tools.htm).
   Back to text.
   
   
5. Mississippi State's echo cancellation software is available via ftp access here: fir_echo_canceller
   Back to text.
   
   
6. SCLITE software is available via NIST's web page (http://www.nist.gov/speech/tools/tools.htm).
   Back to text.