Overview

• Data Selection for Broadcast News CSR Evaluations, William M. Fisher, Walter S. Liggett, Audrey Le, Jonathan G. Fiscus, David S. Pallett: NIST.
  Composition of the 1997 Hub-4 broadcast news test set is discussed. The composition is based on concurrent selection of a statistically-equivalent test set for a future evaluation, adjustment of the set to match the training data, and other considerations. This paper discusses both the principles involved and the specific algorithms used.
  50 K 375 K
• 1997 Broadcast News Benchmark Test Results: English and Non-English, David S. Pallett, Jonathan G. Fiscus, Alvin Martin, Mark A. Przybocki: NIST.
  This paper documents use of Broadcast News test materials in DARPA-sponsored Automatic Speech Recognition (ASR) Benchmark Tests conducted late in 1997. This year's English-language tests differed from last year's in that statistical selection procedures were used in selecting a three-hour test set comprised of 158 story-length segments, in contrast to last year's two-hour test set which was comprised of 4 half-hour segments. The increased number of segments is intended to provide a statistically-equivalent reserve test set for a future evaluation.
  124 K 238 K
• Insights from the Broadcast News Benchmark Tests, Walter S. Liggett, William M. Fisher: NIST.
  The broadcast news benchmark tests have potential as a source of ideas for improving continuous speech recognition systems. This paper presents a data analysis method for uncovering such ideas and applies the method to the 1996 and 1997 DARPA CSR Hub-4 results. The method is based on a latent variables model instead of a more familiar regression model. The method identifies certain portions of the test material that result in wide performance differences among systems. Such portions, because some systems could handle them and others could not, are worth thinking about in terms of what system features lead to the performance differences. Identification of specific system differences that are responsible for performance differences may lead to system improvements.
  59 K 551 K

Hub-4 Non-English Systems

• The Development of the 1997 CMU Spanish Broadcast News Transcription System, Juan M. Huerta, Eric Thayer, Mosur Ravishankar, Richard M. Stern: CMU.
  This paper describes the 1997 CMU DARPA Hub 4 Spanish Broadcast News Transcription system. The system we present is based on the CMU Sphinx-III recognizer and uses a single set of acoustic and language models. The decoding process is performed in two passes: a Viterbi search and a directed acyclic graph (DAG) search are performed on the first recognition stage. The second recognition stage is similar to the first stage except that it is performed using models adapted through maximum-likelihood linear regression (MLLR). We describe the issues relating to the design and development of the acoustic models, language models and lexicon. Developmental results and an analysis are presented.
  39 K 73 K

English Systems I

• The 1997 BBN Byblos System Applied to Broadcast News Transcription, Francis Kubala, Jason Davenport, Hubert Jin, Daben Liu, Tim Leek, Spyros Matsoukas, David Miller, Long Nguyen, Fred Richardson, Richard Schwartz, John Makhoul: BBN.
  In this paper, we describe the BBN Byblos system used for the 1997 DARPA Hub-4 Broadcast News evaluation and discuss numerous improvements made to the system in 1997. We focused our effort entirely upon the two conditions containing studio-quality uncorrupted speech from native speakers, the so-called F0 (prepared speech) and F1 (spontaneous speech) conditions. In particular, we did not bother to create a separate acoustic model for narrow-band telephone speech. Our overall 1997 Hub-4 evaluation result was 20.4% WER, but our error rate on the F0/F1 conditions was only 14%. We ran regression tests on development test data that show we reduced word error rate by 22-30% on the F0/F1 conditions compared to our 1996 system. Sizable gains were achieved on all the other conditions as well, even though no extra effort was spent toward improving them. Brief summaries of three related efforts are also given covering the use of Byblos for Spanish news transcription, near real-time transcription, and automatic extraction of named entities from broadcast news.
  134 K 131 K
• The 1997 HTK Broadcast News Transcription System, P.C. Woodland, T. Hain, S.E. Johnson, T.R. Niesler, A. Tuerk, E.W.D. Whittaker, S.J. Young: Cambridge University.
  This paper presents the recent development of the HTK broadcast news transcription system. Previously we have used data type specific modelling based on adapted Wall Street Journal trained HMMs. However, we are now using data for which no manual pre-classification or segmentation is available and therefore automatic techniques are required and compatible acoustic modelling strategies must be adopted. A number of recognition experiments are presented that compare data-type specific and non-specific models; differing amounts of training data; the use of gender-dependent modelling and the effects of automatic data-type classification. Based on these experiments, the HTK system for the 1997 broadcast news evaluation was designed. A detailed description of this system is given which includes a class-based language modelling component. The complete system yields an overall word error rate of 22.0% on the 1996 unpartitioned broadcast news development test data and just 15.8% on the 1997 evaluation test set.
  58 K 85 K
• The 1997 ABBOTT System for the Transcription of Broadcast News, G.D. Cook, A.J. Robinson: Cambridge University.
  This paper describes the development of a connectionist-hidden Markov model (HMM) system for the 1997 DARPA Hub-4E CSR evaluations. We describe both system development and the enhancements designed to improve performance on broadcast news data. Both multilayer perceptron (MLP) and recurrent neural network acoustic models have been investigated. We assess the effect of using gender-dependent acoustic models, and the impact on performance of varying both the number of parameters and the amount of training data used for acoustic modelling. The use of context-dependent phone models is described, and the effect of the number of context classes is investigated. We also describe a method for incorporating syllable boundary information during search. Results are reported on the 1997 DARPA Hub-4E development test set. We then describe the CU-CON evaluation system and report results on the 1997 Hub-4E test set.
  69 K 101 K
• The 1997 CMU Sphinx-3 English Broadcast News Transcription System, K. Seymore, Stanley Chen, S. Doh, M. Eskenazi, E. Gouvea, B. Raj, Mosur Ravishankar, Ronald Rosenfeld, M.A. Siegler, Richard M. Stern, Eric Thayer: CMU.
  This paper describes the 1997 Hub-4 Broadcast News Sphinx-3 speech recognition system. This year's system includes full-bandwidth acoustic models trained on Broadcast News and Wall Street Journal acoustic training data, an expanded vocabulary, and a 4-gram language model for N-best list rescoring. The system structure, acoustic and language models, and adaptation components are described in detail, and results are presented to establish the contributions of multiple recognition passes. Additionally, experimental results are presented for several different acoustic and language model configurations.
  46 K 66 K
• Dragon Systems' 1997 Broadcast News Transcription System, Steven Wegmann, Francesco Scattone, Ira Carp, Larry Gillick, Robert Roth, Jonathan P. Yamron: Dragon Systems.
  This paper describes Dragon Systems' 1997 Broadcast News evaluation system.
  45 K 454 K

English Systems II

• IBM's LVCSR System for Transcription of Broadcast News Used in the 1997 Hub-4 English Evaluation, Scott Shaobing Chen, M.J.F. Gales, P.S. Gopalakrishnan, R.A. Gopinath, H. Printz, D. Kanevsky, P. Olsen, L. Polymenakos: IBM.
  This paper describes IBM's large vocabulary continuous speech recognition (LVCSR) system used in the 1997 Hub4 English evaluation. It focusses on extensions and improvements to the system used in the 1996 evaluation. The recognizer uses an additional 35 hours of training data over the one used in the 1996 Hub4 evaluation. It includes a number of new features: optimal feature space for acoustic modeling (in training and/or testing), filler-word modeling, Bayesian Information Criterion (BIC) based segmentation and segment clustering, an improved implementation of iterative MLLR, variance adaptation, and 4-gram language models. Results using the 1996 and 1997 DARPA Hub4 evaluation data sets are presented.
  112 K 107 K
• The LIMSI 1997 Hub-4E Transcription System, J.L. Gauvain, Lori Lamel, G. Adda: LIMSI.
  In this paper we report on the LIMSI system used in the Nov '97 Hub-4E benchmark test on transcription of American English broadcast news shows. There are two main differences from the LIMSI system developed for the Nov '96 evaluation. The first concerns the preprocessing stages for partitioning the data, and the second concerns a reduction in the number of acoustic model sets used to deal with the various acoustic signal characteristics. The LIMSI system for the November 1997 Hub-4E evaluation is a continuous mixture density, tied-state cross-word context-dependent HMM system. The acoustic models were trained on the 1995 and 1996 official Hub-4E training data containing about 80 hours of transcribed speech material. The 65K word trigram language models are trained on 155 million words of newspaper texts and 132 million words of broadcast news transcriptions. The test data is segmented and labeled using Gaussian mixture models, and non-speech segments are rejected. The speech segments are classified as telephone or wide-band, and according to gender. Decoding is carried out in three passes, with a final pass incorporating cluster-based test-set MLLR adaptation. The overall word transcription error of the Nov '97 unpartitioned evaluation test data was 18.5%.
  41 K 73 K
• Development of the CSLU LVCSR: The 1997 DARPA Hub-4 Evaluation System, Yonghong Yan, Xintian Wu, Johan Schalkwyk, Ron Cole: Oregon Graduate Institute.
  This paper presents the CSLU Broadcast News transcription system used in the DARPA 1997 evaluation. The system was built using the software developed for the CSLU LVCSR project started in January 1997. This 25K-word vocabulary system used continuous HMMs for acoustic modeling and the standard backoff trigram as the language model. The search used a single pass decoder with MLLR based adaptation technique. Although on the standard DARPA 20K WSJ task our system obtained 11.6% word error, the 39% error on this year's evaluation suggests there are still many aspects needed to be learned by a newcomer like us.
  118 K 235 K
• Automatic Transcription of English Broadcast News, Peter Beyerlein, Xavier Aubert, Reinhold Haeb-Umbach, Dietrich Klakow, Meinhard Ullrich, Andreas Wendemuth, Patricia Wilcox: Philips.
  In this paper the Philips Broadcast News transcription system is described. The Broadcast News task aims at the recognition of "found" speech in radio and television broadcasts without any additional side information (e.g. speaking style, background conditions). The system was derived from the Philips continuous mixture density crossword HMM system, using MFCC features and Laplacian densities. A segmentation was performed to obtain sentence-like partitions of the broadcasts. Using data-driven clustering, the obtained segments were grouped into clusters with similar acoustic conditions for adaptation purposes. Gender independent word-internal and crossword triphone models were trained on 70 hours of the HUB4 training data. No focus condition specific training was applied. Channel and speaker normalization was done by mean and variance normalization as well as VTN and MLLR. The transcription was produced by an adaptive multiple pass decoder starting with phrase-bigram decoding using word-internal triphones and finishing with a phrase-trigram decoding using MLLR-adapted crossword models.
  142 K 95 K
• Development of SRI's 1997 Broadcast News Transcription System, Ananth Sankar, Fuliang Weng, Ze'ev Rivlin, Andreas Stolcke, Ramana Rao Gadde: SRI.
  This paper describes SRI's 1997 broadcast news transcription system used for the 1997 DARPA H4 evaluations. Our system had several novel components. These include automatic segmentation of entire broadcast shows, word-internal and crossword acoustic models robustly estimated with a new Gaussian Merging-Splitting (GMS) algorithm, the use of trigram language models (LMs) in lattices instead of for rescoring N-best lists, and an LM pruning algorithm that allows efficient representation of high-order (like 4- or 5-gram) LMs. We briefly describe these features and give comparative experimental results. We achieved a 18.7% relative improvement in performance on our 1996 H4 partitioned evaluation (PE) development test set as compared to our 1996 H4 PE evaluation system.
  47 K 64 K

Acoustic Modeling

• Experiments with a Gaussian Merging-Splitting Algorithm for HMM Training for Speech Recognition, Ananth Sankar: SRI.
  It is well known that the expectation-maximization (EM) algorithm, commonly used to estimate hidden Markov model (HMM) parameters for speech recognition, is sensitive to the initial model parameter values, making appropriate parameter initialization important. We investigate the use of iterative Gaussian splitting and EM training to initialize the desired number of Gaussians per HMM state (or state cluster). We then study merging of Gaussians which contain little training data as an approach to robust parameter estimation. Finally Gaussian merging and splitting is combined to form the Gaussian Merging-Splitting (GMS) algorithm. Detailed experimental studies show that Gaussian splitting gives similar performance to our previous training algorithm, even though the two algorithms give very different parameter values. The robust parameter estimation from Gaussian merging results in better performance than our old algorithm for speaker-independent models that have a large number of parameters relative to the amount of training data. In addition, in one experiment, speaker adaptation gave a 7% relative improvement in word error rate over the SI models when the SI models were trained with Gaussian splitting alone, as compared to a 15.5% improvement when the SI models were trained with both splitting and merging, even though the unadapted SI models gave similar performance. Finally, experiments with the GMS algorithm show that, for a given number of Gaussian parameters, better performance is achieved by reducing the number of HMM state clusters and increasing the number of Gaussians per state cluster.
  83 K 107 K
• Fast Robust Inverse Transform SAT and Multi-stage Adaptation, Hubert Jin, Spyros Matsoukas, Richard Schwartz, Francis Kubala: BBN.
  We present a new method of Speaker Adapted Training (SAT) that is more robust, faster, and results in lower error rate than the previous methods. The method, called Inverse Transform SAT (ITSAT) is based on removing the differences between speakers before training, rather than modeling the differences during training. We develop several methods to avoid the problems associated with inverting the transformation. In one method, we interpolate the transformation matrix with an identity or diagonal transformation. We also apply constraints to the matrix to avoid estimation problems. We show that by using many diagonal-only transformation matrices with constraints we can achieve performance that is comparable to that of the original SAT method at a fraction of the cost. In addition, we describe a multi-stage approach to Maximum Likelihood Linear Regression (MLLR) unsupervised adaptation and we show that is more effective than a single stage regular MMLR adaptation. As a final stage, we adapt the resulting model at a finer resolution, using Maximum A Posteriori (MAP) adaptation. With the combination of all the above adaptation methods we obtain a 13.6% overall reduction in WER relative to Speaker Independent (SI) training and decoding.
  156 K 200 K
• Constrained Maximum Likelihood Modeling with Gaussian Distributions, R.A. Gopinath: IBM.
  Maximum Likelihood (ML) modeling of multiclass data using gaussian distributions for classification often suffers from the following problems: a) data insufficiency implying overtrained or unreliable models b) large storage requirement c) large computational requirement and/or d) ML is not discriminating between classes. Sharing parameters across classes (or constraining the parameters) clearly tends to alleviate the first three problems. It this paper we show that in some cases it can also lead to better discrimination (as evidenced by reduced misclassification error). The parameters considered are the means and variances of the gaussians and linear transformations of the feature space (or equivalently the gaussian means). Some forms of sharing (either explicit or implicit via constraints) on the parameters are shown to lead to Linear Discrimination Analysis (a well-known result) while others (like diagonal, block-diagonal and factor analyzed covariances) are shown to lead to optimal feature spaces. The key idea is that in constrained ML modeling one may be able to better model the data after it is linearly transformed, perhaps in a class dependent fashion. If the constrains are invariant to linear transformations (ILT), then, the original feature space is as good as any to model the data. Results using optimal feature spaces for diagonal covariances is shown using the speech recognition problem as an example.
  132 K 129 K
• Acoustic Modeling in the Philips Hub-4 Continuous Speech Recognition System, Reinhold Haeb-Umbach, Xavier Aubert, Peter Beyerlein, Dietrich Klakow, Meinhard Ullrich, Andreas Wendemuth, Patricia Wilcox: Philips.
  In this paper we describe some characteristics of the acoustic modeling used in the Philips continuous-speech recognition system for the DARPA Hub-4 1997 evaluation, which are related to robustness issues. We aimed at a conceptually simple system: We trained two model sets on 70 hours of the Hub-4 training data, one for within-word and one for cross-word decoding. These model sets were used for both genders and all environmental conditions. In order to be able to do so, channel normalization (mean, variance normalization) and speaker normalization (vocal tract length normalization, realized by an appropriate shift of the center frequencies of the mel filter bank) have been applied, as well as adaptation techniques. MLLR-based unsupervised batch adaptation on clusters of segments was conducted both after a first within-word decoding and a cross-word decoding pass. The training strategy and the effects of the various normalization and adaptation techniques will be discussed in the paper.
  96 K 75 K

Broadcast News Recognition

• Improvements in Spontaneous Speech Recognition, Daben Liu, Long Nguyen, Spyros Matsoukas, Jason Davenport, Francis Kubala, Richard Schwartz: BBN.
  Recognition of spontaneous conversational speech is made difficult by the severe pronunciation variants, mostly due to accents, co-articulation and speaking mode. Most recognition systems only take some of these phenomena into account. The result is that even though they get high accuracy on prepared or carefully read speech, the performance on spontaneous speech is poor. In this paper, we summarize some of the most profound features in spontaneous speech, such as non-speech events, co-articulation, and deleted phonemes. This paper will show that just by introducing some simple but efficient solutions into our BYBLOS recognition system, we can significantly improve the performance of recognizing spontaneous speech. The experimental results have shown that for Hub4 96 development test set, the WER for the spontaneous speech (F1) was reduced by about 35% and for the Hub4 96 evaluation set, the improvement was about 25%.
  39 K 333 K
• Speaker, Environment and Channel Change Detection and Clustering via the Bayesian Information Criterion, Scott Shaobing Chen, P.S. Gopalakrishnan: IBM.
  In this paper, we are interested in detecting changes in speaker identity, environmental condition and channel condition; we call this the problem of acoustic change detection. The input audio stream can be modeled as a Gaussian process in the cepstral space. We present a maximum likelihood approach to detect turns of a Gaussian process; the decision of a turn is based on the Bayesian Information Criterion (BIC), a model selection criterion well-known in the statistics literature. The BIC criterion can also be applied as a termination criterion in hierarchical methods for clustering of audio segments: two nodes can be merged only if the merging increases the BIC value. Our experiments on the Hub4 1996 and 1997 evaluation data show that our segmentation algorithm can successfully detect acoustic changes; our clustering algorithm can produce clusters with high purity, leading to improvements in accuracy through unsupervised adaptation as much as the ideal clustering by the true speaker identities.
  212 K 313 K
• Segment Generation and Clustering in the HTK Broadcast News Transcription System, T. Hain, S.E. Johnson, A. Tuerk, P.C. Woodland, S.J. Young: Cambridge University.
  This paper describes the segmentation, gender detection and segment clustering scheme used in the 1997 HTK broadcast news evaluation system and presents results on both the unpartitioned 1996 development and the 1997 evaluation sets. The stages of our approach are presented, namely classification, segmentation and gender detection, gender relabelling, and clustering of speech segments. The evaluation audio stream has been segmented according to audio type with a frame accuracy up to 95%. Further segmentation and gender labelling gave up to 99% frame accuracy with 127 multiple speaker segments. Experiments using two different segmentation approaches and three clustering schemes are presented.
  43 K 76 K
• New Developments in Lattice-Based Search Strategies in SRI's H4 System, Fuliang Weng, Andreas Stolcke, Ananth Sankar: SRI.
  We describe new developments in SRI's lattice-based progressive search strategy. These developments include the implementation of a new bigram lattice algorithm, lattice optimization techniques, and expansion of bigram lattices to trigram lattices. The new bigram lattice generation algorithm is based on generation of backtrace entries using a word-dependent N-best list decoding pass, followed by lattice generation from the backtrace entries. We present an algorithm to reduce the size of the bigram lattices while maintaining all valid paths. This algorithm is shown to reduce the size of the lattice by about 50%, allowing easier processing in later stages such as expansion to trigram lattices. We describe two algorithms to expand bigram lattices to trigram lattices. The first is a conventional method, while the second is a novel approach that results in compact trigram lattices that were found to be a factor of six smaller than lattices created with the conventional approach. Decoding with the new trigram lattices gave a 5% improvement in word error rate as compared to our previous search strategy which used trigram LMs to rescore N-best lists.
  65 K 109 K
• Japanese Broadcast News Transcription and Topic Detection, Sadaoki Furui, Koh'ichi Takagi, Atsushi Iwasaki, Katsutoshi Ohtsuki, Tatsuo Matsuoka, Shoichi Matsunaga: Tokyo Institute of Technology and NTT Human Interface Labs.
  This paper reports recent advances in Japanese broadcast news transcription and automatic topic detection from the transcribed news speech. To cope with the variability of the readings for each word, a new method for incorporating reading probability of each word in the decoding process is proposed. As a realistic solution to the new-word problem, a new method is proposed, in which new words are manually registered and OOV language model is applied to the new word. To detect topic words for news speech, two methods are proposed; one uses a relevance measure between each word in the news and each word in the topic word set, and the other uses a significance measure for each word based on the frequency ratio.
  59 K 360 K
• Transcribing Multilingual Broadcast News Using Hypothesis Driven Lexical Adaptation, Petra Geutner, Michael Finke, Peter Scheytt, Alex Waibel, Howard Wactlar: CMU.
  This paper describes first results of our DARPA-sponsored efforts toward recognizing and browsing foreign language, more specifically, Serbo-Croatian broadcast news. For Serbo-Croatian as well as many other than the most common well studied languages, the problems of broadcast quality recognition are complicated by 1) the lack of available acoustic and language data, and 2) the excessive vocabulary growth in heavily inflected languages that lead to unacceptable OOV-rates. We present a Serbo-Croatian large vocabulary system that achieves a 74% recognition rate, despite limited training data. Our system achieves this rate by a multipass strategy that dynamically adapts the recognition dictionary to the speech segment to be recognized by generating morphological variations (Hypothesis Driven Lexical Adaptation). We will outline the bootstrapping and training process of the Janus Recognition Toolkit (JanusRTk) based broadcast news recognition engine: data collection, segmentation and labeling of the data according to different acoustic conditions, dictionary design, language modeling and training. The Hypothesis Driven Lexical Adaptation (HDLA) approach has been tested both on Serbo-Croatian and German news data and has achieved considerable recognition improvements. OOV-rates were reduced by 35-45%; on the Serbo-Croatian broadcast news data from 8.7% to 4.8% thereby also decreasing word error rate from 29.5% to 26%.
  111 K 430 K

Information Retrieval & Spoken Document Retrieval

• Information Extraction and Speech Recognition, Ralph Grishman: New York University.
  Information extraction is the process of analyzing natural language and collecting information about specified types of entities, relationships, or events. This paper provides an overview of a range of information extraction tasks, and briefly describes the structure of systems for performing these tasks. The paper then considers the impact of speech recognition errors on such systems, and describes some experiments which indicate how the effect of such errors may be reduced. Finally, the paper speculates on avenues of research which may enhance the performance of extraction systems on speech transcripts.
  56 K 534 K
• TREC-6 1997 Spoken Document Retrieval Task Overview and Results, John S. Garofolo, Ellen M. Voorhees, Vincent M. Stanford, Karen Sparck Jones: NIST and Cambridge University.
  This paper describes the 1997 TREC-6 Spoken Document Retrieval (SDR) Track which implemented a first evaluation of retrieval of broadcast news excerpts using a combination of automatic speech recognition and information retrieval technologies. The motivations behind the SDR Track and background regarding its development and implementation are discussed. The SDR evaluation collection and topics are described and summaries and analyses of the results of the track are presented. Finally, plans for future SDR tracks are described.
  143 K 268 K
• Experiments in Information Retrieval from Spoken Documents, A.G. Hauptmann, R.E. Jones, K. Seymore, S.T. Slattery, M.J. Witbrock, M.A. Siegler: CMU.
  This paper describes the experiments performed as part of the TREC-97 Spoken Document Retrieval Track. The task was to pick the correct document from 35 hours of recognized speech documents, based on a text query describing exactly one document. Among the experiments we described here are: Vocabulary size experiments to assess the effect of words missing from the speech recognition vocabulary; experiments with speech recognition using a stemmed language model; using confidence annotations that estimate of the correctness of each recognized word; using multiple hypotheses from the recognizer. And finally we also measured the effects of corpus size on the SDR task. Despite fairly high word error rates, information retrieval performance was only slightly degraded for speech recognizer transcribed documents.
  76 K 593 K
• An Overview of the AT&T Spoken Document Retrieval System, John Choi, Don Hindle, Julia Hirschberg, Ivan Magrin-Chagnolleau, Christine Nakatani, Fernando Pereira, Amit Singhal, Steve Whittaker: AT&T Labs Research.
  We present an overview of a spoken document retrieval system developed at AT&T Labs-Research for the HUB4 Broadcast News corpus. This overview includes a description of the intonational phrase boundary detection, classification, speech recognition, information retrieval and user interface components of the system, along with updated system assessments based on the 49-query task defined for the TREC-6 SDR track. Results from a comparative ranking study, based on queries taken from AP Newswire headlines from the same time period that the Broadcast News corpus was recorded, are presented. For the AP task, retrieval accuracy is assessed by comparing the documents retrieved from ASR generated transcriptions with those from human generated transcriptions.
  69 K 220 K

Topic Detection and Tracking

• Topic Detection & Tracking (TDT): Overview & Perspective, Charles L. Wayne: NSA.
  Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data (e.g., newswire and broadcast news). Work on TDT began about a year ago, is now expanding, and will be a regular feature at future Broadcast News workshops.
  21 K 178 K
• Topic Detection and Tracking Pilot Study: Final Report, James Allan, Jaime Carbonell, George Doddington, Jonathan P. Yamron, Yiming Yang: Univ. of Mass., CMU, DARPA, and Dragon Systems.
  Topic Detection and Tracking (TDT) is a DARPA-sponsored initiative to investigate the state of the art in finding and following new events in a stream of broadcast news stories. The TDT problem consists of three major tasks: (1) segmenting a stream of data, especially recognized speech, into distinct stories; (2) identifying those news stories that are the first to discuss a new event occurring in the news; and (3) given a small number of sample news stories about an event, finding all following stories in the stream. The TDT Pilot Study ran from September 1996 through October 1997. The primary participants were DARPA, Carnegie Mellon University, Dragon Systems, and the University of Massachusetts at Amherst. This report summarizes the findings of the pilot study.
  263 K 608 K
• Dragon Systems' Automatic Transcription of New TDT Corpus, Larry Gillick, Yoshiko Ito, Linda Manganaro, Michael Newman, Francesco Scattone, Steven Wegmann, Jonathan P. Yamron, Puming Zhan: Dragon Systems.
  Dragon Systems has agreed to provide automatically generated transcripts for around 1000 hours of Broadcast News, annotated with word-level time-markings and confidence estimates. Pilot transcripts of about 30 hours of data will be available sometime in February, with the project completed by mid-July. In this paper, we describe how we took our 1997 Hub4 evaluation system which ran at around 140 times real time, and modified it to run around 6 times faster with virtually no increase in error rate. We describe in detail the confidence algorithm, and show how it is a good predictor of which words were recognized correctly.
  38 K 307 K
• The Topic Detection and Tracking Phase 2 (TDT2) Evaluation Plan, George Doddington: SRI.
  The purpose of the TDT2 project is to advance the state of the art in Topic Detection and Tracking. The general TDT task domain is to be explored and technology is to be developed in the context of an evaluation-driven paradigm, in which key technical challenges are defined and supported by formal evaluations. This document presents these formal task definitions and the performance measures and procedures to be used to direct the research and evaluate technical capabilities and research progress.
  41 K 459 K

Voice Mail Transcription

• Transcription of New Speaking Styles-Voicemail, M. Padmanabhan, B. Ramabhadran, E. Eide, G. Ramaswamy, L.R. Bahl, P.S. Gopalakrishnan, Salim Roukos: IBM.
  We reported transcription word error rates on a new testbed representing telephone-bandwidth spontaneous speech i.e., the task of voicemail transcription. We described several techniques that were used to construct the acoustic models including (i) the use of compound words and linguistically derived phonological rules to model co-articulation effects that occur in spontaneous speech (ii) a new model-complexity adaptation technique that uses a discriminant measure to allocate gaussians to the mixtures modelling allophones. We also investigated the efficacy of some well known acoustic adaptation techniques on this task. We also described experiments related to building language models using the limited amount of training data available in this domain. We also report experimental results on two different sized training sets to show the effect of increasing the training data on different (acoustic and linguistic) components of the recognizer.
  132 K 137 K
• Issues Involved in Voicemail Data Collection, M. Padmanabhan, G. Ramaswamy, B. Ramabhadran, P.S. Gopalakrishnan, C. Dunn: IBM.
  We describe the collection of database of Voicemail messages. Voicemail represents a very large volume of real-world speech data, which is however not particularly well represented in existing databases. We describe the various issues related to the collection of the data, its transcription and various statistics associated with voicemail messages.
  80 K 122 K

World-Wide Speech Research

• An Overview of EU Programs Related to Conversational/Interactive Systems, Joseph Mariani, Lori Lamel: LIMSI.
  Since 1984, the European Commission has supported projects related to spoken language processing within various programs (particularly ESPRIT, ESPRIT-BRA and TELEMATICS-LANGUAGE ENGINEERING). The projects address various aspects of spoken language processing including basic research, technology development, infrastructure (language resources and evaluation) and applications in different areas (telecommunications, language training, office systems, games, cars, information access and retrieval, banking, aids to the handicapped...). The next program, the 5th Framework Program is now being prepared, and should unite the activities in the area of Human Language Technology and its components in a single sector. The three sub-areas which have been identified are "Multilinguality", "Natural Interactivity" and "Active Content". Several on-going projects are addressing the design of conversational systems. Three example applications are: the ESPRIT Mask project which aims to develop a Multimodal-Multimedia Automated Service Kiosk providing access to rail travel information; the LE Arise project which aims to provide similar information via the telephone for the Dutch, French, and Italian railways; and the LE Vodis project whose goal is to develop a vocal interface to enhance the usability and functionality of in-car driver assistance and information services.
  61 K 90 K
• An Overview of the Philips Dialog System, Harald Aust, Olaf Schroer: Philips.
  In 1994, we introduced the world's first publicly available natural-language dialog system, which has given information on the schedule of the German railways. Since then, the software package used for the implementation of this system has been improved in various ways. In addition, the original train timetable information system has not remained the sole application: A number of other systems have become available during the past few years and are now in everyday use. In this paper, we will give a brief overview of our system, the underlying technology, and the most important applications realized so far.
  55 K 495 K
• Multilingual and Crosslingual Speech Recognition, T. Schultz, Alex Waibel: Univ. of Karlsruhe and CMU.
  This paper describes the design of a multilingual speech recognizer using an LVCSR dictation database which has been collected under the project GlobalPhone. This project at the University of Karlsruhe investigates LVCSR systems in 15 languages of the world, namely Arabic, Chinese, Croatian, English, French, German, Italian, Japanese, Korean, Portuguese, Russian, Spanish, Swedish, Tamil, and Turkish. For our experiments we used six of these languages to train and test several recognition engines in monolingual, multilingual, and crosslingual setups. Based on a global phoneme set we built a multilingual speech recognition system which can handle five different languages. The acoustic models of the five languages are combined into a monolithic system and context dependent phoneme models are created using language questions.
  183 K 486 K

Language Modeling

• Language-Model Investigations Related to Broadcast News, Dietrich Klakow, Xavier Aubert, Peter Beyerlein, Reinhold Haeb-Umbach, Meinhard Ullrich, Andreas Wendemuth, Patricia Wilcox: Philips.
  In this paper we present some experiments that have been performed while developing language models for the PHILIPS Broadcast News system. Three main issues will be discussed: construction of phrases, adaptation of remote corpora to this task, and the combination of the different models. Also, perplexities on the 1997 evaluation data are reported.
  144 K 88 K
• Evaluation Metrics for Language Models, Stanley Chen, Douglas Beeferman, Ronald Rosenfeld: CMU.
  The most widely-used evaluation metric for language models for speech recognition is the perplexity of test data. While perplexities can be calculated efficiently and without access to a speech recognizer, they often do not correlate well with speech recognition word-error rates. In this research, we attempt to find a measure that like perplexity is easily calculated but which better predicts speech recognition performance. We investigate two approaches; first, we attempt to extend perplexity by using similar measures that utilize information about language models that perplexity ignores. Second, we attempt to imitate the word-error calculation without using a speech recognizer by artificially generating speech recognition lattices. To test our new metrics, we have built over thirty varied language models. We find that perplexity correlates with word-error rate remarkably well when only considering n-gram models trained on in-domain data. When considering other types of models, our novel metrics are superior to perplexity for predicting speech recognition performance. However, we conclude that none of these measures predict word-error rate sufficiently accurately to be effective tools for language model evaluation in speech recognition.
  87 K 147 K
• Meeting Browser: Tracking and Summarizing Meetings, Alex Waibel, M. Bett, Michael Finke, R. Stiefelhagen: CMU.
  To provide rapid access to meetings between human beings, transcription, tracking, retrieval and summarization of ongoing human-to-human conversation has to be achieved. In DARPA and DoD sponsored work (projects GENOA and CLARITY) we aim to develop strategies to transcribe human discourse and provide rapid access to the structure and content of this human exchange.
  67 K 584 K
• Named Entity Extraction from Speech, Francis Kubala, Richard Schwartz, Rebecca Stone, Ralph Weischedel: BBN.
  We report results using a hidden Markov model to extract information from broadcast news. IdentiFinder was trained on the broadcast news corpus and tested on both the 1996 HUB-4 development test data and the 1997 HUB-4 evaluation test data with respect to the named entity (NE) task.
  53 K 462 K

Hub-5 Status and Research

• Summer Research Workshops on Speech and Language, Frederick Jelinek: Johns Hopkins U.
  Vision of LVCSR Workshops; The Workshop Environment at CLSP; Workshop Planning Process; 1997 LVCSR Workshop; 1998 NSF Sponsored Speech and Language Workshop
  103 K 304 K
• Utilizing Untranscribed Training Data to Improve Performance, George Zavaliagkos, Thomas Colthurst: BBN.
  In the past few years, the Large Vocabulary Conversational Speech Recognition (LVCSR) community has attempted to address the problem of speech recognition on languages other than English. Work on the CallHome Corpora has verified that current technology is largely language independent, and that the dominant factor with regards to performance on a certain language is the amount of training data available. This brings forth the question of what is the appropriate course of action when we need to quickly bring a recognizer up in a new language, were little or no training is available. This is exactly the question we will address in this paper. We will assume that, while only a couple of hours of transcribed data is available, much more untranscribed data can be found, and we will explore ways to utilize it.
  149 K 282 K
• Studies with Fabricated Switchboard Data, Don McAllaster, Larry Gillick, Francesco Scattone, Michael Newman: Dragon Systems.
  We present a study of data simulated using acoustic models trained on Switchboard data, and then recognized using various Switchboard-trained acoustic models. The Switchboard-trained models yield word error rates of about 47%, on real Switchboard conversations. When data is simulated using the acoustic models, but in a way that insures that the pronunciations in our recognition dictionary are "perfect", the WER drops by nearly a factor of five. If instead we use hand-labeled phonetic transcriptions to fabricate data that more realistically represents the way words are pronounced -- rendering our recognition pronunciations imperfect -- we obtain WERs in the low 40's, rates that are fairly similar to those seen in actual speech data. Taken as a whole, these and other experiments we describe in the paper suggest that there is a substantial mismatch between real speech data and our speech models. The use of simulation in speech recognition research appears to be a promising tool in our efforts to understand and reduce the size of this mismatch.
  38 K 57 K
• DYNAMO: An Algorithm for Dynamic Acoustic Modeling, Francoise Beaufays, Mitch Weintraub, Yochai Konig: SRI.
  This paper summarizes part of SRI's effort to improve acoustic modeling in the context of the Large Vocabulary Continuous Speech Recognition (LVCSR) project. It concentrates on two problems that are believed to contribute to the large error rates observed with LVCSR databases: (1) the lack of discriminative power of the speech models in the acoustic space, and (2) the discrepancy between the criterion used to train the models (typically frame-level maximum likelihood) and the task expected from the models (word-level recognition). We address the first issue by searching for features that help in narrowing the model distributions, and by proposing a neural-network-based architecture to combine these features. The neural networks (NNET) are used in association with a set of large Gaussian mixture models (GMM) whose mixture weights are dynamically estimated by the neural networks, for each frame of incoming data. We call the resulting algorithm DYNAMO, for dynamic acoustic modeling. To address the second problem, we propose two discriminative training criteria, both defined at the sentence level. We report preliminary results with the Spanish Callhome database.
  80 K 99 K

Demos

• Designing Conversational Interfaces with Multimodal Interaction, Joshua Bers, Scott Miller, John Makhoul: BBN.
  Our current research focuses on developing conversational interfaces to on-line applications through speech recognition technology. We have developed a prototype system that combines pen and speech input from the on-line user in a web-browser. VoiceLog is a voice-enabled connection to a web-server that allows one to obtain vehicle diagrams and to place orders for specific parts in these diagrams. VoiceLog features a novel client-server approach to speech recognition, modular reusable components and a simple Java-based interface. This paper briefly describes the system and its architecture including the handling of simultaneous input from pen and speech, the production of audio and visual feedback, and the management of multimodal dialogue.
  71 K 472 K
• The GTNPhone Dialog System, David Stallard, Joshua Bers, Christopher Barclay: BBN.
  BBN's GTNPhone is an over-the-telephone spoken language dialog interface to GTN, a DOD logistics information website. GTNPhone allows any user with a telephone, wired or cellular, to call in and retrieve information from the GTN web-site using voice only. No web browser or computer is required. The main body of the system is implemented in Java, with native code interfaces to commercial off-the-shelf components (such as a speech synthesizer) implemented in C. It runs on a Solaris workstation server located at BBN.
  33 K 310 K