Note: All papers are available as PostScript or Adobe Acrobat Portable Document Format (PDF) files; click on the appropriate icon for that paper.

Most of the papers are also viewable as Web-accessible HTML documents. Their titles appear as hotlinks. If your computer does not have the software to view either PDF or PostScript documents, click here for instructions on obtaining the necessary viewing software.

Overview

• 1998 Broadcast News Benchmark Test Results, David S. Pallett, Jonathan G. Fiscus, John S. Garofolo, Alvin Martin, Mark A. Przybocki: NIST.
  This paper documents the use of Broadcast News test materials in DARPA-sponsored Automatic Speech Recognition (ASR) Benchmark Tests conducted late in 1998. As in last year's tests, statistical selection procedures were used in selecting test materials. Two test epochs were used, each yielding (nominally) one and one-half hours of test material. One of the test sets was drawn from the same test epoch as was used for last year's tests, and the other was drawn from a more recent period. Results are reported for two types of systems: one (the "Hub", or "baseline" systems) for which there were no limits on computational resources, and another (the "less than 10X real-time spoke" systems) for systems that ran in less than 10 times real-time.The lowest word error rate reported this year for the "Hub" systems was 13.5%, contrasting with last year's lowest word error rate of 16.2%. For the "less than 10X real-time spoke" systems, the lowest reported word error rate was 16.1%. Results are also reported, for the second year, on non-English language Broadcast News materials in Spanish and Mandarin.
  75 K 854 K
  Click on this paper's title (or here) to access HTML version.

• 1998 Hub-4 Information Extraction Evaluation, Mark A. Przybocki, Jonathan G. Fiscus, John S. Garofolo, David S. Pallett: NIST.
  This paper documents the Information Extraction Named-Entity Evaluation (IE-NE), one of the new spokes added to the DARPA-sponsored 1998 Hub-4 Broadcast News Evaluation. This paper discusses the information extraction task as posed for the 1998 Broadcast News Evaluation. This paper reviews the evaluation metrics, the scoring process, and the test corpus that was used for the evaluation. Finally, this paper reviews the results of the first running of a Hub-4 IE-NE Evaluation. The Baseline IE-NE evaluation, in which BBN's IdentiFinder was run on the primary system transcripts submitted for the Hub-4 Broadcast News evaluation, found that the transcripts generated by LIMSI's automatic speech recognition system produced the "highest" F-measure score (82.39). In the Quasi IE-NE evaluation, where sites ran their own NE-taggers on a set of three baseline recognizer transcripts, the SRI developed tagger achieved the highest F-measure score for baseline recognizers 1 & 3, while the BBN developed tagger achieved the highest score for baseline recognizer 2. In the Full IE-NE evaluation, where sites implemented their own NE-tagger on the their own automatic speech recognizer transcripts, BBN achieved the highest overall F-measure score of 82.22.
  73 K 529 K
  Click on this paper's title (or here) to access HTML version.

• NIST's 1998 Topic Detection and Tracking Evaluation (TDT2), Jonathan G. Fiscus, George Doddington, John S. Garofolo, Alvin Martin: NIST.
  This paper presents a summary of the 1998 Topic Detection and Tracking (TDT) tasks and the results of the 1998 TDT evaluation. The purpose of TDT is to develop technologies for retrieval and automatic organization of Broadcast News and Newswire stories and to evaluate the performance of those technologies. The TDT project builds on and extends the technologies of Automatic Speech Recognition and Document Retrieval with three tasks: 1) Story Segmentation, 2) Topic Detection and 3) Topic Tracking. Each of the tasks simulates a hypothetical operational system that requires incoming data to be processed time synchronously. The 1998 TDT evaluation (TDT2) continues the work of the TDT pilot study conducted in 1997 (TDT1) and is the first open evaluation of TDT tasks.
  137 K 767 K
  Click on this paper's title (or here) to access HTML version.

Information Extraction

• Overview: Information Extraction from Broadcast News, Patricia Robinson, Erica Brown, John D. Burger, Nancy Chinchor, Aaron Douthat, Lisa Ferro, Lynette Hirschman: MITRE/SAIC.
  Broadcast news is a rich domain for information extraction, but one that presents new challenges for evaluation. In this paper we present an overview of the first evaluation of information extraction from broadcast news that was conducted as part of the DARPA-funded Hub 4 1998 workshop. We discuss the work that was required to design and administer the evaluation, describe some of the challenges that we encountered, and summarize the results of the evaluation.
  36 K 575 K
  Click on this paper's title (or here) to access HTML version.

• Studies in Data Annotation Effectiveness, Michael R. Crystal, Francis Kubala, Robert MacIntyre: BBN.
  As we develop more train-by-example systems, we have an increasing need for highly effective, low-cost training data - the examples. In most cases, preparing training data is a laborious task conducted by humans. The study related herein quantifies the effectiveness and cost of different approaches to annotating data for our named-entity extraction from speech system. We conclude that annotator experience and annotation volume are the two most important factors for effecting increase model performance.
  34 K 288 K
  Click on this paper's title (or here) to access HTML version.

• Named Entity Extraction from Broadcast News, David Miller, Richard Schwartz, Ralph Weischedel, Rebecca Stone: BBN.
  In this paper, we contrast the two tasks of named entity extraction from speech and text both qualitatively and quantitatively in the context of the DARPA 1998 Hub4e-IE evaluation. We will present some top level observations and a detailed engineering analysis of our system's failures and successes. We explore the effects of word error rate, loss of textual clues, amount of training data, changes in guidelines, and out-of-vocabulary errors.
  44 K 81 K
  Click on this paper's title (or here) to access HTML version.

• Information Extraction from Broadcast News Speech Data, David D. Palmer, John D. Burger, Mari Ostendorf: MITRE/Boston Univ..
  In this paper we describe a robust algorithm for information extraction from spoken language data. Our probabilistic algorithm builds on results in language modeling, using class-based smoothing to produce state-of-the-art performance for a wide range of speech error rates. We show that our system performs well with sparse data, as well as with out-of-domain data.
  54 K 520 K

• Baseline IE-NE Experiments Using the SPRACH/LASIE System, Steve Renals, Yoshi Gotoh, Robert Gaizauskas, Mark Stevenson: University of Sheffield.
  We have developed two conceptually different systems that are able to identify named entities from spoken audio. One (referred to as SPRACH-S) has a stochastic finite state machine structure for use with an acoustic model that identifies both words and named entities from speech data. The other (referred to as SPRACH-R) is a rule-based system which uses matching against stored name lists, part-of-speech tagging, and light phrasal parsing with specialised named entity grammars. We provide an overview of the two approaches and present results on the Hub-4E IE-NE evaluation task.
  45 K 60 K
  Click on this paper's title (or here) to access HTML version.

• Named Entity Extraction from Speech: Approach and Results Using the TextPro System, Douglas E. Appelt, David Martin: SRI International.
  This paper describes the application of the TextPro system to the task of recognition of named entities in speech. TextPro is a lightweight engine for interpreting cascaded finite-state transducers. Although originally intended for processing text, the experience of this evaluation demonstrates that the system can easily be adapted to processing transcripts generated by a speech recognizer as well.
  18 K 572 K
  Click on this paper's title (or here) to access HTML version.

TDT: Segmentation

• The TDT-2 Text and Speech Corpus, Chris Cieri, David Graff, Mark Liberman, Nii Martey, Stephanie Strassel: Univ. of Pennsylvania Linguistic Data Consortium.
  This paper describes the creation and content of the TDT-2 corpus in the context of the TDT-2 research project it supports and in comparison to previous and subsequent efforts.
  74 K 711 K
  Click on this paper's title (or here) to access HTML version.

• Combining Words and Speech Prosody for Automatic Topic Segmentation, Andreas Stolcke, Elizabeth Shriberg, Dilek Hakkani-Tur, Gokhan Tur, Ze'ev Rivlin, Kemal Sonmez: SRI International.
  We present a probabilistic model that uses both prosodic and lexical cues for the automatic segmentation of speech into topic units. The approach combines hidden Markov models, statistical language models, and prosody-based decision trees. Lexical information is obtained from a speech recognizer, and prosodic features are extracted automatically from speech waveforms. We evaluate our approach on the Broadcast News corpus, using standard evaluation metrics. Results show that the prosodic model alone outperforms the word-based segmentation method. Furthermore, we achieve an additional reduction in error by combining the prosodic and word-based knowledge sources.
  45 K 56 K
  Click on this paper's title (or here) to access HTML version.

• Story Segmentation and Topic Detection in the Broadcast News Domain, S. Dharanipragada, M. Franz, Jeffrey S. McCarley, S. Roukos, Todd Ward: IBM.
  In this paper we present algorithms for story segmentation and topic detection. Both algorithms are online algorithms and use a combination of machine learning, statistical natural language processing and information retrieval techniques. The story segmentation algorithm is a two stage algorithm that uses a decision tree based probabilistic model in the first stage and incorporates aspects of our detection system via an information-retrieval based refinement scheme in the second stage. The topic detection algorithm is an incremental clustering algorithm that employs a novel dynamic cluster-dependent similarity measure between documents and clusters. C_seg and topic-weighted C_det for these algorithms on the 1998 TDT2 Evaluation are 0.1651 and 0.0042.
  131 K 129 K

• A Cluster-Based Approach to Tracking, Detection, and Segmentation of Broadcast News, D. Eichmann, Miguel Ruiz, Padmini Srinivasan, Nick Street, Chris Culy, Filippo Menczer: Univ. of Iowa.
  We present results of the University of Iowa topic tracking and detection as well as story segmentation efforts. Topic tracking is performed for the "boundaries given" case. The DET curves for all the runs are consistently smooth and concave suggesting no sudden changes in expectation required from the user. The effect of reducing the training size of relevant stories is examined. The detection runs are performed using a "pipeline" model to utilize the advantage of the deferral period. Performance is strongly influenced by the fact that roughly 2000 to 3000 declared topic clusters are generated during the detection runs. Performance is analyzed with respect to changing the cluster threshold. In segmentation, an agglomerative clustering strategy is adopted. The decision to declare a boundary depends upon both lexical similarity of neighboring segments as well as the pause duration. The algorithmic complexity of the method is O(k log k) where k is the number of pause delimited sentences in the file. The tracking, detection and segmentation modules provide a sound framework for future extension and experimentation.
  80 K 2514 K
  Click on this paper's title (or here) to access HTML version.

• Segmentation of Automatically Transcribed Broadcast News Text, P. van Mulbregt, Ira Carp, Larry Gillick, Stephen A. Lowe, Jonathan P. Yamron: Dragon Systems.
  Expertise in the automatic transcription of broadcast speech has progressed to the point of being able to use the resulting transcripts for information retrieval purposes. In this paper, we describe the Segmentation system used by Dragon Systems in the Segmentation task of the 1998 TDT evaluation, highlighting improvements made since the September 1998 dryrun. Segmentation of closed-caption and human transcripts of news is contrasted with the results of segmenting the ASR transcripts. This is followed by a discussion of the metric, in particular how the value of the segmentation metric relates to the value of the tracking metric, when the these latter two tasks are performed on automatically segmented ASR text, rather than the ASR text with correctly marked boundaries.
  67 K 322 K
  Click on this paper's title (or here) to access HTML version.

Speech 1: Overview and Baselines

• Broadcast News is Good News, Francis Kubala: BBN.
  The past decade has been witness to the rapid maturation of automatic speech recognition technology. In the past nine years, the field has grown from small artificial research problems and discrete word commercial products and is now productively engaged on large real-world problems of unconstrained spoken language. In the research arena, the highly variable and content-rich domain of broadcast news has proved to be the right catalyst for producing robust scalable algorithms that methodically improve recognition accuracy every year. In the commercial arena, we've seen the appearance of wonderful shrink-wrapped dictation software and broad-based user acceptance. Now at the end of the millennium, speech recognition seems finally poised to realize its great promise. The Hub4 research effort of the 1990's and its recent focus on broadcast news have played a major role in placing automatic speech recognition on the launchpad.
  43 K 403 K

• Recent Improvements to IBM's Speech Recognition System for Automatic Transcription of Broadcast News, Scott S. Chen, Ellen Eide, M.J.F. Gales, Ramesh A. Gopinath, D. Kanevsky, P. Olsen: IBM.
  We describe recent improvements to IBM's system for automatic transcription of broadcast news. Some of the improvements are: Bayesian Information Criteria (BIC) applied to choosing the number of components in a Gaussian mixture model, tail distribution modelling using Richter distributions and power exponential distributions, pronunciation networks, adaptive training techniques such as clustered adaptive training (CAT) and a modified version of speaker adaptive training (SAT) which is efficient for large tasks, factor analysis invariant to linear transforms. We also experimented with changes such as changing the acoustic vocabulary, reducing the number of phonemes and insertion of short pauses. The models were combined in a single system using NIST's script voting machine known as ROVER.
  347 K 235 K

• Automatic Modeling of Pronunciation Variations, Ellen Eide: IBM.
  We report on an automatic method for discovering an appropriate model topology for each context-dependent phoneme. The model allows for such phenomena as reduced pronunciaitons and substituted phonemes where warranted by observation on training data.
  62 K 62 K
  Click on this paper's title (or here) to access HTML version.

• The LIMSI 1998 Hub-4E Transcription System, Jean-Luc Gauvain, Lori Lamel, Gilles Adda, Michele Jardino: LIMSI-CNRS.
  In this paper we report on our Nov98 Hub-4E system, which is an extension of our Nov97 system. The LIMSI system for the November 1998 Hub-4E evaluation is a continuous mixture density, tied-state cross-word context-dependent HMM system. The acoustic models were trained on the 1995, 1996 and 1997 official Hub-4E training data containing about 150 hours of transcribed speech material. 65K word language models were obtained by interpolation of backoff n-gram language models trained on different text data sets. Prior to word decoding a maximum likelihood partitioning algorithm segments the data into homogenous regions and assigns gender, bandwidth and cluster labels to the speech segments. Word decoding is carried out in three steps, integrating cluster-based MLLR acoustic model adaptation. The final decoding step uses a 4-gram language model interpolated with a category trigram model. The main differences compared to last year's system arise from the use of additional acoustic and language model training data, the use of divisive decision tree clustering instead of agglomerative clustering for state-tying, generation graph word using adapted acoustic models, the use of interpolated LMs trained on different data sets instead of training a single model on weighted texts, and a 4-gram LM interpolated with a category model. The overall word transcription error on the Nov98 evaluation test data was 13.6%.
  64 K 103 K
  Click on this paper's title (or here) to access HTML version.

• The 1998 OGI-Fonix Broadcast News Transcription System, Xintian Wu, Chaojun Liu, Yonghong Yan, Doughwa Kim, Seth Cameron, Randy Parr: OGI/Fonix.
  This paper describes the OGI-FONIX large vocabulary system developed for the 1998 broadcast news evaluation. The main differences from our last year's system are: (1) A multiple pass decoder is used. (2) Long periods of silence are deleted in both training and decoding features. Cepstral mean subtraction is applied individually to automatically derived speaker clusters. These methods result in a 3% absolute improvement on a self-constructed 1000 seconds test set. (3) Different dictionaries are used for the within word and crossword decoding passes. Long segments from the first pass are chopped at hypothesized word boundaries and then merged back in the crossword decoding pass. These two methods yield 2.3% absolute improvement on the hub4e_97 evaluation set. (4) The system vocabulary size has been increased from 25K to 56K. With these changes, our 1998 system achieves a word error rate 27.9% on the h4e_98_1 test set, and 23.6% on the h4e_98_2 test set for the hub4e_98 evaluation.
  31 K 58 K
  Click on this paper's title (or here) to access HTML version.

TDT: Detection and Tracking 1

• UMASS Approaches to Detection and Tracking at TDT2, Ron Papka, James Allan, Victor Lavrenko: Univ. of Mass..
  The following work describes our solutions to the detection and tracking problems defined by the Topic Detection and Tracking (TDT2) research initiative. We discuss the implementation and results of the approaches which were recently tested on the TDT2 evaluation corpus. Our solutions to these problems extend text-based ranked retrieval techniques previously used for document clustering and filtering tasks. We present the effects of different on-line hierarchic clusterings on the detection task. In addition we compare adaptive and static approaches for building linear text classifiers for the tracking task.
  93 K 3153 K

• CMU Report on TDT-2: Segmentation, Detection and Tracking, Jaime Carbonell, Yiming Yang, John Lafferty, Ralf D. Brown, Tom Pierce, Xintian Liu: CMU.
  This paper reports the results achieved by Carnegie Mellon University on the Topic Detection and Tracking Project's second-year evaluation for the segmentation, detection, and tracking tasks. Additional post-evaluation improvements are also presented.
  18 K 42 K
  Click on this paper's title (or here) to access HTML version.

• GE Tracker: A Robust, Lightweight Topic Tracking System, Tomek Strzalkowski, Gees C. Stein, G. Bowden Wise: GE Corporate R&D.
  We describe a topic tracking system developed at GE R&D Center in connection with our participation in DARPA TDT evaluations. The TDT tracking task is specified as follows: given Nt training news stories on a topic, the system must find all subsequent stories on the same topic in all tracked news sources.
  23 K 204 K
  Click on this paper's title (or here) to access HTML version.

• The Beta-Binomial Mixture Model and its Application to TDT Tracking and Detection, Stephen A. Lowe: Dragon Systems.
  This paper describes a continuous-mixture statistical model for word occurrence frequencies in documents, and the application of that model to the TDT topic identification tasks. This model was originally proposed by Gillick as a means to account for variation in word frequencies across documents more accurately than the binomial and multinomial models. Further mathematical development of the model will be presented, along with performance results on the DARPA TDT December 1998 Evaluation Tracking Task. Application to the Detection Task will also be discussed.
  99 K 560 K

• Topic Tracking in a News Stream, Jonathan P. Yamron, Ira Carp, Larry Gillick, Stephen A. Lowe, P. van Mulbregt: Dragon Systems.
  In this paper we describe a Topic Tracking system based on unigram models, submitted by Dragon Systems in the December 1998 Topic Detection and Tracking (TDT) evaluation. We focus on the most recent developments, including improvements in the smoothing of sparse unigram models, a better discriminator, and the implementation of unsupervised adaptation. We give results on the default test conditions, namely, tracking in newswire and automatically recognized broadcast given four story samples, as well as several variations: one story sample, automatically recognized broadcast only, and automatically recognized broadcast with automatically determined story boundaries. Finally, we show the effect of interpolating this system with Dragon's other tracking system based on a Beta-Binomial model.
  83 K 369 K

Related Work 1

• Audio-Visual Speaker Recognition for Video Broadcast News, Chalapathy Neti, Andrew Senior: IBM.
  Significant progress has been made in the transcription of the audio stream in the broadcast news domain for both radio and TV news (HUB4 task). Such transcripts provide an excellent means of indexing video content for search and retrieval. Speaker identification is an important technology in this domain both for selecting high-accuracy speaker-dependent models for transcription and as an index for search and retrieval of video content. However, the transcription accuracy under acoustically degraded conditions (such as background noise) and channel mismatch (telephone) still needs further improvements. To make improvements in such degraded conditions is a hard problem. We have begun investigating the combination of audio-based processing with visual processing for both speech and speaker recognition to improve the accuracy in acoustically degraded conditions. The use of two independent sources of information brings significantly increased robustness to signal degradation since degradations in the two channels are uncorrelated, and the use of visual information allows a much faster speaker identification than possible with acoustic information. In this paper, we present some encouraging preliminary results for audio-visual speaker recognition for TV broadcast news data (CNN).
  100 K 87 K

• Recent Improvements in Voicemail Transcription, George Zweig, G. Saon, Mukund Padmanabhan, J. Huang, S. Basu: IBM.
  We report recent improvements in voicemail transcription. The voicemail transcription task was introduced last year as representing a style of conversational telephone speech that is somewhat different from the Switchboard and CallHome databases. Last year, the speaker independent and speaker adapted word error rates (WER) on this task were reported at 41.94% and 38.18% respectively. This year, we report a relative improvement of 18% in the speaker independent performance and 11% in the speaker adapted performance over last year. This improvement is a result of some new algorithms and an increase in the amount of training data. We describe the contribution of several components to improving the word error rate.
  112 K 104 K
  Click on this paper's title (or here) to access HTML version.

Speech 2: Baselines and Non-English

• The Philips/RWTH System for Transcription of Broadcast News, P. Beyerlein, X. Aubert, R. Haeb-Umbach, M. Harris, Dietrich Klakow, A. Wendemuth, Sirko Molau, Michael Pitz, A. Sixtus: Philips Research Lab, Germany/Aachen Univ..
  The paper contains a description of the Philips/RWTH 1998 HUB4 system which was build in a joint effort of Philips Research Laboratories Aachen and Aachen University of Technology. We will focus our discussion on recent improvements compared to the original 1997 HUB4 system and evaluate them on the HUB4'97 evaluation data. The paper will deal with (1) a rough system overview, including slight modifications in feature extraction, training and recognition; (2) improvements obtained in audio stream segmentation and segment clustering; (3) log-linear interpolation of "distance" language models; and (4) the integration of various acoustic and language models via Discriminative Model Combination (DMC). The performance of the described system is 23% (relative) better than the performance of the 1997 Philips HUB4 system. A word error rate of 17.9% was achieved on the 1997 HUB4 evaluation set, compared to 23.5% using the original 1997 system.
  184 K 348 K
  Click on this paper's title (or here) to access HTML version.

• Automatic Transcription Verification of Broadcast News and Similar Speech Corpora, Michael Pitz, Sirko Molau, Ralf Schluter, Hermann Ney: RWTH Aachen.
  In this paper we present a method to automatically detect faulty training scripts. Based on the Hub-4 task we will report on the efficiency of error detection with the proposed method and investigate the effect of both manually and automatically cleaned training corpora on the word error rate (WER) of the RWTH large vocabulary continuous speech recognition (LVCSR) system.
  34 K 42 K
  Click on this paper's title (or here) to access HTML version.

• An Overview of the SPRACH System for the Transcription of Broadcast News, Gary Cook, James Christie, Dan Ellis, Eric Fosler-Lussier, Yoshi Gotoh, Brian Kingsbury, Nelson Morgan, Steve Renals, Tony Robinson, Gethin Williams: Cambridge Univ./ICSI/Sheffield Univ.
  This paper describes the SPRACH system developed for the 1998 Hub-4E broadcast news evaluation. The system is based on the connectionist-HMM framework and uses both recurrent neural network and multi-layer perceptron acoustic models. We describe both a system designed for the primary transcription hub, and a system for the less-than 10 times real-time spoke. We then describe recent developments to CHRONOS, a time-first stack decoder. We show how these developments have simplified the evaluation system, and led to significant reductions in the error rate of the 10x real-time system. We also present a system designed to operate in real-time with negligible search error.
  65 K 109 K
  Click on this paper's title (or here) to access HTML version.

• Reducing Errors by Increasing the Error Rate: MLP Acoustic Modeling for Broadcast News Transcription, Nelson Morgan, Dan Ellis, Eric Fosler-Lussier, Adam Janin, Brian Kingsbury: International Computer Science Institute.
  We describe some aspects of a Broadcast News recognition system based on hybrid HMM/MLP acoustic modeling. These include the use of novel 'modulation spectrogram' features which are combined with conventional models at the posterior probability level, some experiments with nonlinear segment normalization, and an investigation of the interaction of model size and training set size for an multi-layer perceptron (MLP) acoustic classifier. We also report preliminary results of incorporating gender-dependence into this system.
  38 K 198 K
  Click on this paper's title (or here) to access HTML version.

• Not just what, but also when: Guided Automatic Pronunciation Modeling for Broadcast News, Eric Fosler-Lussier, Gethin Williams: International Computer Science Institute/Sheffield Univ.
  In this paper, we describe improvements to the pronunciation model featured in the 1998 SPRACH Broadcast News evaluation system. Various smoothing and pruning techniques and the integration of confidence scores into the pronunciation model training provided a 4% relative improvement over the baseline model. We also report on promising new techniques that did not appear in the evaluation system.
  100 K 150 K
  Click on this paper's title (or here) to access HTML version.

• The 1998 Carnegie Mellon University Sphinx-3 Spanish Broadcast News Transcription System, Juan M. Huerta, Stanley Chen, Richard M. Stern: CMU.
  This paper describes the 1998 CMU Hub 4 Spanish broadcast news transcription system. We focus on the development and improvements of the system with respect to the 1997 system. Both the 1997 and 1998 systems were developed using exactly the same acoustic and language model training material, thus the improvements obtained resulted from a better utilization and modeling of these corpora and a better decoding configuration and strategy. Specifically, we employed several language models, a larger lexicon and larger acoustic models than our 1997 system. Due to these improvements, we achieved a reduction of 28% in the error rate on this year's development material.
  29 K 272 K

• The IBM LVCSR System Used for 1998 Mandarin Broadcast News Transcription Evaluation, XueFeng Guo, WeiBin Zhu, Qin Shi, Scott S. Chen, Ramesh A. Gopinath: IBM China Research Lab.
  This paper presents the technologies implemented in the IBM's Large Vocabulary Continuous Speech Recognition(LVCSR) system which was designed for 1998 Mandarin broadcast news transcription evaluation task. Compared with the 1997 system, it focuses on acoustic improvements by implementing several new schemes such as LDA and MLLT transformation matrix, BIC model selection criterion, SAT and CAT models. In addition, new language model components and new vocabulary were built. Some other schemes which were tried we also described.
  30 K 268 K
  Click on this paper's title (or here) to access HTML version.

• Dragon Systems' 1998 Broadcast News Transcription System for Mandarin, Puming Zhan, Steven Wegmann, Larry Gillick: Dragon Systems.
  In this paper we shall describe Dragon Systems' 1998 Broadcast News transcription system for Mandarin. We shall describe our music classifier, which was unique to our Mandarin system, as well as our speaker change detection algorithm, which was used in our English and Mandarin systems. We shall also report on preliminary, post-evaluation experiments with pitch.
  31 K 311 K
  Click on this paper's title (or here) to access HTML version.

TDT: Detection and Tracking 2

• Topic Detection and Tracking Using idf-Weighted Cosine Coefficient, J. Michael Schultz, Mark Liberman: Univ. of Penn.
  The goal of TDT Topic Detection and Tracking is to develop automatic methods of identifying topically related stories within a stream of news media. We describe approaches for both detection and tracking based on the well-known idf-weighted cosine coefficient similarity metric. The surprising outcome of this research is that we achieved very competitive results for tracking using a very simple method of feature selection, without word stemming and without a score normalization scheme. The detection task results were not as encouraging though we attribute this more to the clustering algorithm than the underlying similarity metric.
  82 K 240 K

• Topic Detection in Broadcast News, Frederick Walls, Hubert Jin, Sreenivasa Sista, Richard Schwartz: BBN Technologies.
  We propose a system for the Topic Detection and Tracking (TDT) detection task concerned with the unsupervised grouping of news stories according to topic. We use an incremental k-means algorithm for clustering stories. For comparing stories, we utilize a probabilistic document similarity metric and a traditional vector-space metric. We note that the clustering algorithm requires two different types of metrics and adapt similarity metrics for each purpose. The system achieves a topic-weighted miss rate of 12% at a false accept rate of 0.22%.
  63 K 102 K
  Click on this paper's title (or here) to access HTML version.

• Topic Tracking for Radio, TV Broadcast, and Newswire, Hubert Jin, Richard Schwartz, Sreenivasa Sista, Frederick Walls: BBN Technologies.
  We present our tracking system for the 1998 Topic Detection and Tracking project (TDT-2). This project addresses multiple sources of information in the form of both text and speech from newswire, radio and television news broadcast programs. The technical challenge of TDT-2 tracking is to follow the topics being discussed in the stories from multiple sources. Our tracking system is probability based and we successfully solve the problem of score normalization across topics. Our automatic score normalization is simple, efficient and very effective. Tested on the 20K TDT-2 stories collected between March and April 1998, our tracking system achieves the performance of 1.5% miss error (on a combination of closed caption and newswire) and 3.0% miss error (on a combination of automatic speech recognition output and newswire) at the cost of 0.1% false alarm error. In the 1998 TDT-2 evaluation, our tracking system was ranked the best with the official topic-weighted Ctrack of 0.0057.
  193 K 12091 K
  Click on this paper's title (or here) to access HTML version.

• A Description of the CIDR System as Used for TDT-2, Dragomir Radev, Vasileios Hatzivassiloglou, Kathleen R. McKeown: Columbia Univ.
  We describe several experimental parameters and a parallelization technique used in our online document clustering system, CIDR. These modifications were introduced into CIDR to reduce the running time so that incoming documents be clustered in almost real time. We discuss how several of these parameters are justified on linguistic grounds and report preliminary quantitative results on the effects that these parameters have on speed and accuracy.
  64 K 583 K

• Topic Tracking with the PRISE Information Retrieval System, Douglas W. Oard: Univ. of Maryland.
  Information retrieval systems offer an attractive alternative to construction of a topic tracking system from scratch. The freely available PRISE vector space text retrieval system was applied to the TDT-2 topic tracking task. A simple version of the Rocchio formula was used for profile formulation and a retrieval status value threshold was used in conjunction with a temporal cutoff to make hard decisions. The results indicate that our simple approach produced a credible system, but comparison with results achieved by other systems indicates that there is room for improvement. The paper concludes by identifying some promising directions for further work that would be compatible with our approach.
  46 K 130 K
  Click on this paper's title (or here) to access HTML version.

Related Work 2

• 1998 TREC-7 Spoken Document Retrieval Track Overview and Results, John S. Garofolo, Ellen M. Voorhees, Cedric G. P. Auzanne, Vincent M. Stanford, Bruce Lund: NIST.
  This paper describes the 1998 TREC-7 Spoken Document Retrieval (SDR) Track which implemented an evaluation of retrieval of broadcast news excerpts using a combination of automatic speech recognition and information retrieval technologies. The motivations behind the SDR Track and background regarding its development and implementation are discussed. The SDR evaluation collection and topics are described and summaries and analyses of the results of the track are presented. Alternative metrics for automatic speech recognition as applicable to retrieval applications are also explored. Finally, plans for future SDR tracks are described.
  87 K 740 K
  Click on this paper's title (or here) to access HTML version.

• AT&T at TREC-7 SDR Track, Amit Singhal, John Choi, Donald Hindle, Julia Hirschberg, Fernando Pereira, Steve Whittaker: AT&T Labs-Research.
  AT&T participated in the Spoken Document Retrieval (SDR) track of TREC-7. Our speech retrieval system uses modern Information Retrieval (IR) methods in conjunction with in-house automatic speech recognition. The novel feature of our TREC-7 work is the use of document expansion to reduce the performance loss due to ASR errors. Results show that retrieval from automatic transcriptions of speech is quite competitive with doing retrieval from human transcriptions. Our experiments indicate that document expansion can be used to further improve retrieval from automatic transcripts. This paper presents some analysis of document expansion in context of the TREC-7 SDR track task.
  59 K 81 K
  Click on this paper's title (or here) to access HTML version.

• Improvements in Japanese Broadcast News Transcription, Katsutoshi Ohtsuki, Sadaoki Furui, Naoyuki Sakurai, Atushi Iwasaki, Zhi-Peng Zhang: Tokyo Institute of Technology/NTT.
  This paper reports on recent improvements in Japanese broadcast news transcription and topic extraction. We constructed a language model that depends on the readings of words in order to prevent recognition errors caused by context-dependent readings of Japanese characters. We also introduced interjection modeling into the language model. To improve the model's performance for a series of sentences spoken by one speaker, an on-line incremental speaker adaptation was applied. We investigated a method for extracting topic-words from the speech recognition results that was based on a significance measure. This paper also proposes a new formulation for speech recognition/understanding systems, in which the a posteriori probability of a message that the speaker intends to address given an observed acoustic sequence is maximized. We applied the formulation to rescoring the recognition hypotheses.
  40 K 456 K
  Click on this paper's title (or here) to access HTML version.

• Human Language Technologies Evaluation in the European Framework, J. Mariani, P. Paroubek: LIMSI-CNRS.
  Several initiatives have been conducted in Europe on Human Language Technologies Evaluation, both for spoken and written language processing. Some have been supported within the programs of the European Commission, such as SQALE, DISC or EAGLES. Others have been conducted within national programs, such as the GRACE action at CNRS, or the Aupelf-Uref ARCs, or internationally (Senseval/Romanseval) . A proposal for installing a comparative technology-evaluation infrastructure in the Fifth Framework program has been discussed within the Telematics-LE ELSE project, and the conclusions have been submitted to the European Commission. The EU-US Multilingual Information Access and Management (MLIAM) common initiative may provide a possible framework for cooperative activities in this area.
  41 K 542 K
  Click on this paper's title (or here) to access HTML version.

New Directions

• Event99: A Proposed Event Indexing Task for Broadcast News, Lynette Hirschman, Erica Brown, Nancy Chinchor, Aaron Douthat, Lisa Ferro, Ralph Grishman, Patricia Robinson, Beth Sundheim: MITRE/SAIC/NYU/SPAWAR.
  The goal of the proposed Event99 task is to evaluate event-level indexing into news stories, including news wire, radio, and television sources. The Event99 task is distinguished from earlier, related evaluations in its focus on indexing, its application to multiple media, and the relative extensibility of its task definition to new event types. The task involves identifying instances of high-interest event types, along with some basic attributes of the events (e.g., actors, effects, date, location). The current guidelines include sample event definitions for death, natural disaster and bombing events, with other event types under development. The output from the task is primarily designed to support event-based browsing and search, but the human-prepared key also provides a kind of topic-specific summarization. This paper provides background concerning the task design, an overview of the definition, annotation and scoring of the task, results of an interannotator agreement study, and information concerning current status and schedule.
  32 K 297 K
  Click on this paper's title (or here) to access HTML version.

• Performance Measures for Information Extraction, John Makhoul, Francis Kubala, Richard Schwartz, Ralph Weischedel: BBN Technologies.
  While precision and recall have served the information extraction community well as two separate measures of system performance, we show that the F-measure, the weighted harmonic mean of precision and recall, exhibits certain undesirable behaviors. To overcome these limitations, we define an error measure, the slot error rate, which combines the different types of error directly, without having to resort to precision and recall as preliminary measures. The slot error rate is analogous to the word error rate that is used for measuring speech recognition performance; it is intended to be a measure of the cost to the user for the system to make the different types of errors.
  20 K 66 K
  Click on this paper's title (or here) to access HTML version.

Speech 3: 10x RT (incl. Baselines)

• The 1998 BBN BYBLOS Primary System Applied to English and Spanish Broadcast News Transcription, Spyros Matsoukas, Long Nguyen, Jason Davenport, Jay Billa, Fred Richardson, Manhung Siu, Daben Liu, Richard Schwartz, John Makhoul: BBN Technologies.
  In this paper, we describe the BBN BYBLOS system used for the 1998 Hub-4E primary and Hub-4Sp evaluation benchmarks, and discuss the improvements made to the system in 1998. We focus on the techniques that were new in this year's system, including processing of the acoustic training data, test segmentation, revised cepstral normalization and Vocal Tract Length Normalization (VTLN), band-specific models, Diagonal transform Speaker Adaptive Training (DSAT), and a modified ROVER method for system combination. We show that by combining all the above techniques, we were able to improve the recognition accuracy on the 1997 Hub-4E evaluation test by 27% relative to our 1997 system (from 20.4% to 14.8%). We also present our results on the 1998 Hub-4E and Hub-4Sp benchmarks, and discuss the differences between the English and Spanish transcription systems.
  49 K 70 K
  Click on this paper's title (or here) to access HTML version.

• The 1998 BBN BYBLOS 10x Real Time System, Jason Davenport, Long Nguyen, Spyros Matsoukas, Richard Schwartz, John Makhoul: BBN Technologies.
  In this paper we describe the BBN Byblos 10x real time system used for the 1998 Hub-4 English tests. Given our state of the art primary system running at 230 times real time (230 xRT) we show that eliminating and approximating many computationally expensive components speeds up the system by a factor of 23 with a relative loss in WER of 18%. This is accomplished without retraining or changing the primary system structure. The components of the primary system that are refined include segmentation, adaptation, decoding, cross-word rescoring with adaptation, and system combination. The time saving algorithms used include fast Gaussian computation, grammar spreading, nbest tree rescoring, and block diagonal adaptation.
  24 K 250 K
  Click on this paper's title (or here) to access HTML version.

• The 1998 HTK Broadcast News Transcription System: Development and Results, P.C. Woodland, T. Hain, G.L. Moore, T.R. Niesler, D. Povey, Andreas Tuerk, E.W.D. Whittaker: Cambridge Univ.
  This paper presents the development of the HTK broadcast news transcription system for the November 1998 Hub4 evaluation. Relative to the previous year's system The system a number of features were added including vocal tract length normalisation; cluster-based variance normalisation; double the quantity of acoustic training data; interpolated word level language models to combine text sources; increased broadcast news language model training data; and an extra adaptation stage using a full-variance transform. Overall these changes to the system reduced the error rate by 13% on the 1997 evaluation data and the final system had an overall word error rate of 13.8% for the 1998 evaluation data sets.
  45 K 65 K
  Click on this paper's title (or here) to access HTML version.

• The CUHTK-Entropic 10xRT Broadcast News Transcription System, Julian J. Odell, P.C. Woodland, T. Hain: Entropic Ltd./Cambridge Univ.
  This paper describes the development of the CUHTK-Entropic 10xRT Broadcast News Transcription System. Previous HTK broadcast news transcription systems have focused on maximising accuracy with few constraints on compute power available. In order to develop a system running in under 10 times real time on a single CPU, detailed investigation and optimisation of the system architecture and mode of operation was required. This paper outlines those developments and discusses the way in which operation under 10xRT was ensured despite variability of the data to be recognised. On the 1998 test the system produced an average word error rate of 16.1% running in 9.5xRT.
  50 K 106 K
  Click on this paper's title (or here) to access HTML version.

• Dragon Systems' 1998 Broadcast News Transcription System, Steven Wegmann, Puming Zhan, Ira Carp, Michael Newman, Jonathan P. Yamron, Larry Gillick: Dragon Systems.
  In this paper we shall describe key improvements to Dragon's Broadcast News Transcription System, which include: the addition of a speaker-change detection algorithm to our preprocessing subsystem, a new diagonalizing transformation trained using semi-tied covariances, and the addition of probabilities on pronunciations. This new transcription system yields a word error rate of 15.2% on the 1997 evaluation test data, and 14.5% on the 1998 evaluation test data.
  32 K 295 K
  Click on this paper's title (or here) to access HTML version.

• SRI's 1998 Broadcast News System - Toward Faster, Better, Smaller Speech Recognition, Ananth Sankar, Ramana Rao Gadde, Fuliang Weng: SRI International.
  We describe several new research directions we investigated toward the development of our broadcast news transcription system for the 1998 DARPA H4 evaluations. Our goal was to develop significantly faster and smaller speech recognition systems without degrading the word error rate of our 1997 system. We did this through significant algorithmic research creating various new techniques. A sample of these techniques was used to put together our 1998 broadcast news system, which is conceptually much simpler, faster, and smaller, but gives the same word error rate as our 1997 system. In particular, our 1998 system is based on a simple phonetically tied mixture (PTM) model with a total of only 13,000 Gaussians, as compared to a 67,000-Gaussian state-clustered system we used in 1997.
  67 K 115 K
  Click on this paper's title (or here) to access HTML version.