NIST Speech Group Website

Information Technology Lab, Information Access Division

NIST: National Institute of Standards and Technology

Multimodal Information Group Home

Benchmark Tests

2001 Speaker Recognition Evaluation

The 2001 NIST Speaker Recognition evaluation is part of an ongoing series of yearly evaluations conducted by NIST. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. They are intended to be of interest to all researchers working on the general problem of text independent speaker recognition. To this end the evaluation was designed to be simple, to focus on core technology issues, to be fully supported, and to be accessible.

Evaluation Documentation

Updated 03/01/01 (version 05.9):

Modification to support model identification with the extended data task.

The Evaluation Plan for the 2001 NIST Speaker Recognition Evaluation is available in postscript and pdf format.

See the paper "Some Experiments on Idiolectal Differences among Speakers (in postscript and pdf format) for motivation behind the extended training task.

Supporting documentation for the evaluation test material, NIST Speech Discs r65_1_1 through r65_9_1.

Protocol for submitting system results

Supporting Ahumada DevTest Documentation

MIT-LL has provided a table documenting handset label type and probability for the Ahumada developtment data Format per line:

< basefilename > MALE 1 < HS > Pr(HS)

HS := CARB|ELEC

Schedule

February 24, 2001 - Deadline for signing up to participate in the evaluation
March 12, 2001 - Evaluation data to arrive at participating sites

April 9, 2001 - Evaluation results due at NIST

April 30, 2001 - Deadline for registration and payment for the Speaker Recognition Evaluation Workshop.

May 14-15, 2001 - Two day Speaker Recognition Evaluation Workshop.

Evaluation Data

At stated in the evaluation plan, NIST has created new index files to be used for the one speaker detection task. These index files are a super set of the trials used in the 2000 evaluation and were augmented to help with analysis in areas of interest.

While the majority of the test segments are still tested against 11 putative models, some test segments will now be tested against up to 48 putative models.

These index files will be made available below on March 12th, 2001.

[Now Available] :: One file, all updated one-speaker detection trials (male & female).

-OR-

[Now Available] :: Index file for CD #3, Female data
[Now Available] :: Index file for CD #4, Male data
[Now Available] :: Index file for CD #5, Female data
[Now Available] :: Index file for CD #5, Male data

Resources for the extended data task are now available.

The development and evaluation data used for this evaluation is provided free of charge. To obtain these data, participants are required to sign limited license agreements which restrict the use of the data for this evaluation:

Participants which are not members of the LDC must complete the LDC's License Agreement and return it to the LDC

Participants performing the optional one-speaker detection task using the Spanish Ahumada data must complete the Ahumada License Agreement (Word Document, Postscript, or PDF) and return it to:

Dr. Javier Ortega-Garcia
EUIT Telecomunicacion - UPM
Ctra. Valencia km. 7
28031 Madrid
SPAIN

Scoring Software

Modified scoring software for the tasks of tracking and segmentation is now available. This version-0 of the scoring software is designed to work with the development test sets. The official software will be a more versatile package, and will be released with the results.

The generic segmentation scoring script which was used to evaluate the segmentation tasks of this evaluation, is now available.

The evaluation keys are also available: main key, cell Key

[ SRE Home ]

Page Created: Month Day, Year
Last Updated: November 4, 2008

Multimodal Information Group is part of IAD and ITL
NIST is an agency of the U.S. Department of Commerce