<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> RT Spring 2006 Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • Rich Transcription Spring 2006 Evaluation

    The Rich Transcription 2006 Spring Meeting Recognition evaluation will be the fourth NIST sponsored evaluation of speech technologies within the meeting domain.  This year's evaluation will build on the success of the RT-04 Spring and RT-05 Spring.  The evaluation is related to the Classification of Events, Activities and Relationships (CLEAR) Evaluation in that both communities work on multimodal signals and sensor fusion experiments in the meeting domain.  The RT-06S evaluation workshop  will be held May 3-4, 2006 and will  immediately follow the MLMI 2006 Workshop at the same venue (See below for registration information).

    The RT-06S evaluation included three meeting domain tasks in the evaluation: (1) Speech-To-Text (STT), Diarization (2) "Who Spoke When" (SPKR also known as "Speaker Diarization"), and (3) "Speech Activity Detection" (SAD).  The evaluation plan describes the tasks and evaluation metrics. 

    A new evaluation task, Text Recognition in Video, will be added this year.  Unlike the STT, SPKR, and SAD tasks, the Text Recognition tasks will make use of Broadcast News data. 

    In order to participate in the evaluation, you must complete evaluation agreement and all licenses as specified below.

    Evaluation Plan 

    The evaluation plan is now available from the link below. 

    Data Resource

    The following data resources are available to evaluation participants.  The page is password protected.   Contact NIST for the password.

    Participation Agreement and Licenses

    Participants of the RT-06S evaluation must complete and return to NIST the RT-06S Evaluation Participation Agreement as well as the AMI, CHIL and LDC data licensing agreements. There are two documents that must be downloaded, printed, signed and faxed back to NIST. The documents are:

    The evaluation participation agreement contains the fax number and address for mailing the forms if a fax is unavailable. No training data or evaluation data will be distributed until NIST receives the above forms.

    Software

    • spkr2sad-v1.pl Convert an RTTM reference file for the Speaker Dairization task (or system output) to a Speech Activity Detection RTTM file. The output must be smoothed with rttmSmooth.pl to remove the overlapping segments.
    • rttmSmooth-v2.pl Smooth the 'SPEAKER' RTTM objects using a 0.3 second smoothing window and removes any overlapping segments.
    • rttmSort-v1.pl Sorts RTTM records leaving the SPKR-INFO fields sorted to the top and the time-based records below.
    • SCTK 2.1.1 The latest version of SCTK.
    • md-eval-v21.pl The diarization evaluation tool.

    RT-06S Evaluation Workshop and MLMI-06 Workshop

    The RT-06S evaluation workshop will occur beginning May 3, 2006 after lunch and continue through 6:00 pm May 4, 2006.  The RT-06S evaluation workshop page contains the registration and hotel information.

    The evaluation workshop will be proceeded by the MLMI-06 Workshop beginning May 1, 2006 and ending before lunch on May 3rd.   MLMI registration and hotel accommodations are handled separately through the MLMI-06  website.

    RT-06S Schedule (this schedule is subject to change)

    Date
    Milestone Description
    Jan. 17, 2006
    Conference call to ratify evaluation plan: 11:00 am EST
    Feb. 10, 2006
    Hard commitment to participate
    Mar. 1, 2006
    Evaluation data distributed to sites
    Mar. 22, 2006
    Conference result due to NIST 5:00 pm EDT
    Mar. 29, 2006
    Conference data results posted by NIST
    Lecture data results due to NIST 5:00 pm EDT
    April 3, 2006
    Lecture data results posted by NIST
    April 20, 2006
    System description papers and presentations due
    May 1-3, 2006
    MLMI 2006 in Washington DC
    May 3-4, 2006
    RT-06S Evaluation Workshop in Washington DC

    Conference Call Minutes

    [ RT Home ]

     

     

    Page Created: September, 2006
    Last Updated: November 4, 2008

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA