<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> NIST Speech Group Website
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • Rich Transcription 2009 Evaluation

    The Rich Transcription Evaluation will be held during the Spring of 2009. The evaluation will build on the previous RT evaluation held in 2007. If you'd like to join the RT community and/or participate in the evaluation, send your inquiries to rteval@nist.gov.

    Evaluation Planning

    The second verision of the RT-09 Evaluation Plan (v2) is now available as well as a Changes from v1 version.

    We continue the teleconferences as needed. The notes from previous telecons is below.

    RT-09 Workshop

    The RT-09 Evaluation Workshop will take place May 28-29, 2009 at the Florida Institute of Technology. Workshop attendance is limited to invities and evaluation participants. The workshop information page has moved and now includes the registration fee, agenda, and accomodations. Workshop attendance is limited to participants/invited guests.

    The username and password for the evaluation results has been sent to the RT participants. Contact NIST for the password if you need it. For travel planning purposes, plan on a full 2-day of meeting.

    Contributing Conference Room Meeting Recordings for the Evaluation Set

    Each year members of the RT community contributed meeting recordings to form the conference room test set.  If you are in a position to donate recordings, here are the requirements:
    • The data must be delivered to NIST by December 13, 2008
    • The meetings must be from the conference room meeting sub-domain which means: there must be 2-12 meeting participants and the meeting should be goal-oriented.
    • Each participant must wear a close talking, cardiod or super-cardiod microphone.
    • There must be at least 3 table top microphones located between the particiapants, one of which should be designated as the most-central microphone.
    • There must be at least one digital video stream.  NIST will distributed mpeg-2, NTSC, 30 frames-per-second, 12 or 15 I-frames per second.  If you are not collecting this kind of video, the video must be of sufficient quality to trans-code it to the distribution format.
    • The microphone recordings and video recordings must be synchronized.  If the data collection infrastructure can not provide the synchronization, the meeting recordings must include directors "claps" before the meeting begins and after it ends using a modified directors clapboard that includes a simultaneous flash.  The paper "Synchronizing Multimodal Data Streams Acquired Using Commodity Hardware" (Michel, Stanford) contains a description of how the director's clapboard is used by NIST to synchronize its audio and video streams.  A modified clapboard is easy to make from the pictures of NIST's clapboard.

    Evaluation Agreements and Data Licenses

      NIST distributes the conference meeting data sets. There the are two agreements/licenses needed to obtain the data for the evaluation:

      ELDA distributes the lecture meeting data sets used for past RT evaluations.

    Data Resources

      Participants of the evaluation are given access to RT data resources. The password protected page contains textual resources for previous year's evaluation corpora and links/pointers to other resources. Contact NIST for details.

    Software

    • The latest version of SCTK can be found on the NIST tools page.  This package includes support for 2007 tasks.
    • SASTT-kit-v2 The starter kit for SASTT scoring. 

    RT-09 Schedule (Tentative)

    Date Milestone Description
    Feb 27, 2009 Signed Commitment to Participate
    Mar 2, 2009 Evaluation data released
    April 9, 2009 Submissions due 17:00 GMT
    April 16, 2009 Results released
    May 19, 2009 Electronic copies of papers and presentations due
    May 28-29, 2009

    Rich Transcription Evaluation and Evaluation Workshop

    FIT Melbourne, Florida

     

     

     

    Page Created: Nov. 15, 2007
    Last Updated: May 11, 2009

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA