<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> RT 2007 Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • Rich Transcription 2007 Evaluation


    The Rich Transcription 2007 Meeting Recognition evaluation was held during the Spring of 2007 in conjunction with the 2007 Classification of Events, Activities and Relationships (CLEAR) Evaluation and Evaluation Workshop.

    The RT evaluation series is a collaborative effort of several projects: The Augmented Multi-Party Interaction (AMI) program, the Computers In the Human Interaction Loop (CHIL), the NIST Speech Group, and the Video Analysis and Content Exploitation (VACE) program.  The RT evaluation was the 5th open evaluation of technologies for the meeting domain.  The presentations and preliminary system description papers from the RT Workshop held May 10-11, 2007 can be found on the Agenda from the meeting.

    We are currently beginning to plan the next RT evaluation.  If you'd like to join the RT community and/or participate in the evaluation, send your inquiries to rteval@nist.gov.

    Evaluation Planning

    The RT-07 Evaluation Plan-V2 was released 2007.2.15. The changes compared to V1 include: coffee breaks added as a new sub domain, the IHM reference segmentation condition was added , and clarifications to the LDC corpora resources.

    We meet via conference calls to discuss topics.  The next call is scheduled for December 5, 2006 at 11:00 EST.  The minutes of the conference calls are in the following table:
    • Minutes of the Tuesday, October 17, 2006 conference call
    • Minutes of the Tuesday, November 14, 2006 conference call
    • Minutes of the Tuesday, December 5, 2006 conference call
    • Minutes of the Thusday, December 21, 2006 conference call
    • Minutes of the Friday, January 19, 2007 conference call

    Contributing Conference Room Meeting Recordings for the Evaluation Set

    Each year members of the RT community contributed meeting recordings to form the conference room test set.  If you are in a position to donate recordings, here are the requirements:
    • The data must be delivered to NIST by December 13, 2006
    • The meetings must be from the conference room meeting sub-domain which means: there must be 2-12 meeting participants and the meeting should be goal-oriented.
    • Each participant must wear a close talking, cardiod or super-cardiod microphone.
    • There must be at least 3 table top microphones located between the particiapants, one of which should be designated as the most-central microphone.
    • There must be at least one digital video stream.  NIST will distributed mpeg-2, NTSC, 30 frames-per-second, XX I-frames per second.  If you are not collecting this kind of video, the video must be of sufficient quality to trans-code it to the distribution format.
    • The microphone recordings and video recordings must be synchronized.  If the data collection infrastructure can not provide the synchronization, the meeting recordings must include directors "claps" before the meeting begins and after it ends using a modified directors clapboard that includes a simultaneous flash.  The paper "Synchronizing Multimodal Data Streams Acquired Using Commodity Hardware" (Michel, Stanford) contains a description of how the director's clapboard is used by NIST to synchronize its audio and video streams.  A modified clapboard is easy to make from the pictures of NIST's clapboard.

    Evaluation Agreements and Data Licenses

      Participants of the RT-07 evaluation must complete and return to NIST the RT-07 Evaluation Participation Agreement and the CHIL Data use license. Participants must also complete the AMI license which is available online at the URL corpus.amiproject.org. There are evaluation agreement and CHIL license documents that must be downloaded:

      No training data or evaluation data will be distributed until NIST receives the above forms.

    Data Resources

      Participants of the evaluation are given access to RT data resources. The password protected page contains textual resources for previous year's evaluation corpora and links/pointers to other resources.  Participants may be eligble to recieve selected  LDC corpora for free.  Contact NIST for details.

    Software

    • SCTK v2.2.4 20070501-1541 The latest version of SCTK.  This package includes support for the SASTT task and now includes the rttmSmooth.pl, rttmSort.pl, and md-eval.pl scripts.
    • SASTT-kit-v2 The starter kit for SASTT scoring. 
    • spkr2sad-v1.pl Convert an RTTM reference file for the Speaker Diarization task (or system output) to a Speech Activity Detection RTTM file. The output must be smoothed with rttmSmooth.pl to remove the overlapping segments.

    RT-07 System Description Papers and Workshop Presentations

    RT Participants are expected to prepare a publication-quality system description paper for each task they participated in and presenters are expected to provided PDFs of their slides for the workshop.  The papers and presentations will be made available online during the RT-workshop and then later on the RT website. The submission deadline for papers is May 3, 2007  and the deadline for submitting PDFs of your presentations (two slides per page) is May 7, 2007.

    A workshop proceedings will be published by Springer under the LNCS bookseries jointly with CLEAR. Therefore we ask you to submit revised final versions of your papers after the workshop. The tentative deadline will be June 1, 2007. A copy of the proceedings will be send to each workshop participant.

    Please use the Springer LNCS style to format your paper. A style file for Latex2e can be found here. Further style files for Latex, Word and Framemaker can be found on Springer's website.

    Paper Length: You may submit a paper of up to 12 pages for every evaluation task in which you participated.

    Please send your papers and presentations to Jerome Ajot (ajot at nist dot gov).

    RT-07 Schedule

    The table below lists the approximate dates for the major milestones in the evaluation.  The CLEAR and RT organizers have developed a joint CLEAR/RT evaluation schedule
    Date
    Milestone Description
    June 1, 2007
    RT papers due to NIST
    June 29, 2007
    RT Reviews final
    July 18, 2007
    Final RT papers due
    [ RT Home ]

     

     

    Page Created: Fall, 2006
    Last Updated: November 4, 2008

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA