<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> RT Fall 2003 Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • Rich Transcription Fall 2003 Evaluation

    The Rich Transcription Fall 2003 Evaluation (RT-03F), implemented in October 2003, focused on MDE tasks. The MDE tasks included "Who Said What" speaker diarization, SU detection, and disfluencies detection for English broadcast news speech and conversational telephone speech. The RT-03F evaluation plan can be found in the Documentation section. The transcription and annotation guidelines are also given in the Documentation section below.

    Documentation

    • RT Fall 2003 Evaluation Plan, v9, 10/09/03 [pdf, doc]
    • RT Transcription Guidelines, v2.2, 2/23/03 [pdf, doc]
    • Simple Metadata Annotation Specification, v5.0, 5/15/03 [pdf, doc]
    • RTTM Format Specification, v13, 9/17/03 [pdf, doc]

    Dry Run Disc Documentation

    Evaluation Disc Documentation

      rt03f-evaldiscdoc - contains data needed to run the evaluation, also contains the evaluation results (password protected)

    Format Conversion Software

    • ctm+mdtm-to-rttm-v03.pl - converts CTM+MDTM into RTTM format
    • rttm_to_ctm+mdtm_v02.pl - converts RTTM into CTM+MDTM. This script is no longer being supported.
    • ag-to-rttm+uems-v15.pl - converts LDC's AG format into RTTM format and generates the rt03f_mde and the spkr scoring-uem files
      • Note: the -I option in the ag-to-rttm+uems script now makes unnecessary the task formerly done by the ag-to-rttm-wrapper-v01.pl script

    Scoring Software

    • df-eval-v13.pl, 10/1/03 - scores the system's performance on disfluency detection
    • su-eval-v15.pl, 10/7/03 - scores the system's performance on SU detection
    • rt_eval_v2.3.zip, 10/6/03 - alternative scoring software developed by BBN to score the system's performance on disfluency detection, SU detection, speaker attributed STT, and rich transcription.

    Workshop Presentations

    Update History

    • December 17, 2003: Added link to page containing presentations given at the RT-03F Workshop.
    • October 22, 2003: Updated link to the results page tar file. See the results page "Update History" for details.
    • October 21, 2003: Updated versions of df-eval (v13) and su-eval (v15) from George Doddington; indicated rt03f-evaldiscdoc now contains the evaluation results
    • October 10, 2003: Updated link to latest ctm+mdtm-to-rttm-v03
    • October 9, 2003: Latest eval plan: v9; latest scoring tools: df-eval-v12, su-eval-v12, rteval_v2.3; latest format conversion tools: ctm+mdtm-to-rttm-v02, rttm_to_ctm+mdtm_v02, ag-to-rttm-v15
    [ RT Home ]

     

     

    Page Created: Month Day, Year
    Last Updated: November 4, 2008

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA