<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> GALE Phase 3 Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • NIST Machine Translation Evaluation for GALE
    Phase 3 / Phase 3.5

    The GALE Translation evaluation will test machine translation of text and recorded speech data. The test will include language data from both Arabic and Chinese, with system performance tallied separately for each language and separately for text and recorded speech sources.

    GALE contractors will be the only participants in this evaluation, and the participants must meet specific Go/No-Go levels of performance. This page provides information regarding the 2008 GALE Phase 3 / Phase 3.5 Translation evaluations.

    Documentation

    Software

    For P3/P3.5 evaluation, we will the following software:

    • BBN/UMD-created Java scoring software v0.7.2. This code can be obtained here or directly from the author's website at http://www.cs.umd.edu/~snover/tercom (link updated 06/06/2008)

      * Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul, "A Study of Translation Edit Rate with Targeted Human Annotation," Proceedings of Association for Machine Translation in the Americas, 2006.

    • Post-editing software v1.1.2. NIST is developing the post editing software package using Java. Our working environment is NETBEANS IDE 5.0 on MAC OS X. Choose to download the MTPostEditor NETBEANS Java Project or the MTPostEditor JAR file (link updated 06/09/2008)

    Schedule

    Dates Event
    Nov-01-06 to Dec-22-06 P2/P2.5 evaluation epoch
    Jun-01-07 to Jun-30-07 P3/P3.5 evaluation epoch

    Site collected data originated within P2/P2.5/P3/P3.5 epochs are off-limit to training.

    LDC collected data originated within the P3/P3.5 epoch or overlap in content with the P2.5 sequestered data are off-limit to training.

      P3
    Jun-30-08 GALE Arabic translation evaluation starts
    Jul-31-08 Translations of text and audio due at NIST
    Aug-18-08 Post-editing begins
    Sep-29-08 Post-editing ends
    Sep-30-08 Final scores to DARPA
      P3.5
    Nov-17-08 GALE Out-of-Sync Arabic and Chinese translation evaluation begins
    Dec-08-08 Translation of text due at NIST
    Dec-15-08 Translation of audio due at NIST
    Dec-22-08 Post-editing begins
    Mar-01-08 Post-editing ends
    Mar-04-08 Final scores to DARPA
    TBD GALE PI meeting

     

    [ GALE Home ]

     

     

    Page Created: December 13, 2006
    Last Updated: December 16, 2009

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA