<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> NIST Open Machine Translation (OpenMT) Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Information Group Home

  • Benchmark Tests

  • Tools

  • Test Beds

  • Publications

  • Links

  • Contacts
    •

    NIST Open Machine Translation
    2009 Evaluation (MT09)

    MT09 took place in June 2009 in accordance with the NIST Open Machine Translation 09 Evaluation Plan (v2d, March 23 2009).

    Highlights

    • Language pairs and data sets:
      • Arabic to English (Current test, Progress test)
      • Chinese to English (Progress test)
      • Urdu to English (Current test)
    • Training conditions:
      • Constrained training
      • Unconstrained training
    • New! System categories:
      • Single System
      • System Combination

    Schedule

    Training data off-limits periods. All data created, posted, or published during these periods is off-limits for system training and development.
    February 24 2009
    		 Evaluation Plan released.
    May 14 2009
    		 Registration deadline. See Documentation for required forms.
    June 8 2009
    9:00am EDT
    		 Evaluation test data e-mailed to participants.
    June 12 2009
    12:00 noon EDT
    		 Deadline for submission of results to NIST.
    June 17 2009
    		 Distribution of system translations to Informal System Combination track participants.
    June 19 2009
    		 Preliminary release of results to participants.
    June 26 2009
    		 Deadline for submission of Informal System Combination track results to NIST.
    June 26 2009
    		 Deadline for submitting system descriptions to NIST.
    July 6 2009
    		 Human assessment tasks available.
    August 15 2009
    		 Deadline for finishing human assessments.
    August 31 - September 1 2009
    		 Evaluation workshop, open to participants and government sponsors only, held in Ottawa, ON, Canada (co-located with MT Summit XII).
    October 30 2009
    		 Official public release of results.

    Documentation

    Software

    • mteval-v13a.pl
      • Release date: October 1 2009
      • Modified the scoring functions to prevent division-by-zero errors when a system segment is empty; affected methods: 'bleu_score' and 'bleu_score_smoothing'
    • mteval-v13.pl
      • Release date: March 23 2009
      • Calculates BLEU with original calculation of brevity penalty by default, BLEU with NIST's calculation of brevity penalty optionally
    • mteval-v12.pl
      • Release date: January 28 2008
      • Tokenization based on UTF-8 categories
    • mteval-v11b.pl
      • Release date: May 20 2005
      • Digits join was removed from the text normalization process
    • splitUTF8Characters.c
      • Release date: November 23 2007
      • Version 1.1: C program that will enclose any non-ASCII (UTF8-encoded) character between two spaces
    • splitUTF8Characters.pl
      • Release date: November 23 2007
      • Version 1.0: Perl script that will enclose any non-ASCII (UTF-8 encoded) character between two spaces

    Joining the NIST OpenMT Evaluation Community

    You can subscribe to the general OpenMT mailing list hosted by NIST, mt_list@nist.gov, by sending e-mail to listproc@nist.gov; put "subscribe mt_list" in the body.

    [ MT Home ]

     

     

    Page Created: January 7, 2009
    Last Updated: October 27, 2009

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
         		Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA