NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This page of results is limited to those of the Progress Test data for the MT09 tests of Arabic-to-English and Chinese-to-English text-to-text translation.

The Progress test was designed as a means to demonstrate true system improvement for a particular site over time, irrespective of inherent data differences that come with each new test set. In order to keep the focus on progress over time, only results from sites that participated in both OpenMT08 and OpenMT09 are reported.

There may be issues related to processing older test sets that make across-site comparisons less meaningful.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Note that all BLEU-4 scores reported on this page were computed using version 13a of the mteval script. Last year, scores were computed using mteval-v11b, which uses a different brevity penalty. This year's script (mteval-v13a) uses the 'closest reference translation length' (same as in IBM BLEU) instead of the 'shortest reference translation length', to compute the brevity penalty.

Participants

The following table lists the submissions received from the sites participating in the Progress Test. Cells in bold indicate submissions for which scores are reported:

• Submissions were valid and on-time
• The site also participated in MT08
• The MT08 system was submitted for the same training condition as the OpenMT 2009 system.

Progress Test Results - Single System Track

Arabic-to-English (Table 1)

⁽¹⁾The MT08 system was a combined system

Chinese-to-English (Table 2)

⁽¹⁾The MT08 system was a combined system