NIST Metrics for Machine Translation Challenge
NIST coordinates MetricsMATR, a series of research challenge events
for machine translation (MT)
metrology, promoting the development of innovative, even revolutionary, MT metrics. MetricsMATR
focuses entirely on MT metrics.
NIST provides the evaluation infrastructure, the source files being MT system output
The participants develop MT metrics to assess the quality of the source files
The goal is to create intuitively interpretable automatic metrics which correlate highly with human assessment of MT quality.
Motivation and Goals
There are several drawbacks to the current methods employed for the evaluation of MT technology:
- Automatic metrics have not yet been proved able to predict the usefulness and reliability
of MT technologies with respect to real applications with confidence.
- Automatic metrics have not demonstrated that they are meaningful in target languages
other than English.
- Human assessments are expensive, slow, subjective, and difficult to standardize.
These problems, and the need to overcome them through the development of improved automatic
(or even semi-automatic) metrics, have been a constant point of discussion at past NIST MT
MetricsMATR aims to provide a platform to address these shortcomings. Specifically, the goals of MetricsMATR are:
- To inform other MT technology evaluation campaigns and conferences with regard to improved metrology.
- To provide an infrastructure that encourages the development of innovative metrics.
- To build a diverse community which will bring new perspectives to MT metrology research.
- To provide a forum for MT metrology discussion and for establishing future directions of MT metrology.
The MetricsMATR challenge is designed appeal to a wide and varied audience including
researchers of MT technology and metrology, acquisition programs such as MFLTS,
and commercial vendors. We welcome submissions from a wide range of disciplines including
computer science, statistics, mathematics, linguistics, and psychology.
NIST encourages submissions from participants not currently active in the field of MT.
Current and Recent NIST MetricsMATR Activities
The most recent NIST MetricsMATR challenge was MetricsMATR08.
You can subscribe to the MetricsMATR mailing list hosted by NIST,
by sending e-mail to
Include "subscribe metricsmt_list" in the body.
Find more information (including registration and documentation for the current challenge) by clicking a specific year in the tabs below.
Contact firstname.lastname@example.org for more general information or to pose related questions.
Page Created: January 29, 2008
January 21, 2010