<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> NIST Metrics for Machine Translation Challenge (MetricsMATR)
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology

  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • NIST Metrics for Machine Translation Challenge


    NIST coordinates MetricsMATR, a series of research challenge events for machine translation (MT) metrology, promoting the development of innovative, even revolutionary, MT metrics. MetricsMATR focuses entirely on MT metrics.

  • NIST provides the evaluation infrastructure, the source files being MT system output
  • The participants develop MT metrics to assess the quality of the source files
  • The goal is to create intuitively interpretable automatic metrics which correlate highly with human assessment of MT quality.

    Motivation and Goals

    There are several drawbacks to the current methods employed for the evaluation of MT technology:

    • Automatic metrics have not yet been proved able to predict the usefulness and reliability of MT technologies with respect to real applications with confidence.
    • Automatic metrics have not demonstrated that they are meaningful in target languages other than English.
    • Human assessments are expensive, slow, subjective, and difficult to standardize.

    These problems, and the need to overcome them through the development of improved automatic (or even semi-automatic) metrics, have been a constant point of discussion at past NIST MT evaluation events.

    MetricsMATR aims to provide a platform to address these shortcomings. Specifically, the goals of MetricsMATR are:

    • To inform other MT technology evaluation campaigns and conferences with regard to improved metrology.
    • To provide an infrastructure that encourages the development of innovative metrics.
    • To build a diverse community which will bring new perspectives to MT metrology research.
    • To provide a forum for MT metrology discussion and for establishing future directions of MT metrology.

    Target Audience

    The MetricsMATR challenge is designed appeal to a wide and varied audience including researchers of MT technology and metrology, acquisition programs such as MFLTS, and commercial vendors. We welcome submissions from a wide range of disciplines including computer science, statistics, mathematics, linguistics, and psychology. NIST encourages submissions from participants not currently active in the field of MT.

    Current and Recent NIST MetricsMATR Activities

    The most recent NIST MetricsMATR challenge was MetricsMATR08.

    You can subscribe to the MetricsMATR mailing list hosted by NIST, metricsmt_list@nist.gov, by sending e-mail to listproc@nist.gov. Include "subscribe metricsmt_list" in the body.

    Find more information (including registration and documentation for the current challenge) by clicking a specific year in the tabs below.

    Contact mt_poc@nist.gov for more general information or to pose related questions.

    [ 2008]



    Page Created: January 29, 2008
    Last Updated: January 21, 2010

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA