Introduction

Thank you for your interest in the human assessments of Machine Translation technologies for the NIST Open Machine Translation 2008 Evaluation (MT08). Please read this introduction to help to determine whether you would like to participate in the human assessments.

What is Machine Translation?

Machine Translation (MT) aims at automatically translating text from one language (the source language) into another (the target language). The goal is for the translated output to adequately represent the meaning of the original. Assessments of the quality of MT output by human judges are an important aspect of evaluating MT.

Do I need to know any foreign language to do the assessments?

No, you only need to be a native or near-native speaker of English. The target language is always English. In order to not need judges to know both the target and the source language, correct human translations of the source language sentences into English have been produced. These are called Reference translations. You will be assessing only English text, comparing Machine translations to Reference translations.

How are the human assessments for MT08 done?

You will perform multiple judgments of Adequacy, and multiple judgments of Preference. See a sample of each type below:

1. Adequacy judgment: determines how adequately the meaning of a Reference translation is represented in the Machine translation.

Reference translation:

Erdogan Confirms Turkey Will Resist Any Pressure to Recognize Cyprus

System translation:

Ardogan Stresses That Would Reject Any Pressure on Turkey to Urge Recognition of Cyprus

How much of the meaning expressed in the Reference translation is also expressed in the System translation?
All Half None	Does the Machine translation mean essentially the same as the Reference translation? Yes No






2. Preference judgment: determines which of two Machine translations is a better representation of the Reference translation.

Reference translation:

Erdogan Confirms Turkey Will Resist Any Pressure to Recognize Cyprus

System Translations:

Which translation do you prefer ?

Ardogan Confirms That Turkey Any Pressure, Urging Them to Recognizing the Cyprus	Ardogan Stresses That Would Reject Any Pressure on Turkey to Urge Recognition of Cyprus
I prefer this translation	I prefer this translation


No preference