NIST websiteMT08 Online Evaluation Platform

Guidelines for making preference judgments

The Preference judgments are concerned with finding the better Machine translation out of two, compared to a Reference translation.

You will be presented with a Reference translation of a sentence followed by two Machine translations of the same sentence. Answer the question: Which machine translation do you prefer?

Note that we are not distinguishing between adequacy and fluency for the Preference assessments. Use your best judgment to determine which Machine translation is the overall better representation of the Reference translation.

Choose one translation whenever possible. In cases where you truly do not have a preference between two translations, select No preference.

Make your selection by clicking the appropriate radio button, then click on Proceed to the next comparison to go to the next item.

Example:

Reference translation:

Erdogan Confirms Turkey Will Resist Any Pressure to Recognize Cyprus

System translations:

Which translation do you prefer ?

Ardogan Confirms That Turkey Any Pressure, Urging Them to Recognizing the Cyprus

Ardogan Stresses That Would Reject Any Pressure on Turkey to Urge Recognition of Cyprus

I prefer this translation
I prefer this translation

No preference