NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This release page is limited to the Current Test for the Arabic-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Results for submissions from GALE participants, who had previous access to the test data, will be reported separately from results submitted by those that are not part of the GALE program.

Participants

The following table lists the submissions received from the sites participating in the Arabic-to-English Current Test.

Site IDOrganizationLocationCurrent Test Set, Arabic-to-English
Single System TrackSystem Combination Track
amsterdamUniversity of AmsterdamNetherlandsYes(1)-
apptekAppTekUSAYes-
bbnBBN TechnologiesUSAYes-
cmu-smtCarnegie Mellon LTI interACTUSAYes-
cmu-statxferCarnegie Mellon StatXferUSAYes-
columbiaColumbia UniversityUSAYes-
cuedCambridge University Engineering DepartmentUKYes-
edinburghUniversity of EdinburghUKYes-
fbkFondazione Bruno KesslerItalyYes-
ibmIBMUSAYesYes
isi-lwUniversity of Southern California / Language Weaver Inc.USAYesYes
jhuJohns Hopkins UniversityUSALate and/or debugged submission-
kcslKCSL Inc.CanadaYes-
limsiLaboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur - CNRSFranceYes-
lium-systranUniversité du Maine (Le Mans) / SYSTRAN.Yes(1)-
rwthRWTH-Aachen University, Chair of Computer SciencesGermanyYes-
sakhrSakhr SoftwareEgyptYes-
sriSRI InternationalUSAYesYes
stanfordStanford UniversityUSAYes-
telavivTel Aviv UniversityIsraelYes-
tubitak-uekaeTUBITAK-UEKAETurkeyYes-
umdUniversity of MarylandUSAYes-
upc-lsiUPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)SpainYes-

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Arabic-to-English, submissions from participants involved in GALE (Table 1)

Site IDSystemBLEU-4 (mteval-v13a)
OverallNewswireWeb
Constrained Data Track
cuedCUED_a2e_cn_primary0.48340.56410.3960
stanfordstanford_a2e_cn_primary0.47810.56730.3843
isi-lwisi-lw_a2e_cn_primary0.47630.55900.3810
ibmIBM_a2e_constrained_primary0.47080.55470.3833
bbnBBN_a2e_cn_primary0.46800.55660.3783
rwthRWTH_a2e_cn_primary0.45340.54020.3538
sriSRI_a2e_cn_primary0.45270.53660.3634
edinburghEdinburgh_a2e_cn_primary0.44790.52400.3605
umdUMD_a2e_cn_primary0.44090.53400.3415
cmu-smtCMU-SMT_a2e_cn_primary0.43040.50550.3473
columbiacolumbia_a2e_cn_primary0.41570.49320.3331
cmu-statxferCMU-Stat-Xfer_a2e_cn_primary0.37740.44480.2986
sakhrSAKHR_a2e_cn_primary0.36810.41850.3147
UnConstrained Data Track
ibmIBM_arabic_un_primary0.49810.57130.4214
apptekAppTek_a2e_un_primary0.47900.51650.4352

Arabic-to-English, submissions from participants not involved in GALE (Table 2)

Site IDSystemBLEU-4 (mteval-v13a)
OverallNewswireWeb
Constrained Data Track
lium-systranLIUM-SYSTRAN_a2e_cn_primary(1)0.47730.56290.3800
fbkFBK_a2e_cn_primary0.45670.54180.3615
limsiLIMSI_Moses_a2e_cn_primary0.43840.52420.3471
tubitak-uekaeTUBITAK_a2e_cn_primary0.41120.48260.3310
upc-lsiUPC.LSI_a2e_cn_primary0.35880.43440.2778
amsterdamUvA_a2e_cn_primary(1)0.32210.38200.2565
kcslKCSL_a2e_cn_primary0.14220.16700.1161
telavivTLVEBMT_a2e_cn_primary0.07030.08720.0527

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - System Combination Track

Arabic-to-English, submissions from participants involved in GALE (Table 3)

Site IDSystemBLEU-4 (mteval-v13a)
OverallNewswireWeb
Constrained Data Track
isi-lwisi-lw_a2e_cn_combo10.48020.56000.3914
ibmIBM_a2e_cn_combo00.47750.56360.3871
sriSRI_a2e_cn_combo10.46310.54720.3731
UnConstrained Data Track
ibmIBM_a2e_un_combo00.50960.59130.4241

Current Test Results - Single System Track and System Combination Track - All metrics

Arabic-to-English, submissions from participants involved in GALE (Table 4)

Site IDSystemBLEU-4 (mteval-v13a)IBM BLEU (bleu-1.04)NIST (mteval-v13a)TER (tercom-0.7.25)METEOR (meteor-0.7)
OverallNewswireWebOverallNewswireWebOverallNewswireWebOverallNewswireWebOverallNewswireWeb
Constrained Data Track
cuedCUED_a2e_cn_primary0.48340.56410.39600.48330.56400.395911.0111.319.6030.44890.38610.50930.65700.71520.5999
isi-lwisi-lw_a2e_cn_combo10.48020.56000.39140.48010.55980.391310.8511.249.3960.46430.38870.53710.66420.73190.5969
stanfordstanford_a2e_cn_primary0.47810.56730.38430.47770.56680.384010.9711.449.3920.43990.37090.50650.65140.71530.5882
ibmIBM_a2e_cn_combo00.47750.56360.38710.47730.56340.387011.0311.549.4660.43940.37130.50510.65260.71420.5924
isi-lwisi-lw_a2e_cn_primary0.47630.55900.38100.47600.55880.380810.8511.309.2920.45900.38260.53280.65440.72330.5864
ibmIBM_a2e_constrained_primary0.47080.55470.38330.47070.55450.383110.9711.469.4060.44240.37730.50510.64780.70910.5876
bbnBBN_a2e_cn_primary0.46800.55660.37830.46780.55640.378110.8511.469.3040.46030.38000.53780.65610.71250.6015
sriSRI_a2e_cn_combo10.46310.54720.37310.46290.54700.373010.8611.259.3060.45920.39740.51890.64610.70630.5866
rwthRWTH_a2e_cn_primary0.45340.54020.35380.45330.54000.353710.6511.129.0280.46660.39800.53280.65320.71540.5918
sriSRI_a2e_cn_primary0.45270.53660.36340.45260.53650.363310.7111.119.2510.46890.40030.53510.64590.70710.5859
edinburghEdinburgh_a2e_cn_primary0.44790.52400.36050.44780.52380.360410.5810.929.1740.48570.42090.54820.64540.70690.5847
umdUMD_a2e_cn_primary0.44090.53400.34150.44080.53380.341310.5311.258.5900.45910.39090.52480.62870.69820.5595
cmu-smtCMU-SMT_a2e_cn_primary0.43040.50550.34730.43020.50530.346910.3310.728.8960.48230.41820.54420.63650.69880.5748
columbiacolumbia_a2e_cn_primary0.41570.49320.33310.41560.49310.333010.2710.738.7850.47470.41600.53130.62690.68470.5698
cmu-statxferCMU-Stat-Xfer_a2e_cn_primary0.37740.44480.29860.37720.44470.29849.73110.078.3990.51580.46280.56690.60820.66840.5484
sakhrSAKHR_a2e_cn_primary0.36810.41850.31470.36800.41840.31479.8679.9828.8870.50750.46260.55080.63200.67370.5913
UnConstrained Data Track
ibmIBM_a2e_un_combo00.50960.59130.42410.50950.59120.424011.5411.9210.020.41680.35070.48040.67680.73660.6181
ibmIBM_arabic_un_primary0.49810.57130.42140.49790.57110.421411.4111.709.9730.42530.36530.48330.67000.72930.6117
apptekAppTek_a2e_un_primary0.47900.51650.43520.47870.51620.434811.2110.9810.320.43420.39690.47020.68180.72070.6433

Arabic-to-English, submissions from participants not involved in GALE (Table 5)

Site IDSystemBLEU-4 (mteval-v13a)IBM BLEU (bleu-1.04)NIST (mteval-v13a)TER (tercom-0.7.25)METEOR (meteor-0.7)
OverallNewswireWebOverallNewswireWebOverallNewswireWebOverallNewswireWebOverallNewswireWeb
Constrained Data Track
lium-systranLIUM-SYSTRAN_a2e_cn_primary(1)0.47730.56290.38000.47720.56270.379910.9611.389.4120.45650.38510.52520.65260.71850.5879
fbkFBK_a2e_cn_primary0.45670.54180.36150.45650.54170.361310.7511.229.2520.47210.39520.54620.65330.71700.5910
limsiLIMSI_Moses_a2e_cn_primary0.43840.52420.34710.43830.52400.346910.4010.988.7380.47240.40510.53730.62120.68510.5582
tubitak-uekaeTUBITAK_a2e_cn_primary0.41120.48260.33100.41210.48270.331210.0110.438.6590.51290.44450.57890.62590.68900.5627
upc-lsiUPC.LSI_a2e_cn_primary0.35880.43440.27780.35880.43450.27779.40410.057.6550.51880.46300.57250.58660.65150.5216
amsterdamUvA_a2e_cn_primary(1)0.32210.38200.25650.32180.38160.25638.6219.0637.4580.59950.54370.65330.58630.64570.5278
kcslKCSL_a2e_cn_primary0.14220.16700.11610.14200.16690.11586.6476.9595.8200.65900.63530.68180.49370.53980.4482
telavivTLVEBMT_a2e_cn_primary0.07030.08720.05270.07030.08720.05263.8794.3703.1650.74830.72740.76850.38530.42620.3450

(1)A late and/or debugged system was also submitted, not reported here.