NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This release page is limited to the Current Test for the Arabic-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Results for submissions from GALE participants, who had previous access to the test data, will be reported separately from results submitted by those that are not part of the GALE program.

Participants

The following table lists the submissions received from the sites participating in the Arabic-to-English Current Test.

Site ID	Organization	Location	Current Test Set, Arabic-to-English
Site ID	Organization	Location	Single System Track	System Combination Track
amsterdam	University of Amsterdam	Netherlands	Yes⁽¹⁾	-
apptek	AppTek	USA	Yes	-
bbn	BBN Technologies	USA	Yes	-
cmu-smt	Carnegie Mellon LTI interACT	USA	Yes	-
cmu-statxfer	Carnegie Mellon StatXfer	USA	Yes	-
columbia	Columbia University	USA	Yes	-
cued	Cambridge University Engineering Department	UK	Yes	-
edinburgh	University of Edinburgh	UK	Yes	-
fbk	Fondazione Bruno Kessler	Italy	Yes	-
ibm	IBM	USA	Yes	Yes
isi-lw	University of Southern California / Language Weaver Inc.	USA	Yes	Yes
jhu	Johns Hopkins University	USA	Late and/or debugged submission	-
kcsl	KCSL Inc.	Canada	Yes	-
limsi	Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur - CNRS	France	Yes	-
lium-systran	Université du Maine (Le Mans) / SYSTRAN	.	Yes⁽¹⁾	-
rwth	RWTH-Aachen University, Chair of Computer Sciences	Germany	Yes	-
sakhr	Sakhr Software	Egypt	Yes	-
sri	SRI International	USA	Yes	Yes
stanford	Stanford University	USA	Yes	-
telaviv	Tel Aviv University	Israel	Yes	-
tubitak-uekae	TUBITAK-UEKAE	Turkey	Yes	-
umd	University of Maryland	USA	Yes	-
upc-lsi	UPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)	Spain	Yes	-

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Arabic-to-English, submissions from participants involved in GALE (Table 1)

Site ID	System	BLEU-4 (mteval-v13a)
Site ID	System	Overall	Newswire	Web
Constrained Data Track
cued	CUED_a2e_cn_primary	0.4834	0.5641	0.3960
stanford	stanford_a2e_cn_primary	0.4781	0.5673	0.3843
isi-lw	isi-lw_a2e_cn_primary	0.4763	0.5590	0.3810
ibm	IBM_a2e_constrained_primary	0.4708	0.5547	0.3833
bbn	BBN_a2e_cn_primary	0.4680	0.5566	0.3783
rwth	RWTH_a2e_cn_primary	0.4534	0.5402	0.3538
sri	SRI_a2e_cn_primary	0.4527	0.5366	0.3634
edinburgh	Edinburgh_a2e_cn_primary	0.4479	0.5240	0.3605
umd	UMD_a2e_cn_primary	0.4409	0.5340	0.3415
cmu-smt	CMU-SMT_a2e_cn_primary	0.4304	0.5055	0.3473
columbia	columbia_a2e_cn_primary	0.4157	0.4932	0.3331
cmu-statxfer	CMU-Stat-Xfer_a2e_cn_primary	0.3774	0.4448	0.2986
sakhr	SAKHR_a2e_cn_primary	0.3681	0.4185	0.3147
UnConstrained Data Track
ibm	IBM_arabic_un_primary	0.4981	0.5713	0.4214
apptek	AppTek_a2e_un_primary	0.4790	0.5165	0.4352

Arabic-to-English, submissions from participants not involved in GALE (Table 2)

Site ID	System	BLEU-4 (mteval-v13a)
Site ID	System	Overall	Newswire	Web
Constrained Data Track
lium-systran	LIUM-SYSTRAN_a2e_cn_primary⁽¹⁾	0.4773	0.5629	0.3800
fbk	FBK_a2e_cn_primary	0.4567	0.5418	0.3615
limsi	LIMSI_Moses_a2e_cn_primary	0.4384	0.5242	0.3471
tubitak-uekae	TUBITAK_a2e_cn_primary	0.4112	0.4826	0.3310
upc-lsi	UPC.LSI_a2e_cn_primary	0.3588	0.4344	0.2778
amsterdam	UvA_a2e_cn_primary⁽¹⁾	0.3221	0.3820	0.2565
kcsl	KCSL_a2e_cn_primary	0.1422	0.1670	0.1161
telaviv	TLVEBMT_a2e_cn_primary	0.0703	0.0872	0.0527

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - System Combination Track

Arabic-to-English, submissions from participants involved in GALE (Table 3)

Site ID	System	BLEU-4 (mteval-v13a)
Site ID	System	Overall	Newswire	Web
Constrained Data Track
isi-lw	isi-lw_a2e_cn_combo1	0.4802	0.5600	0.3914
ibm	IBM_a2e_cn_combo0	0.4775	0.5636	0.3871
sri	SRI_a2e_cn_combo1	0.4631	0.5472	0.3731
UnConstrained Data Track
ibm	IBM_a2e_un_combo0	0.5096	0.5913	0.4241

Current Test Results - Single System Track and System Combination Track - All metrics

Arabic-to-English, submissions from participants involved in GALE (Table 4)

Site ID	System	BLEU-4 (mteval-v13a)			IBM BLEU (bleu-1.04)			NIST (mteval-v13a)			TER (tercom-0.7.25)			METEOR (meteor-0.7)
Site ID	System	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web
Constrained Data Track
cued	CUED_a2e_cn_primary	0.4834	0.5641	0.3960	0.4833	0.5640	0.3959	11.01	11.31	9.603	0.4489	0.3861	0.5093	0.6570	0.7152	0.5999
isi-lw	isi-lw_a2e_cn_combo1	0.4802	0.5600	0.3914	0.4801	0.5598	0.3913	10.85	11.24	9.396	0.4643	0.3887	0.5371	0.6642	0.7319	0.5969
stanford	stanford_a2e_cn_primary	0.4781	0.5673	0.3843	0.4777	0.5668	0.3840	10.97	11.44	9.392	0.4399	0.3709	0.5065	0.6514	0.7153	0.5882
ibm	IBM_a2e_cn_combo0	0.4775	0.5636	0.3871	0.4773	0.5634	0.3870	11.03	11.54	9.466	0.4394	0.3713	0.5051	0.6526	0.7142	0.5924
isi-lw	isi-lw_a2e_cn_primary	0.4763	0.5590	0.3810	0.4760	0.5588	0.3808	10.85	11.30	9.292	0.4590	0.3826	0.5328	0.6544	0.7233	0.5864
ibm	IBM_a2e_constrained_primary	0.4708	0.5547	0.3833	0.4707	0.5545	0.3831	10.97	11.46	9.406	0.4424	0.3773	0.5051	0.6478	0.7091	0.5876
bbn	BBN_a2e_cn_primary	0.4680	0.5566	0.3783	0.4678	0.5564	0.3781	10.85	11.46	9.304	0.4603	0.3800	0.5378	0.6561	0.7125	0.6015
sri	SRI_a2e_cn_combo1	0.4631	0.5472	0.3731	0.4629	0.5470	0.3730	10.86	11.25	9.306	0.4592	0.3974	0.5189	0.6461	0.7063	0.5866
rwth	RWTH_a2e_cn_primary	0.4534	0.5402	0.3538	0.4533	0.5400	0.3537	10.65	11.12	9.028	0.4666	0.3980	0.5328	0.6532	0.7154	0.5918
sri	SRI_a2e_cn_primary	0.4527	0.5366	0.3634	0.4526	0.5365	0.3633	10.71	11.11	9.251	0.4689	0.4003	0.5351	0.6459	0.7071	0.5859
edinburgh	Edinburgh_a2e_cn_primary	0.4479	0.5240	0.3605	0.4478	0.5238	0.3604	10.58	10.92	9.174	0.4857	0.4209	0.5482	0.6454	0.7069	0.5847
umd	UMD_a2e_cn_primary	0.4409	0.5340	0.3415	0.4408	0.5338	0.3413	10.53	11.25	8.590	0.4591	0.3909	0.5248	0.6287	0.6982	0.5595
cmu-smt	CMU-SMT_a2e_cn_primary	0.4304	0.5055	0.3473	0.4302	0.5053	0.3469	10.33	10.72	8.896	0.4823	0.4182	0.5442	0.6365	0.6988	0.5748
columbia	columbia_a2e_cn_primary	0.4157	0.4932	0.3331	0.4156	0.4931	0.3330	10.27	10.73	8.785	0.4747	0.4160	0.5313	0.6269	0.6847	0.5698
cmu-statxfer	CMU-Stat-Xfer_a2e_cn_primary	0.3774	0.4448	0.2986	0.3772	0.4447	0.2984	9.731	10.07	8.399	0.5158	0.4628	0.5669	0.6082	0.6684	0.5484
sakhr	SAKHR_a2e_cn_primary	0.3681	0.4185	0.3147	0.3680	0.4184	0.3147	9.867	9.982	8.887	0.5075	0.4626	0.5508	0.6320	0.6737	0.5913
UnConstrained Data Track
ibm	IBM_a2e_un_combo0	0.5096	0.5913	0.4241	0.5095	0.5912	0.4240	11.54	11.92	10.02	0.4168	0.3507	0.4804	0.6768	0.7366	0.6181
ibm	IBM_arabic_un_primary	0.4981	0.5713	0.4214	0.4979	0.5711	0.4214	11.41	11.70	9.973	0.4253	0.3653	0.4833	0.6700	0.7293	0.6117
apptek	AppTek_a2e_un_primary	0.4790	0.5165	0.4352	0.4787	0.5162	0.4348	11.21	10.98	10.32	0.4342	0.3969	0.4702	0.6818	0.7207	0.6433

Arabic-to-English, submissions from participants not involved in GALE (Table 5)

Site ID	System	BLEU-4 (mteval-v13a)			IBM BLEU (bleu-1.04)			NIST (mteval-v13a)			TER (tercom-0.7.25)			METEOR (meteor-0.7)
Site ID	System	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web
Constrained Data Track
lium-systran	LIUM-SYSTRAN_a2e_cn_primary⁽¹⁾	0.4773	0.5629	0.3800	0.4772	0.5627	0.3799	10.96	11.38	9.412	0.4565	0.3851	0.5252	0.6526	0.7185	0.5879
fbk	FBK_a2e_cn_primary	0.4567	0.5418	0.3615	0.4565	0.5417	0.3613	10.75	11.22	9.252	0.4721	0.3952	0.5462	0.6533	0.7170	0.5910
limsi	LIMSI_Moses_a2e_cn_primary	0.4384	0.5242	0.3471	0.4383	0.5240	0.3469	10.40	10.98	8.738	0.4724	0.4051	0.5373	0.6212	0.6851	0.5582
tubitak-uekae	TUBITAK_a2e_cn_primary	0.4112	0.4826	0.3310	0.4121	0.4827	0.3312	10.01	10.43	8.659	0.5129	0.4445	0.5789	0.6259	0.6890	0.5627
upc-lsi	UPC.LSI_a2e_cn_primary	0.3588	0.4344	0.2778	0.3588	0.4345	0.2777	9.404	10.05	7.655	0.5188	0.4630	0.5725	0.5866	0.6515	0.5216
amsterdam	UvA_a2e_cn_primary⁽¹⁾	0.3221	0.3820	0.2565	0.3218	0.3816	0.2563	8.621	9.063	7.458	0.5995	0.5437	0.6533	0.5863	0.6457	0.5278
kcsl	KCSL_a2e_cn_primary	0.1422	0.1670	0.1161	0.1420	0.1669	0.1158	6.647	6.959	5.820	0.6590	0.6353	0.6818	0.4937	0.5398	0.4482
telaviv	TLVEBMT_a2e_cn_primary	0.0703	0.0872	0.0527	0.0703	0.0872	0.0526	3.879	4.370	3.165	0.7483	0.7274	0.7685	0.3853	0.4262	0.3450

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

NIST 2009 Open Machine Translation Evaluation (MT09) Official Release of Results

Introduction

Participants

Current Test Results - Single System Track

Arabic-to-English, submissions from participants involved in GALE (Table 1)

Arabic-to-English, submissions from participants not involved in GALE (Table 2)

Current Test Results - System Combination Track

Arabic-to-English, submissions from participants involved in GALE (Table 3)

Current Test Results - Single System Track and System Combination Track - All metrics

Arabic-to-English, submissions from participants involved in GALE (Table 4)

Arabic-to-English, submissions from participants not involved in GALE (Table 5)

NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results