ITL's Evaluation-Driven Approach to Information Retrieval is Expanding Internationally

The evaluation-driven approach to technology development in the area of information retrieval, pioneered by the Information Access Division's (IAD) Text REtrieval Conference (TREC), has served as the inspiration for two other evaluation forums in Europe and Japan. Two recent meetings, the Cross-Language Evaluation Forum (CLEF) in Europe and the NTCIR3 workshop in Japan, demonstrated the impact of providing good testing methodologies and reference data. NIST software and evaluation methodologies, developed by IAD, were used in these forums, and IAD serves as the English partner for CLEF.

The CLEF workshop evolved from the cross-language text retrieval task run in TREC from 1997- 1999. This task evaluated retrieval using questions in one language and documents in a second language. The task moved to Europe in 2000, testing in 5 different European languages. This year's September meeting used testing in 8 different languages, with 37 research groups submitting results for 3 different tasks. Next year the plan is to evaluate question-answering in Italian and Spanish, along with cross-language question answering to English. Methodology developed in NIST's TREC question-answering task will be used and IAD will help develop the evaluation methodology for the cross-language variation.

NTCIR3 is the third meeting of the Japanese version of TREC. This effort was started in 1998, initially with cross-language retrieval between Japanese and English. This year's October meeting featured 5 tasks, including patent retrieval, question-answering and text summarization, and web retrieval, all from Japanese documents. Additionally there was cross-language retrieval between English, Japanese, Chinese and Korean newspapers. There were 65 research groups that participated, including many major Japanese companies.

IAD will continue to serve as both the mentor and partner in these efforts, and will be able to transfer new testing ideas developed in these projects back to the NIST TREC project.

Contact: Donna Harman, ext. 3569