The ninth workshop in the Text REtrieval Conference (TREC) series was hosted at NIST in Gaithersburg by ITL’s Information Access Division (IAD) November 13-16, 2000. Seventy groups representing 17 different countries participated in the workshop to discuss the results of a year-long cycle of testing that was conducted by the Retrieval Group in IAD.

Included this year was the second running of the Question Answering track, with the goal of retrieving text snippets rather than documents in response to a question.  Over 500 questions drawn from query logs donated by Encarta and Excite were used for testing.  These real questions made the task noticeably more difficult than the TREC-8 task, but despite the increased difficulty, the best performing system was able to extract a correct answer to the question more than 65% of the time.  This system, from Southern Methodist University in Dallas, Texas, demonstrated the power of integrating multiple natural language processing techniques with abductive reasoning to produce a final score almost twice that of the other 27 groups that participated in the testing.  Based on past TREC experience, this type of breakthrough will lead to major changes in the techniques used by systems in next year's TREC, and eventually to improved question answering products on the Web.

Another of the emphases in this year's TREC was an examination of the infrastructure required to evaluate Web search engines. The Web differs from other data collections used in TREC in a variety of ways: size; variety of subject matter, media types, languages, and presentation styles; lack of specific editorial control; explicit links among documents; frequent changes to documents; and generated content. Any of these differences may impact retrieval effectiveness, and the challenge facing the retrieval community is building appropriate test suites that can isolate the effects of these different factors.  A highlight of the conference was an invited talk by Andrei Broder, Vice President of Research and Chief Scientist of AltaVista Company, that provided a perspective on evaluation from a commercial web search engine company.