The Information Access Division ran a new 2-day workshop at NIST on November 17 and18, 2003. Sponsored by NSA’s Advanced Research and Development Activity (ARDA) and NIST, the workshop was attended by an international group of almost 50 researchers and is the successor to the Video Retrieval Track, which was part of the Text Retrieval Conference (TREC) in 2001 and 2002. The goal remained the same: progress in content-based retrieval from digital video via open metrics-based evaluation.
Twenty-four research groups, including five companies, 10 groups from the U.S., 10 from Europe, and four from Asia/Australia participated in one or more of four tasks: shot boundary detection, story segmentation and typing, high-level feature extraction, and search. Development and test data were drawn from about 133 hours of MPEG-1 video from U.S. news broadcasts recorded by the Linguistic Data Consortium in 1998. NIST created truth data for the shot boundary task, manually judged the feature and search submissions, and evaluated the submissions in all four tasks.
Shot boundary detection for cuts can be considered largely solved for most purposes, but detection of gradual transitions left room for improvement. Story segmentation results suggested that the use of visual and audio information beyond text from speech improves effectiveness. Extraction of high-level features (e.g. outdoors, people, aircraft, sporting event, physical violence, etc.) varied greatly across features with promising results for several. Search, starting from a multimedia statement of an information need, remained the most challenging task. The fully interactive systems, including a version of Carnegie Mellon University’s Informedia showing improvement over last year’s, generally outperformed more nearly automatic ones. Interactive systems offered their users interesting combinations of approaches for search and browsing (e.g., chronologically or by image similarity) optionally with respect to various high-level features.
More information is available at: http://www-nlpir.nist.gov/projects/trecvid