This evaluation is intended to help promote technology development for event detection in video surveillance. The goal is to leverage machine learning techniques for detecting a pre-defined set of events. This evaluation task focuses on detecting events in airport surveillance video.

NIST published the results of the evaluation in a summary paper at the Workshop on Applications of Computer Vision (WACV) during the 2009 IEEE Winter Vision Meetings.

Evaluation Planning

As we write the evaluation plan, we will be asking for input from the research community. Planning telecons will be held during the following dates/times:

• Minutes of the Jan. 16, 2008 telecon
• Minutes of the Jan. 23, 2008 telecon
• Minutes of the Jan. 30, 2008 telecon
• Minutes of the Feb. 6, 2008 telecon
• Minutes of the Feb. 13, 2008 telecon
• Minutes of the Feb. 27, 2008 telecon
• Minutes of the March 19, 2008 telecon
• Minutes of the March 27, 2008 telecon

The evaluation plan is evolving and the following resources are available:

• Initial Discussion Documents:
  • The following slides outline several factors of the evaluation which we plan to discuss.
  • These slides outline some additional considerations of the evaluation protocol.
• Evaluation Plans:
  As the evaluation plan changes, two versions will be published: a normal text version and a comparison document. The comparison document will contrast the latest version with the previous version to focus the reader on recent changes.

  Normal Text Version	Comparisons Document
  Straw Man Proposal - V1
  Eval Plan - V2	V1 compared to V2
  Eval Plan - V3
  Eval Plan - V4
  Eval Plan - V5
  Eval Plan - V6
  Eval Plan - V7

• Event Definitions:
  • The following table lists 20 candidate events and responses from the forum.
  • The following paper EventDet-ViPER-format-v4.pdf documents XML Schema Definition (XSD) proposed for a ViPER XML Representation of Events.
  • The following TrecVid08_XSD_v2.tar includes the XSD and a sample XML file for events.
  • An annotation guidelines document TRECVid08_Guidelines_v1.6.pdf is now available.
• Experimental Control Files:
  • The following TRECVid08_ecf_v2.tar includes the XSD and a sample XML file for Experimental Control Files (ECFs).

Software

The evaluation used the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page. The version used for evaluation was: F4DE-1.3-20081120-1334.tar.bz2

Data Resources

A TRECVid Event Detection microcorpus is now available. Please contact travis.rose@nist.gov and jonathan.fiscus@nist.gov for details.

Dry Run

The dry run for the TRECVid Event Detection Evaluation has now commenced. The procedure for participating is as follows:

• Download the Experimental Control File (ECF): expt_08_RetroED_Dev08_eng_s-camera_NIST_061808.xml
• Run experimental system(s) on the video intervals specified in the ECF
• Package system outputs in ViPER format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the software download (above) for a validation script
• Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
• Submissions are due 11:59PM Eastern Time (UTC -4), July 21, 2008
• Results of the dry run will be returned no later than August 1, 2008

Questions may be sent to travis.rose@nist.gov and/or the discussion list. Thanks for your participation!

Formal Evaluation

The procedure for participating in the formal evaluation for the TRECVid Event Detection task is as follows:

• Download the Experimental Control File (ECF): expt_2008_retroED_EVAL08_ENG_s-camera_NIST_1.xml
• Run experimental system(s) on the video intervals specified in the ECF
• Package system outputs in ViPER format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the software download (above) for a validation script
• Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
• Notify jonathan.fiscus@nist.gov, travis.rose@nist.gov, martial.michel@nist.gov
• Submissions are due 11:59PM Eastern Time (UTC -4), September 26, 2008
• Preliminary results of the formal evaluation will be returned on October 1, 2008

Questions may be sent to travis.rose@nist.gov and/or the discussion list. Thanks for your participation!

Related

TRECVid Home Page