<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> TRECVid 2009 Event Detection Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • TRECVid 2009 Evaluation
    for Event Detection

    This evaluation is intended to help promote technology development for event detection in video surveillance. The goal is to leverage machine learning techniques for detecting a pre-defined set of events. The evaluation tasks are focused on detecting events in airport surveillance video. Results of the 2009 evaluation can be found in the TRECVID 2009 Video Surveillance Event Detection Track Overview presentation given at TRECVID 2009.

    Evaluation Planning

    As we write the evaluation plan, we will be asking for input from the research community. Planning telecons will be held during the following dates/times:

    • Minutes of the Feb. 19, 2009 telecon
    • Minutes of the Feb. 26, 2009 telecon

    The evaluation plan is evolving and the following resources are available:

    • Evaluation Plans:
    Normal Text Version
    Eval Plan - V1
    Eval Plan - V2
    Eval Plan - V3

    Software

    Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page.

    Dry Run

    The dry run for the TRECVid Event Detection Evaluation has now commenced. The procedure for participating is as follows:

    • Download the Experiment Control File (ECF): expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
    • Notify jonathan.fiscus@nist.gov, martial.michel@nist.gov
    • Submissions are due 11:59PM Eastern Time (UTC-4), July 22, 2009
    • Results of the dry run will be returned by July 31, 2009 (approx.)

    Formal Evaluation

    The formal evaluation for the TRECVid Event Detection Evaluation has now commenced. The procedure for participating is as follows:

    • Download the Experiment Control File (ECF): expt_2009_retroED_EVAL09_ENG_s-camera_NIST_3.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
    • Notify jonathan.fiscus@nist.gov, martial.michel@nist.gov
    • Submissions are due 11:59PM Eastern Time (UTC-4), September 23, 2009
    • Results of the evaluation will be returned by October 10, 2009

    Data Resources

    i-LIDS Logo
    Licensing/Participation Agreements

    The procedure for licensing the data is as follows:

    1. Apply to participate in the TRECVid event detection evaluation by completing the registration forms (NOTE: The deadline has passed for this step).

    2. The development data for 2009 evaluation consists of the 2008 Event Detection Training and Testing sets. Follow the instructions in the guidelines http://www-nlpir.nist.gov/projects/tv2009/tv2009.html#3.

    3. The evaluation set for the 2009 Event Detection evaluation will be the UK Home Office Scientific Development Branch's (HOSDB) i-LIDS MCTTR dataset. The MCTTR data set is available at no cost. To obtain the corpus, particpants must complete HOSDB's "MCT application form" and relevant end-user license agreement found at:http://scienceandresearch.homeoffice.gov.uk/hosdb/cctv-imaging-technology/video-based-detection-systems/i-lids/ilids-datasets-pricing/?version=1. and submit as indicated on the forms. Upon completion, HOSDB will give participants a unique username and password for accessing online resources.

    MCT Application Form: http://scienceandresearch.homeoffice.gov.uk/hosdb/publications/cctv-publications/FMCT_Dataset_Application_301.pdf

    End User License Agreement (EULA) for Commercial Companies: http://scienceandresearch.homeoffice.gov.uk/images/106966/356423/Legal_-_MCT_EULA_Commercial1.pdf

    End User License Agreement (EULA) for Academic Institutions: http://scienceandresearch.homeoffice.gov.uk/images/106966/356423/Legal_-_MCT_EULA_Academic_01.pdf

    Note: Please be sure on the application form to indicate that you are applying in order to participate in the TRECVid evaluation.

    Development Data Acquisition

    Download the data by either going through the TRECVid data servers (check server lists in the active participants area) or by shipping a hard drive (at least 250 GB) to:

    Martial Michel
    100 Bureau Drive, Mail Stop 8940
    Gaithersburg, MD 20899
    USA

    Test Data Acquisition

    There are two ways to obtain the test data: (a) via download from NIST, (b) via hard drive sent to the UK.

    • Download from NIST: To obtain the test data (i.e., the 2009 i-LIDS MCTTR dataset) from NIST, download the MPEG-2 video using an HOSDB-assigned username and password from the URL http://ir.nist.gov/tv2009/iLIDS/Video/Eval09/. The MPEG-2 video is compatible, (same frame rate, resolution, and de-interlaced) with the Development data set. This MPEG-2 data is 150GB.
    • Via hard drive sent to UK: To obtain the full MCTTR data from the UK you will need to send a 1TB hard drive to HOSDB in the United Kingdom according to the instructions in the MCT application form. The full MCTTR data set consists of the MPEG-2 files above and the original, interlaced video imagery collected from the cameras. In order to maintain compatiblility with the 2008 eval, sites should use the MPEG-2 files.

    Related

    TRECVid Home Page

    AVSS 2009 Challenge Evaluation

    AVSS 2009 Home Page

     

     

     

    Page Created: January 14, 2009
    Last Updated: April 8, 2010

    Multimodal Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA