<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> TRECVid 2010 Surveillance Event Detection Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • TRECVid 2010 Evaluation
    for Surveillance Event Detection

    The Surveillance Event Detection (SED) evaluation is a track within of the TRECVid Evaluation. The goal of the evaluation track is to support the development of technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The evaluation is implemented using 150 hours of multi-camera airport surveillance domain data collected by the Home Office Scientific Development Branch (HOSDB), which was ground-truthed by the University of Pennsylvania Linguistic Data Consortium.

    The 2010 SED evaluation is the 3rd evaluation of event detection technologies. The evaluation task will be similar in structure to previous evaluations, which were held in 2008 and 2009, however there will be some minor changes which includes:

    • The test copora used for the 2009 evaluation will be re-used for the 2010 evaluation.
    • The less frequent events of interest, ElevatorNoEntry, OpposingFlow, and TakePicture, will not be included in the evaluation to focus research on improving performance on events with many observations.
    • A new sub-task: SED Clip Filtering will be introduced as a low entry barrier task.

    We have a mailing list for announcements/discussion -- contact jonathan.fiscus@nist.gov to subscribe.

    Evaluation Planning

    The evaluation plan is evolving and the following resources are available:

    • Evaluation Plans:
    2010 SED Evaluation Plan - V1 V1 is essentially unchanged from the 2009-V3 evaluation plan.

    Software

    Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page.

    Dry Run

    The dry run period for the TRECVid Event Detection Evaluation will run until July 21, 2010. Dry run submissions will be accepted at any time during the period. The procedure for participating is as follows:

    • Sign up as a TRECVid Participant as specified on the TRECVid 2010 web site.
    • Obtain the data sets by completing the licensing agreement as specifiec below. The dry run will use part of the development corpus.
    • Download the Experiment Control File (ECF): expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP
    • Notify jonathan.fiscus@nist.gov, martial.michel@nist.gov
    • Submissions are due 11:59PM Eastern Time (UTC-4), July 21, 2010
    • Results of the dry run will be returned by July 28, 2010 (approx.)

    Formal Evaluation

    The formal evaluation for the TRECVid Event Detection Evaluation has now commenced. The procedure for participating is as follows:

    Data Resources

    i-LIDS Logo
    Licensing/Participation Agreements

    The procedure for licensing the data is as follows:

    1. Apply to participate in the TRECVid event detection evaluation by completing the registration forms (NOTE: The deadline has passed for this step).

    2. The development data for 2010 evaluation consists of the 2008 Event Detection Training and Testing sets. Follow the instructions in the guidelines http://www-nlpir.nist.gov/projects/tv2010/tv2010.html#licenses. NIST will acknowledge reciept and then you should proceed to next step.

    3. The evaluation set for the 2010 Event Detection evaluation will be the UK Home Office Scientific Development Branch's (HOSDB) i-LIDS MCTTR dataset. This is the same evaluation corpus as used for the 2009 evaluation. The MCTTR data set is available at no cost. To obtain the corpus, participants must complete HOSDB's "MCT application form" and relevant end-user license agreement found below and submit as indicated on the forms. Upon completion, HOSDB will give participants a unique username and password for accessing online resources.

    MCT Application Form

    End User License Agreement (EULA) for Commercial Companies

    End User License Agreement (EULA) for Academic Institutions. Note: Please be sure on the application form to indicate that you are applying in order to participate in the TRECVid evaluation.

    Development Data Acquisition

    Download the data by either going through the TRECVid data servers (check server lists in the active participants area) or by shipping a hard drive (at least 250 GB) to:

    Martial Michel
    100 Bureau Drive, Mail Stop 8940
    Gaithersburg, MD 20899
    USA

    Annotations for the Development corpora can by found on the data page.

    Test Data Acquisition

    There are two ways to obtain the test data: (a) via download from NIST, (b) via hard drive sent to the UK.

    • Download from NIST: To obtain the test data (i.e., the 2009 i-LIDS MCTTR dataset) from NIST, download the MPEG-2 video using an HOSDB-assigned username and password from the URL http://ir.nist.gov/tv2009/iLIDS/Video/Eval09/. The MPEG-2 video is compatible, (same frame rate, resolution, and de-interlaced) with the Development data set. This MPEG-2 data is 150GB.
    • Via hard drive sent to UK: To obtain the full MCTTR data from the UK you will need to send a 1TB hard drive to HOSDB in the United Kingdom according to the instructions in the MCT application form. The full MCTTR data set consists of the MPEG-2 files above and the original, interlaced video imagery collected from the cameras. In order to maintain compatibility with the 2008 eval, sites should use the MPEG-2 files.
       

    Related

    AVSS 2010 Challenge Evaluation

    AVSS 2010 Home Page

     

     

     

    Page Created: January 14, 2009
    Last Updated: May 6, 2010

    Multimodal Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA