<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> TRECVID 2011 Surveillance Event Detection Evaluation
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology


  • Multimodal Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • TRECVID 2011 Evaluation
    for Surveillance Event Detection

    The Surveillance Event Detection (SED) evaluation is a track within the TRECVID Evaluation. The goal of the evaluation track is to support the development of technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The evaluation is implemented using 150 hours of multi-camera airport surveillance domain data collected by the Home Office Scientific Development Branch (HOSDB), which was ground-truthed by the University of Pennsylvania Linguistic Data Consortium.

    The 2011 SED evaluation is the 4th evaluation of event detection technologies. The evaluation task will be similar in structure to previous evaluations, which were held in 2008, 2009, 2010 and use the minor changes used in 2010, which includes:

    • The copora used for the 2010 evaluation was re-used for the 2011 evaluation.
    • The less frequent events of interest, ElevatorNoEntry, OpposingFlow, and TakePicture, will not be included in the evaluation to focus research on improving performance on events with many observations.

    We have a mailing list for announcements/discussion -- contact jonathan.fiscus@nist.gov to subscribe.

    Evaluation Planning

    The following resources are available:

    • Evaluation Plans:
    Read the updated TRECVID SED 2011 Evaluation Plan

    Software

    Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page.

    Dry Run

    The dry run period for the 2011 TRECVID Surveillance Event Detection Evaluation will run from July 1st to July 29th 2011. Dry run submissions will be accepted at any time during that period. The procedure for participating is as follows:

    • Sign up as a TRECVID Participant as specified on the TRECVID web site.
    • Obtain the data sets by completing the licensing agreement as specifiec below. The dry run will use part of the development corpus.
    • Download the Experiment Control File (ECF) [same as 2009 and 2010]: expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV11SED-SubmissionChecker tool.
    • Notify martial.michel@nist.gov and jonathan.fiscus@nist.gov. Please include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problem), as well as a possible alternate contact person.
    • Submissions are due 11:59PM Eastern Time (UTC-4), July 29th, 2011
    • Results of the dry run will be returned by August 5th, 2011 (approx.)

    Formal Evaluation

    The formal evaluation for the 2011 TRECVID Surveillance Event Detection Evaluation is due October 19th, 2011. The procedure for participating is as follows:

    • Complete the dry run as specified above.
    • Download the Experiment Control File (ECF) [same as 2009 and 2010]: expt_2009_retroED_EVAL09_ENG_s-camera_NIST_3.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV11SED-SubmissionChecker tool.
    • Notify martial.michel@nist.gov and jonathan.fiscus@nist.gov. Please include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problem), as well as a possible alternate contact person.
    • Submissions are due 11:59AM Eastern Time (UTC-4), October 19th, 2011
    • Results of the formal evaluation will be returned by October 28th, 2011 (approx.)

    Data Resources

    i-LIDS Logo
    Licensing/Participation Agreements

    The procedure for licensing the data is as follows:

    1. Apply to participate in the TRECVID Surveillance Event Detection evaluation by completing the TRECVID registration forms. For more details, see: http://trecvid.nist.gov/

    2. The development data for 2011 evaluation consists of the 2008 Event Detection Training and Testing sets. Follow the instructions in the guidelines http://www-nlpir.nist.gov/projects/tv2010/tv2010.html#licenses. NIST will acknowledge receipt and then you should proceed to next step.

    3. The evaluation set for the 2011 Event Detection evaluation will be the UK Home Office Scientific Development Branch's (HOSDB) i-LIDS MCTTR dataset. This is the same evaluation corpus as used for the 2009 evaluation. The MCTTR data set is available at no cost. To obtain the corpus, participants must complete HOSDB's "MCT application form" and relevant end-user license agreement found below and submit as indicated on the forms. Upon completion, HOSDB will give participants a unique username and password for accessing online resources.

    MCT Application Form

    End User License Agreement (EULA) for Commercial Companies

    End User License Agreement (EULA) for Academic Institutions. Note: Please be sure on the application form to indicate that you are applying in order to participate in the TRECVID evaluation.

    Development Data Acquisition

    Download the data by either going through the TRECVID data servers (check server lists in the active participants area) or by shipping a hard drive (at least 250 GB) to:

    Martial Michel
    100 Bureau Drive, Mail Stop 8940
    Gaithersburg, MD 20899
    USA

    Annotations for the Development corpora can by found on the data page.

    Test Data Acquisition

    There are two ways to obtain the test data: (a) via download from NIST, (b) via hard drive sent to the UK.

    • Download from NIST: To obtain the test data (i.e., the 2009 i-LIDS MCTTR dataset) from NIST, download the MPEG-2 video using an HOSDB-assigned username and password from the URL http://ir.nist.gov/tv2009/iLIDS/Video/Eval09/. The MPEG-2 video is compatible, (same frame rate, resolution, and de-interlaced) with the Development data set. This MPEG-2 data is 150GB.
    • Via hard drive sent to UK: To obtain the full MCTTR data from the UK you will need to send a 1TB hard drive to HOSDB in the United Kingdom according to the instructions in the MCT application form. The full MCTTR data set consists of the MPEG-2 files above and the original, interlaced video imagery collected from the cameras. In order to maintain compatibility with the 2008 eval, sites should use the MPEG-2 files.
       

     

     

     

    Page Created: January 14, 2009
    Last Updated: January 31, 2011

    Multimodal Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA