<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> TRECVID 2012 Surveillance Event Detection Evaluation
Information Technology Lab, Information Access
                Division NIST: National Institute of Standards and
                Technology


  • Multimodal Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • TRECVID 2012 Evaluation
    for Surveillance Event Detection

    The Interactive Surveillance Event Detection (SED) Pilot evaluation is a track within the TRECVID Evaluation. The goal of the 2012 Interactive SED evaluation track is to support the development of interactive (i.e., human in the loop) technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The interactive aspect of SED is new for 2012. The evaluation will be implemented using a 15 hour subset of the 2011 multi-camera airport surveillance domain evaluation data collected by the Home Office Scientific Development Branch (HOSDB), which was ground-truthed by the University of Pennsylvania Linguistic Data Consortium.

    The 2012 SED evaluation is the 5th evaluation of event detection technologies preceeded by the 2008, 2009, 2010, and 2011 evaluations. The interactive evaluation methodology is similar to previous evaluation i.e., systems report the same information and are scored with the same metrics, however, unlike previous evaluations the data set size will be smaller and each search for an event by a searcher can take no more than 25 elapsed minutes (wall clock time), measured from the time the searcher is given the event to look for until the time the result set is considered final. The system may use an unlimited amount of computional resources prior to beginning the interactive search.

    We have a mailing list for announcements/discussion -- contact jonathan.fiscus@nist.gov to subscribe.

    Evaluation Planning

    The following resources are available:

    • Evaluation Plans:
    Read the TRECVID SED 2012 Evaluation Plan-V03
    • V02 - Minor submission instruction changes
    • V03 - Additional minor changes in the submission instructions and the addition of the submission checker

    Software

    Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the MIG Tools Web Page.

    Dry Run

    The dry run period for the 2012 TRECVID Interactive Surveillance Event Detection Evaluation will run from July 2nd to July 30th 2012. Dry run submissions will be accepted at any time during that period. The procedure for participating is as follows:

    • Sign up as a TRECVID Participant as specified on the TRECVID web site.
    • Obtain the data sets by completing the licensing agreement as specifiec below. The dry run will use part of the development corpus.
    • Download the Experiment Control File (ECF) [same as 2009, 2010, and 2011]: expt_2009_retroED_DEV09_ENG_s-camera_NIST_2.xml
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV12SED-SubmissionChecker tool. Note: this has not yet been released
    • Notify martial.michel@nist.gov and jonathan.fiscus@nist.gov. Please include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problem), as well as a possible alternate contact person.
    • Submissions are due 11:59PM Eastern Time (UTC-4), July 30th, 2012
    • Results of the dry run will be returned by August 15th, 2012 (approx.)

    Formal Evaluation

    Submissions for the formal evaluation are due October 1st, 2012. The procedure for participating is as follows:

    • Complete the dry run as specified above.
    • Download the Experiment Control Files (ECF) [THIS IS NEW FOR 2012] :
      There are two ECF files defined in the eval plan: The files are identical, BUT they differ from previous years' ECF files in that only the 15 hours of evaluated material are identified.
    • Run experimental system(s) on the video intervals specified in the ECF
    • Package system outputs in ViPER XML format and make sure they validate against the TrecVid08.xsd: TrecVid08_XSD_v2.tar -- see the F4DE software download (above) for a validation script
    • Prepare a tar/gzip of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV12SED-SubmissionChecker tool.
    • Notify martial.michel@nist.gov and jonathan.fiscus@nist.gov. Please include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problem), as well as a possible alternate contact person.
    • Submissions are due 11:59AM Eastern Time (UTC-4), October 1th, 2012
    • Results of the formal evaluation will be returned by October 5th, 2012 (approx.)

    Data Resources

    • The data resources provided for 2012 are the same as 2011.
    i-LIDS Logo
    Licensing/Participation Agreements

    The procedure for licensing the data is as follows:

    1. Apply to participate in the TRECVID Surveillance Event Detection evaluation by completing the TRECVID registration forms. For more details, see: http://trecvid.nist.gov/

    2. The development data for 2011 evaluation consists of the 2008 Event Detection Training and Testing sets. Follow the instructions in the guidelines http://www-nlpir.nist.gov/projects/tv2010/tv2010.html#licenses. NIST will acknowledge receipt and then you should proceed to next step.

    3. The evaluation set for the 2011 Event Detection evaluation will be the UK Home Office Scientific Development Branch's (HOSDB) i-LIDS MCTTR dataset. This is the same evaluation corpus as used for the 2009 evaluation. The MCTTR data set is available at no cost. To obtain the corpus, participants must complete HOSDB's "MCT application form" and relevant end-user license agreement found below and submit as indicated on the forms. Upon completion, HOSDB will give participants a unique username and password for accessing online resources.

    MCT Application Form

    End User License Agreement (EULA) for Commercial Companies

    End User License Agreement (EULA) for Academic Institutions. Note: Please be sure on the application form to indicate that you are applying in order to participate in the TRECVID evaluation.

    Development Data Acquisition

    Download the data by either going through the TRECVID data servers (check server lists in the active participants area) or by shipping a hard drive (at least 250 GB) to:

    Martial Michel
    100 Bureau Drive, Mail Stop 8940
    Gaithersburg, MD 20899
    USA

    Annotations for the Development corpora can by found on the data page.

    Test Data Acquisition

    There are two ways to obtain the test data: (a) via download from NIST, (b) via hard drive sent to the UK.

    • Download from NIST: To obtain the test data (i.e., the 2009 i-LIDS MCTTR dataset) from NIST, download the MPEG-2 video using an HOSDB-assigned username and password from the URL http://ir.nist.gov/tv2009/iLIDS/Video/Eval09/. The MPEG-2 video is compatible, (same frame rate, resolution, and de-interlaced) with the Development data set. This MPEG-2 data is 150GB.
    • Via hard drive sent to UK: To obtain the full MCTTR data from the UK you will need to send a 1TB hard drive to HOSDB in the United Kingdom according to the instructions in the MCT application form. The full MCTTR data set consists of the MPEG-2 files above and the original, interlaced video imagery collected from the cameras. In order to maintain compatibility with the 2008 eval, sites should use the MPEG-2 files.
       

     

     

     

    Page Created: January 14, 2009
    Last Updated: July 19, 2012

    Multimodal Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA