<%@LANGUAGE="JAVASCRIPT" CODEPAGE="65001"%> NIST Speech Group Website
Information Technology Lab, Information Access Division NIST: National Institute of Standards and Technology

  • Multimodal Information Group Home
  • Benchmark Tests
  • Tools
  • Test Beds
  • Publications
  • Links
  • Contacts
  • 2007 Language Recognition Evaluation

    The 2007 NIST Language Recognition evaluation (LRE-07) is part of an ongoing series of evaluations of language recognition technology. LRE-07 will be held in the second half of calendar year 2007. It is planned for October of 2007, with the evaluation workshop to be held in December in Orlando, Florida. The evaluation will be similar to previous NIST language evaluations held in 1996, 2003, and 2005. A detailed evaluation plan describing the evaluation task and the evaluation protocols is now available (LRE07EvalPlan-v8b.pdf, updated July 20, 2007).

    There will be two data sets and a supplement data set available for training and development:

    • The first set consists of the data used in each of the three preceeding evaluations and the conversations of the 12 language CallFriend Corpus published by the LDC. This set is now available directly from the Linguistic Data Consortium (LDC) to all prospective evaluation participants who request it.
    • The second set consists of newly collected conversations for languages without training data from other sources. This set is available also directly from the LDC in March to all registered evaluation participants who request it. For these conversations, NIST has designated two segments of each of the three durations in each of the conversation sides for development purposes if sites wish to use them. The timestamps for these segments are now available (uploaded May 14, 2007).
    • The supplement set which contains the full conversations from which the primary LRE05 evaluation segments were drawn from is also available to registered evaluation participants. This set is available from NIST.

    Sites planning to participate in the evaluation (including sites that are LDC members) must complete and sign the LDC license agreement and an addendum to the license and fax them to the LDC. Upon receiving the signed licenses, the LDC will send the sites the requested training data sets.

    To register for the evaluation, fill out the registration form and fax it to NIST. Last day to register is September 1, 2007.

    A beta version of the scoring software implementing the performance measurement as described in the LRE07 evaluation plan is now available (score_lre07.v01d.tgz, updated November 26, 2007).

    The latest version of the evaluation data key is now available (lid07key_v5.txt), last updated 10/31/07.

    The schedule for the evaluation is given below:

    Mar 1, 2007 - Additional training data available
    Sep 1, 2007 - Committment from sites to participate
    Oct 1, 2007 - Data arrives at sites
    Oct 17, 2007 - Submissions due to NIST
    Oct 26, 2007 - Preliminary results released to participants
    Dec 11-12, 2007 - Evaluation workshop
    May 1, 2008 - Public results released

    [ LRE Home ]



    Page Created: July 31, 2007
    Last Updated: August 12, 2009

    Multimodal Information Group is part of IAD and ITL
    NIST is an agency of the U.S. Department of Commerce
    Privacy Policy | Security Notices|
    Accessibility Statement | Disclaimer | FOIA