ACE Phase 2b 2002
Resources
Seed Database
ACE
seed database. NEW August
27, 2002. Version 8 supersedes
all previous versions reformats PERSON names and removes more duplicates.
The seed database is used for the database task. New this year, the
seed database has been updated to include entities of type PERSON that
were found in the 1998 CIA Factbook. The earlier versions of the the
seed database were created from the 2001 Factbook, and therefore post
dated the evaluation data.
ACE corpora
The ACE corpora is owned by the LDC. Training data will be made available
to registered participants.
Software
History of software for the 2002 evaluation:
03/17/2003: rdc-eval.v16.pl.
Algorithmic improvement; much faster search process.
09/23/2002: rdc-eval.v12.pl
Used for ACE September 2002 Evaluation.
03/05/2003: rdc-eval.v15.pl.
Fixes a bug in the branch and bound search algorithm.
08/19/2002 : rdc-eval.v08.pl
Modified to accommodate symmetrical relations -- i.e., relations in
which the order of the arguments doesn't matter.
08/15/2002 : rdc-eval.v07.pl Score calculation fix and modified entity
mapping to impose an additional limit on mentions that are allowed
to contribute to the entity mapping score. System output mentions
are now used only if their level matches the level of the corresponding
reference mention.
08/12/2002 : rdc-eval.v05.pl Relation error penalties were modified,
and additional conditional analyses were added to relation evaluation.
08/06/2002 : rdc-eval.v04.pl
Modified to include a value penalty for "incorrect" entity
arguments. Improved to handle XML empty-element short cut tags.
07/12/2002 : rdc-eval.v03.pl
Early feedback from RDC research sites has indicated that performance
of RDC suffers greatly from the requirement that in order for a system
output relation to match a reference relation, all system relation
arguments (entities) must match (i.e., be mapped to) the corresponding
arguments (entities) of the reference relation. In order to decouple
the EDT and RDC research issues insofar as possible, we have relaxed
this requirement. Now, with rdc-eval version 3, the entity arguments
of system output and reference relations are only required to have
at least one mention in common. Although it is true that in order
for a relation to be informative the entities it relates must be correct,
we feel that research progress will be better served by this change,
so that relation-specific research issues are as unclouded as possible
by entity reference/coreference issues.
06/28/2002 : rdc-eval.v02.pl
this scoring script is an updated version of the edt-eval script.
Version 02 includes RDC scoring. The beginning of the script contains
a brief history description.
03/07/2002 : EDT_ref_compare
(v28) and edt-eval
(v22)
edt-eval (v22) provides an option to split multi-role entities into
separate single-role entities (in which all mentions have the same
role). New (sub)entities are created to collect all same-role mentions
from the original entity. Names that match mention heads stay with
the entity that contains the mention. (EDT_ref_compare (v28) unchanged).
02/04/2002: Sample
ASR scoring version 0. Includes a sample set of instructions that
will score the ASR data distributed as part of the original ACE pilot
study data.
02/07/2002:
Sample
OCR scoring version 1. Updated because the reference images
used included some empty bounding boxes. Also, there is a readme.txt
file in the reference directory that explains some instances of "EMPTY
BOUNDING BOXES" in submission files. Includes a sample set
of instructions that will score the OCR data distributed as part of
the original ACE pilot study data.
edt-eval
(v16)
edt-eval (v16) includes a provision for reading in an optional database
of entities. Output entities are checked to see if they appear in
the external database. If they do, then that assignment is used and
no mapping is performed. (In other words, if the system assigns the
entity ID, then that choice overrides the (locally) optimum choice
of "best matching" reference entity.) (EDT_ref_compare
(v28) unchanged).
[ ACE Home ]
Page Created: September 6, 2007
Last Updated: November 4, 2008
|
