[ ACE Home ]
- Read the description of the
ACE scoring tool's output (pdf).
- Current Version
- The improvements will never end. The script ace04-eval-v10.pl
is now available.
It has been observed that the F-measure as computed by ace-eval
unfairly penalizes attribute recognition errors by subtracting
these errors from both precision and recall. It has been suggested
that this can be remedied by subtracting only half of the errors
from both precision and recall. This results in a significant
increase in F-measure values. For example, with a 50 percent
attribute recognition error rate the F-measure improves from
50 percent to 75 percent (assuming no misses or false alarms).,
Version 10 implements this change.
Other notable fixes include (see script header for full list):
* both TYPE and SUBTYPE must match for an entity to be "correct"
* relation type/subtype "OTHER-AFF"/"Other"
added to symmetric list
- The improvements continue. The script ace04-eval-v09.pl corrects
a bug in the calculation of relation scores, performs relation
mention scoring, and contains a simplified value formula for
the TERN evaluation. The utility tern2apf
(see below) has also been improved.
- An updated version of the scoring script is available: ace04-eval-v08.pl
. This version contains some general fixes as well as updates
to the alternative TERN scoring formula. NOTE:
Beginning with ace-eval version 7, the scorer includes a value-based
evaluation of TERN output. Since ace-eval requires input in
apf format, and since current TERN output is in-line encoded,
the utility tern2apf must be used to transform the in-line TERN
output into apf format before running the ace-eval tool
- An updated version of the scoring script is available: ace04-eval-v07.pl
which implements scoring as outlined in the evaluation plan.
NOTE: ace-eval version 7 includes a value-based
evaluation of TERN output. However, since ace-eval requires
input in apf format, and since current TERN output is in-line
encoded, the utility tern2apf must be used to transform the
in-line TERN output into apf format before running the ace-eval
- An updated version of the scoring script is available: ace04-eval-v06.pl
This version does not yet do relation mentions, but it does
entities, relations, and entity mentions according to the evaluation
plan. It handles APF format (but not ALF).
- The new scoring script that will be used for the 2004 ACE
evaluation is ace04-eval-v04.pl. Scoring information can be
found in the PDF file ace04-eval-scoring-v3.pdf.
Page Created: September 6, 2007
Last Updated: November 4, 2008