• Read the description of the ACE scoring tool's output (pdf).
  
  
• Current Version
  • 09/01/2004
    • The improvements will never end. The script ace04-eval-v10.pl is now available.
      
      It has been observed that the F-measure as computed by ace-eval unfairly penalizes attribute recognition errors by subtracting these errors from both precision and recall. It has been suggested that this can be remedied by subtracting only half of the errors from both precision and recall. This results in a significant increase in F-measure values. For example, with a 50 percent attribute recognition error rate the F-measure improves from 50 percent to 75 percent (assuming no misses or false alarms)., Version 10 implements this change.
      
      Other notable fixes include (see script header for full list):
      * both TYPE and SUBTYPE must match for an entity to be "correct"
      * relation type/subtype "OTHER-AFF"/"Other" added to symmetric list
      
      
• History
  • 08/03/2004
    • The improvements continue. The script ace04-eval-v09.pl corrects a bug in the calculation of relation scores, performs relation mention scoring, and contains a simplified value formula for the TERN evaluation. The utility tern2apf (see below) has also been improved.
  • 07/21/2004
    • An updated version of the scoring script is available: ace04-eval-v08.pl . This version contains some general fixes as well as updates to the alternative TERN scoring formula. NOTE: Beginning with ace-eval version 7, the scorer includes a value-based evaluation of TERN output. Since ace-eval requires input in apf format, and since current TERN output is in-line encoded, the utility tern2apf must be used to transform the in-line TERN output into apf format before running the ace-eval tool
    07/12/2004
    • An updated version of the scoring script is available: ace04-eval-v07.pl which implements scoring as outlined in the evaluation plan. NOTE: ace-eval version 7 includes a value-based evaluation of TERN output. However, since ace-eval requires input in apf format, and since current TERN output is in-line encoded, the utility tern2apf must be used to transform the in-line TERN output into apf format before running the ace-eval tool.
  • 6/14/2004
    • An updated version of the scoring script is available: ace04-eval-v06.pl This version does not yet do relation mentions, but it does entities, relations, and entity mentions according to the evaluation plan. It handles APF format (but not ALF).
  • 03/02/2004
    • The new scoring script that will be used for the 2004 ACE evaluation is ace04-eval-v04.pl. Scoring information can be found in the PDF file ace04-eval-scoring-v3.pdf.

ACE04 links: