ACE corpora

Software

Current Version:

03/07/2002 : EDT_ref_compare (v28) and edt-eval (v22)
edt-eval (v22) provides an option to split multi-role entities into separate single-role entities (in which all mentions have the same role). New (sub)entities are created to collect all same-role mentions from the original entity. Names that match mention heads stay with the entity that contains the mention. (EDT_ref_compare (v28) unchanged).

02/04/2002: Sample ASR scoring version 0. Includes a sample set of instructions that will score the ASR data distributed as part of the original ACE pilot study data.

02/07/2002: Sample OCR scoring version 1. Updated because the reference images used included some empty bounding boxes. Also, there is a readme.txt file in the reference directory that explains some instances of "EMPTY BOUNDING BOXES" in submission files. Includes a sample set of instructions that will score the OCR data distributed as part of the original ACE pilot study data.

History of software:

• edt-eval (v16)
  edt-eval (v16) includes a provision for reading in an optional database of entities. Output entities are checked to see if they appear in the external database. If they do, then that assignment is used and no mapping is performed. (In other words, if the system assigns the entity ID, then that choice overrides the (locally) optimum choice of "best matching" refernce entity.) (EDT_ref_compare (v28) unchanged).

• edt-eval (v14) includes a more detailed breakout of ROLE recognition performance. This conditions the ROLE confusion matrix on entity TYPE, which allows folks to see how their ROLE assignments are doing for GPE entities. (EDT_ref_compare (v28) unchanged).

• EDT_ref_compare (v28) and edt-eval (v10)
  edt-eval (v10) conditions the evaluation on the new entity attributes CLASS and USAGE and teh new mention attributes ROLE and STYLE. (EDT_ref_compare (v28) unchanged).
  
  Description of Output:

  • For Entities:
    • MISS, the entity was not detected
    • FA, the entity was falsely detected
  • ERROR, the entity was detected but incorrectly typed (in terms of FAC/GPE/LOC/ORG/PER)
  • For Mentions:
    • MISS, the mention was not detected (in terms of the location of the head of the mention)
    • FA, the mention was falsely detected
    • ERROR, the extent of the mention was incorrectly determined

  
  Current output merely shows these statistics as a function of attribute. In particular, the analysis does NOT display any information regarding whether the new attributes were confused. (E.g., the error results do not show statistics regarding the confusion of generic class entities with specific class entities.) Easy enough to show but the options are so vast, guidance is needed from researchers.

• EDT_ref_compare (v28) and edt-eval (v09)
  
  Converts mention type "LOC" to "LOCATION".
  Adds bonus to the entity mapping score for entities with matching TYPE, CLASS, and USAGE only when unbiased score >0. This is necessary because the introduction of metonymy allows a single mention to refer to more than one entity. Thus the mapping would be ambiguous without additional information.
  
  
• EDT_ref_compare (v27) and edt-eval (v08) contain bugs and should not be used.
  
  
• EDT_ref_compare (v26) and edt-eval (v07)
  
  Fixes a problem in v25. The conversion of GSP to GPE was not being done for mention roles
  
  
• Version 25 of EDT_ref_compare

  Supports the new annotation tags based on 4 new attribute definitions (2 for entities, namely CLASS and USAGE, and 2 for mentions, namely ROLE and STYLE.

• Version 24 of EDT_ref_compare (v24) and edt-eval (v04)
  

Corrects a problem with the mention detection scoring in EDT, namely that mentions with correct extents were being counted twice.

• Version 23 of EDT_ref_compare
  

Same as previous version with a bug fix to reinstate ability to output html visualization files.

ACE Phase 2 Home

Documentation

Schedule

Resources

Contacts

ACE Home