Questions and Answers concerning the Information Retrieval Task.

• What are the file formats used for the SDR99 Information Retrieval Task ?
• What are the tools available ?
• How can I score my results ?

Q: What are the file formats used for the SDR99 Information Retrieval Task ?

A: All SDR retrieval must be submitted to NIST in the standard TREC ret format as follows :

The ret format is a space delimited ASCII table :

• One record per line
• 6 fields per line :
  1. Topic ID
  2. Currently unused (must be Q0)
  3. Story ID of retrieved document
  4. Document rank
  5. Retrieval system score (INT or FP) which generated the rank
  6. Site ID

Here is a sample of a ret file:
23 Q0 19980104_1130_1200_CNN_HDL.0034 1 4238 ibm-cr-att-s1
23 Q0 19980105_1800_1830_ABC_WNT.0143 2 4223 ibm-cr-att-s1
23 Q0 19980105_1130_1200_CNN_HDL.1120 3 4207 ibm-cr-att-s1
23 Q0 19980515_1630_1700_CNN_HDL.0749 4 4194 ibm-cr-att-s1
23 Q0 19980303_1600_1700_VOA_WRP.0061 5 4189 ibm-cr-att-s1
...

For the Unknown Story Boundary condition, the ret file must be generated by UIDmatch.pl. The input format for UIDmatch.pl is basically the same as ret. Only the StoryID field is replaced by a timeID TimeId = <episodeID>:<Time-in-seconds.hundredths>
Example : 19980104_1130_1200_CNN_HDL:34.14

Back to top.

Q: What are the tools available ?

A: Different tools are available, as follows:

• Trec_eval :
  trec_eval is the NIST IR scoring package used for SDR. It can be found at ftp://ftp.cs.cornell.edu/pub/smart/.
• UIDmatch.pl :
  UIDmatch.pl is a perl script used to match timeID generated for the Unknown Story Boundary condition to storyID that can be scored by trec_eval. UIDmatch.pl can be found here.

Back to top.

Q: How can I score my results ?

• Scoring for the Known Story Boundary condition.

  The procedure for implementing scoring for this condition is as follows:
  Example : trec_eval -q SDR99.qrels output.ret

  Note: the SDR99.qrels file will be made available at scoring time

• Scoring for the Unknown Story boundary condition.

  The procedure for implementing scoring for this condition is as follows:

  1. Download UIDmatch.pl from our ftp server.
  2. Edit UIDmatch.pl and set the $ndxdir variable to the path of the directory where all the index files are located on your system.
     If necessary download the ndx files.
     Note that you need to use the ndx files for the known story boundary condition for the maping procedure.
     
  3. Map the timeID to storyID using UIDmatch.pl. UIDmatch my-ret-file.ret > my-mapped-ret-file.ret
  4. Score like in the known story boundary condition.

Back to top.