Department of Information Science

University of Pittsburgh

Pittsburgh, PA 15260

In our experimental environment we can only test and evaluate the use of visual information retrieval interfaces (VIRls) by naive or briefly trained subjects. For users of this type, full-featured VIRIs have led to poorer performance and satisfaction ratings than simple textual retrieval interfaces. Our experiments with de-featured VIRls are significant because:

• they provide better experimental control for investigating the efficacy of underlying metaphors by eliminating extraneous effects or feature-specific capabilities associated with special modes and features.
• they provide a more appropriate comparison with simple textual retrieval interfaces allowing performance and/or preference comparisons between visual and textual interfaces rather than confounding this with the comparison of a simple undemanding interface with a complex and memory loading one.
• our preliminary usability evaluations demonstrate that training problems, comprehension problems, performance problems and "ratings" problems seem to be diminished for the de-featured interfaces.

Researchers in information retrieval (IR) have long searched for ways to make their system more accessible to end users and to develop new ways for users to explore data. Visualization techniques (computer methods for displaying large quantities of information graphically) appear promising as a means for achieving both goals. Information visualization can make multidimensional relationships that are difficult to extract from tabular data apparent to a trained searcher. However, unlike scientific visualizations which are largely developed and used within elite specialties, IR visualizations are targeted toward guiding the public through newly accessible oceans of online information as well. Work on IR visualization systems is at a relatively early stage. Systems such as Bead [1, 2], InfoCrystal [12, 13], and LyberWorld [3], have recently been developed as visual information exploration tools to aid in retrieval tasks. Researchers at the University of Pittsburgh have contributed to the development of information visualization systems with VIBE [11], GUIDO [6, 10], and BIRD [4, 5].

Previous user study research into IR visualization systems has largely focused on formative usability evaluations to provide feedback for system enhancement. Newby [9] tested subjects with the SPACE IR visualization system and standard text-based system called Prism. His findings showed that subjects preferred the Prism system for many of the tasks performed. Spoerri [12, 13] tested InfoCrystal’s interface with a standard Boolean interface. Ten users performed query recognition tasks and query generation tasks. Results show that new users could use the InfoCrystal interface after receiving a short tutorial, InfoCrystal performed as well as the more traditional Boolean interface, and the user feedback was positive.

Koshman [7, 8] conducted extensive comparisons between VIBE and AskSam, a commercially available text-based retrieval interface, using expert online searchers recruited from local libraries and novice searchers. Results showed that performance differences between AskSam and VIBE were minimal. Novices and experts performed the test tasks with similar speed and accuracy. There was some evidence that particular tasks were better suited to performance in the graphical environment while others were more appropriate for the text-based tool.

Results of preference measures were disturbing in that users in both groups rated the textual interface slightly higher while we had expected a preference for the graphical interface whatever its effect on performance. We believe that the expanded capabilities offered by visual browsing are masked in comparisons of this sort which limit tasks to those which can be performed using text-based interfaces. For tasks expressible as Boolean queries a text-based interface may be both more direct and less complicated and less difficult to learn than a full featured VIRI.

Kim [4, 5] examined BIRD usability in three experiments. The first assessed correlations between performance with BIRD and reasoning skill test scores. The second experiment examined users' performance in remembering how to use the system. The third experiment compared subjects' performance in constructing Boolean queries using BIRD and an informationally equivalent text-based interface for contextually constrained (shapes) and unconstrained (vitamins) data sets. Subjects displayed unexpected difficulties in using the VIRI and reported preferences favoring the text-based interface. In reviewing these results and details of the interface design Kim [4, 5] identified a variety of features that might be related to the difficulties. These include the complexity of sequential query construction as compared to nonsequential specification, the irreversibility of some operations, and lack of integration between the representation of term presence through color bars and the representation and manipulation of sets through use of the bins and the separator.

The original version of this interface is called BIRD (Browsing Interface for Retrieval of Documents) [4, 5]. Figure 1 shows a typical BIRD display. Its primary function is to support query formation. The Boolean operators are notoriously difficult to use correctly, even for users who are mathematically trained. The basic idea behind the BIRD interface is to represent documents as sets; the AND operator is effected by a device that takes a funnel shape. The OR and NOT operators are implied from the layout of a 2 x 2 output matrix.

The J. Bird interface is shown in Figure 2. Simplification was achieved by representing documents as sheets of paper, the funnel was replaced by a sorter and the matrix by 2 bins indicating 'matches' and 'does not match.’ To use the device a user selects a query term, then drags the stack of unsorted papers to the sorter. The results are displayed as proportional sized stacks in the output bins.

Korfhage and Olsen [6, 11] developed VIBE (Visual Information Browsing Environment) at the University of Pittsburgh. VIBE (Figure 3) represents points of interest (POIs) or query terms as nodes. Documents are represented by various sized rectangles suspended within the surface bounded by the POls. The internal representation of a document is a term vector, which is usually weighted, and the interface position of the document is calculated as the vector sum of the components indicated by the POls. Seven special-purpose modes are hidden in the menu system.

WebVIBE (Figure 4) retains much of the appearance of the original VIBE but the metaphor has been changed to a naive physics view of magnetism. The user is called upon to view the POIs as magnets and the documents are suspended in a magnetic force field. The position of a particular document can be thought of as being determined by the relative 'pull' of terms that are present in the document.

Both J. Bird and WebVIBE were developed using an iterative design approach and were tested after each of two design phases. Task sets were developed for each interface. Subjects were recruited from the University community and were paid for participating in the study. Participants were given a printed task list that included spaces for answers. No verbal or written instructions were given. Subjects were encouraged to talk aloud during the experiment and these verbal protocols were monitored by one of the investigators. Each subject was also given a post-test questionnaire. This survey was constructed to determine which features were problematic and to gather general impressions about system usability and attractiveness as a search tool.

The migration from BIRD to J. Bird entailed a total reorganization of the interface including a remapping of the machine sort metaphor. The first test of the interface brought up several classes of perceived shortcomings. These issues were addressed in the subsequent redesign. The second test showed a substantially improved system as indicated by the post-test questionnaire. The most interesting outcome of the testing, however, was the results of the actual task performance. Although 8 of the 10 test subjects indicated that they believed that they understood how the Boolean operators worked, only 1 of the 10 could successfully apply them to two problems given in the post-test. These subjects, however, performed Boolean operations with the J. Bird interface with over 90% correct answers!

The process of simplifying VIBE to produce the WebVIBE interface was distinctly different from that used for BIRD. The difference lies in using magnetism as a metaphor bolstered by portraying POIs with magnet icons. Behavior of the interface was changed to enhance this metaphor by animating influenced documents to move with the movement of a POI rather than postponing movement until the mouse button was released. The special-purpose modes and features were eliminated from the interface in order to make it more accessible and less confusing to naive walk-up users. Subjects found document animation to be very useful in performing tasks that required them to discriminate between documents that were positioned in regions that were potentially ambiguous.

Overall, task performance was similar on both trials for the pair of tests. The differences between the first and second round of testing were highly salient for the post-test questionnaires. For both interfaces there were decreases in negative comments and increases in positive ones at the second test. Considering that there were no instructions for using the interface apart from those that were embedded in the view, the fact that novice searchers could accurately form queries using J. Bird and could correctly form desired inferences using WebVIBE attests to the utility of these interfaces for their intended audience.

1. Chalmers, M. (1993). Using a landscape metaphor to represent a corpus of documents. In: Spatial Information Theory: a Theoretical Basis for GIS: European Conference, COSIT ’93 Proceedings (pp.377-390), Berlin; New York: Springer-Verlag.
2. Chalmers, M., & Chitson, P. (1992). Bead: explorations in visualization. In Proceedings of the Fifteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. (Pp. 330-337). New York: ACM Press.
3. Hemmje, M., Kunkel, C., & Willett A. (1994). LyberWorld – a visualization user interface supporting fulltext retrieval. In Proceedings of the Seventeenth Annual International ACM SIGIR Conference (TREC-1) Proceedings (pp. 1-20) Washington: U.S. Government Printing Office.
4. Kim, H. & Korfhage, R.R. (1994). BIRD, a browsing interface for the retrieval of documents. In Proceedings of the 1994 IEEE Symposium on Visual Languages. Los Alamitos: IEEE Computer Society Press.
5. Kim, Hanhwe. (1997) BIRD: Browsing Interface for Retrieving Documents, Dissertation, University of Pittsburgh.
6. Korfhage, R. R. (1991). Information Retrieval in the Presence of Reference Points: Part 2. Pittsburgh: School of Library and Information Science, University of Pittsburgh (Technical Report LIS034/IS91002).
7. Koshman, Sherry L. (1996)User Testing of a Prototype Visualization-Based Information Retrieval System, Dissertation, University of Pittsburgh.
8. Koshman, Sherry L. (1997) VIBE User Study Technical Report, Technical Report #LS062/IS97001, University of Pittsburgh.
9. Newby, G. (1992). Towards Navigation for Information Retrieval. Unpublished doctoral dissertation. Syracuse University.
10. Nuchprayoon, A. & Korfhage, R.R. (1994). GUIDO, a visual tool for retrieving documents. In Proceedings of the 1994 IEEE Symposium on Visual Languages (pp.64-71). Los Alamitos: IEEE Computer Society Press.
11. Olsen, K. A., Korfhage, R. R., Sochats, K. M., Spring, M. B., & Williams, J. G. (1993). Visualization of a Document Collection: the VIBE System. Information Processing & Management, 29(1), 69-81.
12. Spoerri, A. (1993). InfoCrystal: A visual tool for information retrieval. Visualization '93 Proceedings (pp. 150-157). Los Alamitos: IEEE Computer Society Press.
13. Spoerri, A. (1995). InfoCrystal: a visual tool for information retrieval. Unpublished doctoral dissertation, Massachusetts Institute of Technology, MIT