This paper presents a user evaluation of a multimedia tour guide of Genoa 's Coasta Aquarium, the largest European Aquarium and one of Italy 's most visited cultural sites.
The team "performed extensive testing in all phases of design, development, deployment and redesign, including early tests with expert, questionnaires, think-aloud sessions, and ethnographic observations of user behavior."
The team used a participatory design process, with input from aquarium experts and visitors, to design the system. To verify their work, the project team set three measurable usability objectives and ran early tests in the lab and field to meet these goals. The team also ran tests with aquarium experts.
To support immediate use by the public, four key factors were emphasized including: simplicity, low intrusiveness, support for natural interfaces and current-state visibility.
The first evaluation involved user questionnaires over the course of 30 days. Participants who rented the tour guides were asked to complete either a short or long questionnaire upon returning the tour guides (participants were unaware of the study until presented with the questionnaire). The study resulted in more than 150 responses. The results were statistically analyzed to determine which variables impacted the design (including usability, enjoyability, distraction, etc.)
The second evaluation included a user evaluation with ethnographic observation, qualitative and quantitative data. The evaluation included 120 subjects (40 using an audio device, 40 using a multimedia device and 40 using the textual signs in the exhibition). Again, participants were unaware of the study until the end of their visit.
Participants were asked to perform two tasks: a recognition task and an essay task. In the recognition task, participants were given an image of an animal with four answer choices and were given 90 seconds to answer, for a total of 10 questions. In the essay task, participants were asked to describe three of the aquarium tanks they had just visited.
The team measured efficiency by counting the number of correct answers in the recognition tasks. Effectiveness was measured by scoring the essay questions for accuracy and completeness. Satisfaction was based on participants' ratings on a 10-point scale.
The team then evaluated the efficiency, effectiveness and satisfaction ratings for statistical significance. The authors found that the multimedia tour guide supports users significantly better than either the audio or the exhibition signs. The authors also conducted a qualitative assessment to discover that participants using the audio were more prone to remember attributes about an animal (such as habitat information), whereas, the multimedia group was more prone to describe the animal's appearance.
The last evaluation included an ethnographic observation of 10 participants using the multimedia tour guide. The goal was to determine how the guide impacted participants' behavior.
Based on the evaluations, the project team is refining the design and has plans to deploy a wireless tour guide system.
|
