This paper presents the evaluation of two peripheral displays for email notification and discusses some of the techniques used to measure usability, awareness and distraction in ambient displays.
The authors begin with a discussion of related work in the area of evaluation regarding peripheral displays. They cite various techniques used including questionnaires, dual-task lab studies, Likert scales to self-report awareness and distraction, implicit measures of awareness such as priming techniques, field studies and user interviews.
The authors indicate that awareness is usually tested with users self-reporting on questions. The questions, which have not been standardized, are typically based on attention, the use of the periphery and an "overall sense of information"; whereas, distraction is traditionally measured in lab studies in terms of response time. However, the authors indicate that users have been asked to self-report distraction in the field.
The paper next presents the two email notification designs. The first design presents a ticker-like display above the task bar that presents information about incoming email, such as sender, subject, etc. The second display is an off-the-desktop application; a glowing orb that sits next to an individual's computer and changes colors to display notifications about incoming mail.
A heuristic evaluation of both displays was conducted and subsequent changes were made to both devices. The authors then chose a series of evaluation techniques "because there is no consensus about the best approach for evaluating peripheral displays."
The "lab study was a dual-task study in which [the authors] included measures of usability distraction and awareness. [The] study methods included self-reporting of awareness and distraction; and objective records of performance on the primary task and on questionnaires about the peripheral display contents. [The] field study included self-reporting of awareness, distraction and usability; objective records of performance on questionnaires about the peripheral display contents; and qualitative information from interviews."
The lab study was a between subject design with 13 participants using the ticker and another 13 using the orb. Participants were given a primary task and were told to monitor the peripheral display while conducting their task. Baseline data was gathered for the first three minutes and then the peripheral display was used for an additional 12 minutes. Participants were asked to remember as much information from the peripheral display, as they would be given a quizzed at the end of the session. Following the evaluation, participants self-reported on awareness and distraction.
The authors also conducted a field study which included four participants (two using the ticker and two using the orb) over the course of four weeks (one baseline week, followed by two weeks with the display present, followed by another baseline week). At six random intervals throughout the day, participants would be asked a series of questions (via a pop-up window) about awareness (similar to the final questionnaire used in the lab study). Participants were interviewed twice during the filed study as well as answered a final questionnaire and a series of questions based on the heuristics used in the formative heuristic evaluation.
Awareness was measured as follows: self-reporting questions including how much attention users paid to a display and knowledge questions to determine how much information the user retained. Distraction was measured as follows: self-reporting questions and changes in primary task completion including speed and accuracy. Usability was measured as follows: participants rated displays in terms of heuristics.
The findings include:
- Awareness:
- Participants in the lab self-reported a higher awareness of the ticker. In the field, participants self-reported a higher awareness of the orb.
- Participants with the ticker in the field scored higher on knowledge questions.
- In the field, participants stated that the orb provided information on email status, even when participants were not at the computer.
- Distraction:
- Participants in the lab rated distraction the same on both displays.
- Participants in the field said the orb was not distracting, whereas one ticker user asked that the display be revised to be more distracting.
- In the lab, participants' speed and accuracy decreased with the ticker. Whereas, speed remained the same with the orb and accuracy actually increased.
Based on these results, the authors compared the two types of studies. They conclude that the information gathered in each method was similar, yet some new aspects were discovered in the field. The authors also believe that self-reported is "most useful when participants are able to use the display in a realistic setting, which suggests that even a short field study is better than a lab study using this technique."
The objective measures of awareness (knowledge questions) and distraction (speed and accuracy) provided less useful information. The authors believe this is due in part to the fact that the displays performed similarly. The authors conclude that self-reporting and interviews "provided the most helpful information about display design."
|
