This paper presents a framework to empirically evaluate notification systems in such a way that design systems can be compared and results can be used to inform future designs.
The authors support the concept of "critical parameters" which is comprised of three characteristics: "the satisfaction [of the parameters] is critical to the success of the system, [the parameters] are persistent across successive systems and [can be manipulated] by designers." To overcome some of the challenges of critical parameters, the authors identify suggested weaknesses of the concept and offer suggestions to resolve these issues and identify areas for future work.
Employing critical parameters to notification systems, the authors propose three measures including: interruption, reaction and comprehension (IRC). Defining each of these parameters in detail, the authors present an empirical measure for each concept. The first measure, interaction is defined by the cost of interrupting a user's primary task (cost of interruption) and the reduced performance when the notification is presented (primary task sustainment). Reaction is defined by how often a user notices notifications (hit rate) and how quickly he responds to that notification (response time). Comprehension is defined by the ratio of a user's interactions with a system to the number of notifications (perception rate), how much the user remembers from the notification (base comprehension) and how successful the user is in making predictions on future behavior of the system (projection).
To evaluate these parameters, the authors propose an empirical method (involving user evaluation) and an analytical method (based on surveys conducted with experts). To evaluate the effectiveness of the parameters, the authors conducted an analytical comparison of three versions of the same system. A group of experts was shown each system and asked to individually rate the system through a survey. Answers to the survey were used to calculate interruption, reaction and comprehension.
The authors conclude that the IRC parameters were successful in assessing each design and provided a comparison of the designs. The authors are hopeful that others will embrace the idea of critical parameters and develop measures for additional concepts that can be applied to systems. They hope that this acceptance will lead to an environment where results of user evaluations can be re-used and applied to the design of other devices.
|
