View Project Background

• Overview of the project
  
  
• Our goals
  
  
• Review of existing frameworks
  
  
• Development of a ubiquitous computing framework
  
  
• Next steps
  
  
• References

---

Overview of the project

Off-the-desktop, next generation interactive systems are quickly becoming a reality.

These ubiquitous computing applications are charting new waters, utilizing new types of interaction, and providing collaboration among greater numbers of users.

As these new systems weave their way into society, it is critical that these new systems are accepted and utilized. However, it is difficult to determine what makes for a good design and a successful interaction because evaluation methodologies and metrics are in their infancy for these new types of systems.

These so-called “off-the-desktop” systems are challenging to evaluate because they can be multi-modal, collaborative, and involve large amounts of diverse information.

For instance, many of these off-the-desktop systems are based on devices where speech, gesture, physical interactions, as well as traditional graphical and text-based user interfaces, can be used as interaction modalities.

Further, these interactions are made even more complex because many of the applications involve access to large amounts of diverse information, collaboration among users, and multi-task, distracting environments.

The complexity and diversity of these systems has made it difficult to establish common evaluation techniques and practices. However, the necessity of such a framework is overwhelmingly apparent.

Back to top

---

Our goals

Our objective is to develop and validate human-computer interaction (HCI) metrics for evaluating the user interface components of next generation interactive systems.

Working with leading researchers in both academia and industry, we plan to create a framework of metrics and measures, validated through case studies of research systems. These results will enable the development of standard evaluation methodologies and metrics, as well as guidelines and best practices for designing user interfaces for these systems.

The creation of a common approach to evaluation will require a good deal of experimentation and measurement by the research community. This can be facilitated by the establishment of a consistent terminology, an initial set of metrics, and a framework in which researchers can share and learn from each other's evaluations. The benefits from such an approach will be validated metrics, effective discount evaluation techniques, and design guidelines which can then be put to use to improve the human computer interaction of these next generation systems.

Back to top

---

Review of existing frameworks

Definition of usability

Traditional usability evaluations have focused on three metrics:

• efficiency,
• effectiveness, and
• user satisfaction (ISO 92411-11).

While these metrics have proven very helpful in evaluating traditional applications, they are not sufficient for next generation systems.

Usability and Values

Traditional usability metrics have focused on interaction, while not typically including values such as trust, confidence and utility. However, other attempts have been made to incorporate values. For instance: Friedman and Kahn ¹ discuss the distinction between usability and values and suggest 12 key human values with ethical import:

1. human welfare,
2. ownership and property,
3. freedom from bias,
4. privacy,
5. universal usability,
6. trust,
7. autonomy,
8. informed consent,
9. accountability,
10. identity,
11. calmness, and
12. environmental sustainability.

However, usability issues, such as interaction, are not represented.

New Definitions of Usability

To address some of the weaknesses of traditional usability metrics, Jameson ² proposes five new usability challenges:

1. predictability and transparency,
2. controllability,
3. unobtrusiveness,
4. privacy, and
5. breadth of experience.

While Jameson's work focuses solely on usability, values such as trust and confidence are missing.

Need for Ubiquitous Computing Evaluation Framework

Although many of the existing frameworks touch upon important issues, there is a clear need for a cohesive set of metrics and evaluation areas that include elements of usability, interaction, values (such as privacy and trust), and some of the new usability challenges presented by ubiquitous applications.

Back to top

---

Development of a ubiquitous computing framework

In developing an initial framework for HCI evaluations of next generation interactive systems, we proposed a framework ³ which includes the following seven user evaluation areas along with sample metrics and conceptual measures:

1. Attention
2. Adoption
3. Trust
4. Conceptual Models
5. Interaction
6. Invisibility
7. Impact

The initial framework was proposed by Jean Scholtz and Sunny Consolvo in December 2003. View the paper below:

Towards a Discipline for Evaluating Ubiquitous Computing Applications [PDF, 110 Kb]
National Institute of Standards and Technology
Intel Research
December 2003

In February 2004, the framework was presented at the Human Computer Interaction Consortium (HCIC) ⁴ workshop. View the presentation below:

Towards a Discipline for Evaluating Ubiquitous Computing Applications [PDF, 184 Kb]
Human Computer Interaction Consortium (HCIC)
Winter Park , CO
February 2004

Back to top

---

Next steps

The framework is just an initial framework based on evaluations from desktop computing and literature reviews and must now be applied to next generation interactive applications and validated.

Many questions are still unanswered.

• Is the framework complete? Are there any user evaluation areas that are missing – for example robustness and enjoyment?
• Is the framework useful?
• Does it provide metrics and measures that can differentiate systems?
• Can the framework be used to predict which systems will be useful and accepted by users?
• Are there additional metrics?
• Are there other stakeholders? The current metrics focus on direct stakeholders; should we consider including metrics for indirect stakeholders?
• What are the interactions/correlations between the different areas?
• Should indirect measures be included?

The framework needs to be applied to actual off-the-desktop applications and its utility evaluated and validated.

Currently, the evaluation framework has been applied to a Handheld Restaurant Ordering System, but more case studies need to be conducted in order to test the framework with a diverse set of applications. Additional case studies will enable us to fully refine and complete the framework and methodology as well as address the questions identified above.

Back to top

---

References

¹ Friedman, B., Kahn, Jr., P.H., "Human Values, Ethics, and Design," The Human Computer Interaction Handbook, Lawrence Erlbaum Associates, New Jersey (2003) pp. 1177--1201.

² Jameson, A. "Adaptive Interfaces and Agents," The Human Computer Interaction Handbook, Lawrence Erlbaum Associates, New Jersey (2003) pp. 316--318.

³ Scholtz, J, Consolvo, S., “Towards a Discipline for Evaluating Ubiquitous Computing Applications”, NISTIR # TBD.

⁴ The Human Computer Interaction Consortium ( http://www.hcic.org ) was founded in 1988 as a means of fostering interaction among universities, industries, and government research laboratories interested in promoting academic training, basic and applied research, and technology transfer in the area of Human-Computer Interaction. Membership is by organization and only employees of member organizations are eligible to participate in consortium activities. Organizations must apply and show an active HCI research program to be admitted to the Consortium. NIST was admitted this year. There are currently 20 members from top research organizations such as CMU, Georgia Tech, Stanford, PARC, Microsoft, Mitsubishi, Sun Microsystems, etc.

Back to top