EWG Map Navigation Experiment

The following steps were used to develop an experiment to test the the Collaborative Virtual Workspace's support for collaborative planning. We document here our execution of these steps.

Steps in Planning an Experiment

1. Identify the system.
2. Identify what is to be evaluated.
3. Establish hypotheses.
4. Pick a scenario for the evaluation or develop one by identifying work task types the system is going to support.
5. Design the experiment. Identify what will be controlled. Identify what metrics and measures will be used and what methods will be used for data collection.
6. Conduct the experiment.

Step 1: The System

The CVW system is a collaborative system that supports audio and text communication, the sharing of objects in a virtual space, and the sharing of imagery through the use of a whiteboard.

Step 2: What to Evaluate

We want to evaluate how the availability of audio communication in CVW affects collaborative route planning (task type 1).

Step3: Establish Hypotheses

1. People collaboratively plan a route faster when audio communication is available.
2. People collaboratively plan a better route when audio communication is available.
3. Participants will be more satisfied with the collaborative route planning when audio communication is available.
4. Participation is more equal in non-audio mode versus audio mode.
5. Audio will shift communication to more linguistic communication.

   A

   There will be a greater number of turns which will be shorter in length when audio is used.

   B

   The whiteboard is used more in non-audio mode versus audio mode.

Step 4: Develop a Scenario

We first considered a military planning scenario:

Look at a sequence of images of troops and artillery and determine where the next move will be or where a counter move should be made. There are two analysts. One has information about troop mobility and the other has information about artillery mobility. There are several pieces of information that can be divided up among the analysts:

• topographical map
• artillery pattern books (for identifying artillery)
• historical data
• troop mobility

However, since this scenario requires domain expertise, we decided to work with a familiar activity: map navigation.

Two people must work together to determine a route between two locations. One map is shared between them, and each has their own copy of the map with additional annotations on it. Each person has a different set of annotations on their map. The annotations include:

• traffic lights
• traffic restrictions (no right turns, no U-turns, one way, heavy traffic)
• landmarks
• construction areas

The task is completed once they have determined and agreed upon a proper route.

Step 5: Design the Experiment

The experiment will investigate the influence of audio communication availability on collaborative route planning.

The Experiment

Experiments will be run on pairs of people. Each pair will use both system configurations: CVW with audio communication and CVW without audio communication). Each pair (in each configuration) will be asked to perform a study trial and a test trial. The study trial is a scenario similar to the test trial; it serves to familiarize the pair with the scenario.

Independent Variables

There is one independent variable that we will be controlling: the availability of audio communication. The are two versions of CVW, one with the audio tool VAT and one without. The participants will be doing two sets of trials using a different version of CVW for each set. The order participants use the versions will vary across experiments.

Nuisance Factors

These are variables in which we have no real interest but cannot actually be ignored. We will randomize the variables by selecting subjects with no regard to age, sex, or experience.

Design

Each subject pair is tested under both treatment conditions (with audio and without audio). We will prevent any carryover effects by counterbalancing the order of the treatment. Under each condition the participants will perform one study trial and one test trial. We're only interested in the test trial. The test trials will be referred to as trial A and trial B.

To account for any slight differences in the trials, we are alternating the order in which the trials are administered. This will eliminate confounding effects.

Each trial has two private maps with obstacle annotations. These maps are randomly assigned to participants.

There are four different experiment configurations.

1. trial A with audio then trial B without audio
2. trial B with audio then trial A without audio
3. trial A without audio then trial B with audio
4. trial B without audio then trial A with audio

We will run the experiment on eight groups of two. The assignment of groups is presented in the table below.

There are four sets of three maps that will be used by the participants. Each set contains one electronic map to be shared between participants via the CVW whiteboard and two differently-annotated maps, one for each participant. The annotated maps cannot be shared visually.

Experimental Trials

Each pair of participants will work together on two sets of two trials. Two trials (one study and one test) will be performed under each system configuration. The test trial will be more difficult than the study trial and will require

• more sharing of information,
• more decisions and
• more elaborate final results.

Trials

Study Trial

A small number of possible short routes with a small number of obstacles.

Test Trial

A larger number of possible routes with several obstacles.

Experimental Materials

The material needed for this experiment are

• a short background questionnaire,
• a short training session on the software to be used,
• a description of the experiment,
• two study trials and two test trials (each with a shared map and two private maps)
• a blank map for each trial,
• a questionnaire for each set of trials under each condition and
• a configuration comparison questionnaire.

Output Format

Output is to be in the form of an un-annotated map with the route traced from the source location to the destination location. Each participant will be required to draw out the solution individually after completing a trial. The participants will not be permitted to collaborate while recording the solution.

What to Measure

We have identified several measures based on the hypotheses defined earlier.

Hypothesis 1

People collaboratively plan a route faster when audio communication is available.

Metric:

• overall task time

Hypothesis 2

People collaboratively plan a better route when audio communication is available.

Metrics and Measures:

• expert judgment
  • length of route and/or driving time
  • violations (different weights)
  • things that can't physically be done
• user questionnaire
  • I was comfortable with the final solutions. Strongly agree ... Strongly disagree
  • Compared to the other person, I contributed to the activity Much More ... Much Less.
  • Determining the best routes was Very Easy ... Very Difficult
  • Using the tool for collaborative route planning was Very Easy ... Very Difficult

Hypothesis 3

Participants will be more satisfied with the collaborative route planning when audio communication is available.

Metrics and Measures:

• user questionnaire
  • Using this tool for collaborative route planning was Very Pleasant ... Very Unpleasant.
  • Which system configuration did you prefer?
  • Understanding what the other person was communicating was Very Easy ... Very Difficult.
  • Using the tool for collaborative route planning was Very Enjoyable ... Very Unpleasant.
  • Communicating with the other person was Very Easy ... Very Difficult

Hypothesis 4

Participation is more equal in non-audio mode versus audio mode.

Metrics and Measures:

• see Hypothesis 4
• user questionnaire (user perception)
  • I was able to contribute freely to the conversation. Strongly agree ... Strongly disagree

Hypothesis 5

Audio will shift communication to more linguistic communication.

A

There will be a greater number of turns which will be shorter in length when audio is used.
Metrics and Measures:

• turns - automate using logs and count by hand looking at transcripts
• word count or utterance count (aggregated turn level)
• time to type and time to speak

B

The whiteboard is used more in non-audio mode versus audio mode.

Metrics and Measures:

• count whiteboard pixels, annotations, higher level meta annotations, interleaving between whiteboard and audio
• user questionnaire
  • Did your use of the whiteboard differ between modes?

Step 6: Conducting the Experiment

Experimental configuration for each person

The Plan

1. background questionnaire
2. give training (Appendix A)
3. provide and explain instructions (Appendix A)
4. setup for first study trial (STA or STB) (Appendix B)
5. run through of first study trial
6. record solution (participant draws route on map copy)
7. repeat steps 4-6 for first test trial (TTA or TTB)
8. questionnaire for first trial
9. repeat steps 4-6 for second study trial (STA or STB)
10. repeat steps 4-6 for second test trial (TTA or TTB)
11. questionnaire for second trial
12. configuration comparison questionnaire
13. wrap-up

Wrap-up

• answer any remaining questions the participants may have
• discuss any interesting behaviors we would like the participant to explain
• ask user about their overall impressions
• explain what we are trying to do through the experiment

System configuration

Hardware

• Sun Sparcstation
• Sennheiser microphones
• 21" Monitor

Software

Collaborative Virtual Workspace (CVW)

• xcvw
• VAT
• MITRE whiteboard

Materials

• Study Trial A (STA)
  • map mSTAs - common map, electronic version for whiteboard
  • map mSTAx - annotated map, hard copy
  • map mSTAy - annotated map, hard copy
• Test Trial A (TTA)
  • map mTTAs - common map, electronic version for whiteboard
  • map mTTAx - annotated map, hard copy
  • map mTTAy - annotated map, hard copy
• Study Trial B (STB)
  • map mSTBs - common map, electronic version for whiteboard
  • map mSTBx - annotated map, hard copy
  • map mSTBy - annotated map, hard copy
• Test Trial B (TTB)
  • map mTTBs - common map, electronic version for whiteboard
  • map mTTBx - annotated map, hard copy
  • map mTTBy - annotated map, hard copy

Time Allotment

(These times are just estimates; they are meant to be guidelines and not strictly enforced. The participants should be aware that they need to complete the tasks as quickly as possible, but they should not be stopped before the tasks are completed.)

Subjects

8 groups of 2 subjects each

Each group is identified by g and a number. For example, group 1 is represented as g1.

Each participant is identified by p and a number. For example, participant 1 is p1. Participant 1 in group 1 is g1p1.

Appendices

A

Experiment Instructions and Training

B

Experiment Setup

C

Glossary

D

Identifiers