Task:

Determine task-based performance of collaborators comparing a computer supported approach to an in person approach.

Containing Scenario: Double TWIXT

Description:

By varying the tools and collaboration method (in person compared to through CSCW tools) through which TWIXT is played, the team should be able to:

• Determine how collaboration tools compare to in person collaborations in terms of the amount of time required to perform a task.
• Determine how collaboration tools compare to in person collaborations in terms of how well a task is performed.

Depends on:

Running this experiment is dependent on:

• The ability to code the TWIXT game.
• Access to a pool of evaluation subjects.

Capabilities:

This experiment is intended to exercise the ability to:

• Interactively manipulate objects in a shared environment.
• Communicate in a synchronous manner.

Metrics:

Metrics may include:

Performance Data

• The average number of moves it takes to win a game or the number of moves before a game is lost, comparing tools evaluated and comparing computer supported play to in person playing.
• The average amount of time spent discussing moves, comparing tools evaluated and comparing computer supported play to in person playing.
• The average amount of time required to play a game, comparing tools evaluated and comparing computer supported play to in person playing.

Subjective Data[1]

• Logs of communications between players
• Ease of communicating intentions to other player
• Audio quality perceptions
• Visualization medium quality perceptions
• Audio / Video synchronization perceptions
• Ease of operations

Operating Procedures:

The actual evaluation plan would be developed when requested. The general procedures of game play have been described in the Scenario Description.

Systems to be evaluated: TBD