Paper Summary: |
This paper presents the results of a comparison study between two systems: a traditional software interface and an agent-assisted interface. Each system assists in uploading, formatting, and developing an online news site. Both systems convert text-based articles (printed in a daily college newspaper) to an online website. However, the non-assistive system requires that users meticulously sort articles into categories, where the agent-assisted system uses logic based on the file name, author and keywords in the article to place the article into an appropriate category (i.e. news, sports, etc.)
During the study, four online production assistants used both systems to produce the news website. The evaluation included a set of measurable usability objectives, such as a "20% reduction in sorting time per story with no significant increase in error rate." During the evaluation, the following metrics were captured: sorting time, placement time, user errors, agent errors, increase in error rates, user work styles, user preference for a system, user satisfaction, and user estimation of their own and the system's performance.
The study showed that the agent-assisted system met part of the usability objective by reducing sorting time by more than 20%. However, error rates increased from one error per day (with the non-assistive system) to a daily error rate of 1.5 with the agent-assisted system. The authors conclude that the increase was small enough to claim success.
The results also show that users accepted the agents rapidly, quickly adapted their working styles to the agent, and preferred the agent over the non-assistive system.
|
