Home | Proceedings


The Max Model:

A Standard Web Site User Model

Gene Lynch & Susan Palmiter
Design Technologies, Inc.
4435 SW Carl Place; Portland, OR 97201
www.designtech.com
gene.lynch@designtech.com & susan.palmiter@designtech.com

Chris Tilt
WebCriteria, Inc.
2140 SW Jefferson; Suite #210; Portland, OR 97201
www.webcriteria.com
chris@webcriteria.com

 

Abstract

To measure the accessibility of information on a web site, an increasingly important issue for web developers and site owners, we have developed an approach to modeling web site use behavior by creating a standard web site user model called the Max Model. This model user is based on the GOMS [1] modeling approach, but is unique to web usage patterns by incorporating new research on web site behavior and unique assumptions about a standard web site user's characteristics. The Max Model is comprised of a set of characteristics including Cultural Characteristics, Psychological Characteristics, Training and Experience Characteristics, Internet Connections, System Characteristics, and Cognitive Capabilities. WebCriteria is using a specific embodiment of the Max Model called "Max: The Browsing Behavior Agent." Their embodiment of the model can be found in WebCriteria's SiteProfileä , that characterizes a web site and compares it with other web sites. This paper describes a study that compares the Accessibility Metric values from SiteProfileä to actual user time to find comparable information in competitive web sites. This study begins to validate the use of a model web user. Other web usability metrics focus on the HTML for individual pages and individual measures within a page. This approach attempts to incorporate the human, as embodied in the Max Model, in a more holistic measurement while considering the whole web site consisting of all paths and connections.

 

Introduction

With the maturation of the web, usability of web sites has come to the fore. Site owners and developers are no longer content to have a "cool" web site, but are asking if real users can find what they are looking for in a satisfying experience. The usability of a site has been shown to have a strong impact on user re-visitation, which significantly affects the success of a web site. More specifically and with increasing frequency, web site owners and developers want to know if web users are able to access information in their web site more easily than in their competitor's web site.

Accessibility has become a critical aspect in evaluating the usability of web sites. Our research strongly suggests that the accessibility of information within real web sites can be quantitatively and subjectively observed and measured using real users (see "Accessibility Metric Validation" below). Yet, running usability tests on a web site of interest and on those of its competitors to determine the accessibility within sites is often prohibitively expensive and is often too time consuming to be feasible in "Internet time." We present an approach that efficiently characterizes the accessibility of a web site by grounding the analysis in a Standard Web Site User that we call the Max Model. By using the Max Model, a descriptive measure can be calculated to objectively compare web sites based on their structural differences.

 

The Max Model Defined

Standard Web Site User

The Max Model (for Maxwell or Maxine) is based on the GOMS [1] modeling approach with its "model human processor." We have applied and extended this fundamental approach to modeling human behavior in web sites and created the Max Model (Maxwell or Maxine), a standard web site user. This allows us to characterize the standard web site user's experience with a specific web site.

Since the rest of this paper hinges on an understanding of the GOMS modeling approach, we briefly summarize it here. A human model processor is made up of a perceptual system, a motor system, and a cognitive system, each with its own processing cycles, supporting memories, and actions. Card, Moran, and Newell provided this model processor with an instruction set by modeling the human behaviors in an associated GOMS model. A GOMS model decomposes a task and the associated user responses into a set of Goals, Operators, Methods, and Selection rules. Goals range from high level to fine-grained unit tasks (e.g., find a product on a web site). The Operators are individual perceptual, cognitive, or motor actions (e.g., read a word). A method describes how a goal may be accomplished (e.g., move the cursor with a mouse and click on the link). Since there may be more than one method for accomplishing the goal, selection rules are used to choose between the multiple methods. A selection rule determines which method should be applied to accomplish the goal.

In the tradition of standard user Models

The rich environments of color and sound have been modeled using similar standard user modeling techniques to describe an individual human experience. This modeling has allowed for specification, design, and legislation, all dependent on repeatable measures provided by employing the standard models. For example, in 1931, the International Commission on Illumination (Commission Internationale de l'Eclairage, or CIE) developed a standard observer that allowed for the measurement of color and enabled the field of colorimetry [2]. Sound pressure level instruments use various scales to "model" loudness responses and "integrate" the spectrum of sound into a single meaningful measure [3]. It is in this tradition that we employed GOMS modeling techniques to create a standard web site user.

 

How the Max Model works

This standard web site user is a seeker of information. The Max Model contains all the structure, rules, and definitions needed to find a target piece of information on a web site. This model needs to be supplied with a web site along with a starting point, path to be followed and a target within that web site. With that information and his various characteristics, the Max Model can be used to describe user behavior. In the following description we often personify the behavior in the Max Model to reflect the "character" of the model.

The Max Model's Characteristics

Like other standard models, the Max Model is defined by a set of characteristics (see Figure 1) to support that information-seeking behavior. These characteristics feed into the Personal Profile, System Profile, and Cognitive Capabilities that result in a model for web site use behavior.

The Max Model's Personal Profile

First, we will point out a few of the current factors in each of the Max Model's Personal Profile characteristics as seen in Figure 1 and describe how they apply to a model web site user's behavior:

Cultural Characteristics: Language is one of the Max Model's cultural characteristics. The Max Model's language will affect the reading speed and browsing behavior based on the language reading characteristics and the language(s) provided by the web site.

Psychological Characteristics: Max is persistent - he doesn't give up. This allows the model to run until completion. Other types of psychological characteristics for the current embodiment of Max are:

Max has a short-term memory capacity of 7 chunks of information

Max can remember only the structure of the previous page view

Training and Experience Characteristics: Examples of the types of behavior and strategies that our current Max has learned through training and experience, or the lack of it, are as follows:

Max scans the visible view before scrolling to find new elements.

Max doesn't back track in pursuing target information.

Max uses the point and click method of navigation exclusively.

Max looks for information without using search engines.

Max doesn't bookmark.

Max is patient and doesn't bail out because of long page load times.

If Max sees a form, it's on the correct path to his target and he fills it in.

 

The Max Model's System Profile

Following are factors in each of the Max Model's System Profile characteristics as seen in Figure 1 and describe how they apply to a model web site user's behavior:

Internet Connection: The transfer speed provided by the Max Model's internet connection will determine how long Max waits for information and may affect behavior such as how he determines when to click on a link.

System Characteristics: By specifying the Max Model's hardware capabilities, type of browser, and the tool bars included in its settings we can determine the amount of information in a specific page view and when Max needs to scroll.

 

The Max Model's Cognition

The selection rules and behaviors that the Max Model employs is drawn from the emerging body of literature and research concerning user behavior while using the web [4-9] and from previous research concerning display layout [10].

Examples of the rules and behaviors that our current Max's Cognition embodies and are used to find target information are as follows:

Max employs common binary-search behaviors in comparing and selecting paths to "explore." The current embodiment of Max identifies two links, the two are compared (involving cognition time), and one is selected to continue. This type of pairwise comparison and selection for careful link-by-link examination is employed when comparing two individual or groups of links.

Max makes decisions based on purely structural elements, not semantic content. Due to the lack of readily usable semantic content analysis tools, Max currently does not base decisions on semantic content. Instead Max uses the structure of the web site to base decisions upon.

Max decides when to click on a link based on a probabilistic model of "click out" behavior. Instead of requiring Max to wait until an entire page loads, Max sometimes recognizes a link on a page at some time during the page loading state, sometimes after the page is fully loaded as soon as the link is seen, and sometimes Max must look through a fully loaded page more than once to find the link on the path to the target.

Figure 1: The Max Standard Web Site User Model.

The Max Model Family

A specific set of characteristics must be determined and fixed to operationalize the use of the Max Model. For example, the speed of the Internet connection effecting transfer speed is defined prior to using the Max Model to approximate accessibility. Any of these characteristics can be modified to reflect a certain type of user or use behavior for further study. For a simple example, if Max is to be a typical office worker, the Internet connection may be a T1 connection whereas if Max is to be a home user, the Internet connection may be a 28.8 K modem connection.

In our current work, we have confined the Max Model to have information seeking behaviors. The Max Model could be expanded to have characteristics that would describe other web behavior such as searching or online purchasing.

 

The Use of the Max Model to Characterize a Web Site

With the Max Model defined, this standard web use model can be used to characterize the accessibility of a web site as shown in Figure 2. To characterize the accessibility of the entire web site, the whole web site or a portion of it is abstracted or modeled. From this model or abstraction of the web site, a control program uses an operational abstraction of Max to move through the web site starting at a specified starting point and moving through a defined path to a target piece of information. During this path traversal, events and data are stored about the experience. These events and data are accumulated for all possible targets within the web site abstraction resulting in a metric that describes the experience of Max, our model user, in finding information in the web site of interest.

The Max Model is currently embodied in WebCriteria's SiteProfileTM service. The Accessibility Metric of SiteProfileTM measures and compares the accessibility of web sites. SiteProfileTM also compares web sites on the basis of load time, freshness and composition. Currently the SiteProfileTM Accessibility Metric is descriptive of the experience of looking for information within a web site. The Accessibility Metric is an objective comparison between web sites or design versions of a specific web site instead of predictive of a user's experience with a particular web site or design.

As with any human processor modeling, some aspects of human behavior are not taken into account in the Max Model. In the area of web page traversal, the following user behaviors that are currently not incorporated in this model include:

Error behavior in the form of wrong paths

Task modification based on the information contained in the web site

Use of context to search for text patterns

 

Figure 2: Using the Max Model as a Standard Web Site User to measure or characterize a web site.

 

 

Accessibility Metric Validation

To validate the current abstraction of the Max Model, a study was conducted to determine if differences between web site accessibility on specific tasks could be identified. If differences could be detected from the objective and subjective experience of actual users, the WebCriteria accessibility metric, based on the Max Model, should also be able to reflect these differences.

To validate the current abstraction of the Max Model embodied in the Accessibility metric a study (depicted in Figure 3) was conducted to determine if differences between web site accessibility on specific tasks could be identified. If differences could be detected from the objective and subjective experience of actual users, the WebCriteria Accessibility Metric, in the form of specific Accessibility values, should also reflect these differences. The Accessibility Metric is an amalgamation of the specific Accessibility values.

Figure 3: Accessibility Metric validation study.

Strategy

Following is the strategy we are employing to validate the Accessibility metric, an abstraction of the Max Model:

  1. Determine if "accessibility" is definable and a significant factor in web site usage.
  2. Determine if "accessibility" is observable, measurable, and that web sites can be differentiated by it.
  3. Select tasks that are common to different sites, but would be expected to be equally accessible given the nature of the sites.
  4. Verify that the subjective evaluation by real users correlates with the measure of time to task target
  5. Correlate Accessibility Values for the current abstraction of the Max Model with observed user behaviors and subjective ratings.

 

Method

Thirty (30) subjects with age ranges of 20-30 who had used the web for at least 1 year were given 6 information finding tasks. They were asked to find the same information (e.g., "the company's guarantee on its products") on 2 or 3 competing web sites (e.g., REI, LLBean, and SierraDesign) in a randomized block design resulting in a total of 15 tasks for each subject. Each subject was given a home page and the target information to be found. Subjects were not allowed to use search, a site map unless instructed, or browser controls such as bookmarks, history, etc. If a subject was unable to find the target information within 5 minutes, they were stopped and asked to move on to the next task.

Study test facilitators hand recorded the number of clicks per view, amount of scrolling, if a page had loaded before a click led off the page, and the time from the start of the task until the end of the task (or the time limit). In addition, the subjects' performance was logged by use of a proxy server where most of the subjects' operations were recorded.

The facilitator hand calculated time was compared with the time data collected from the proxy log and it was found to be highly correlated (F[1,13] = 409, p < 0.0001).

The data collected for each task included the web page path followed, number of clicks per view, amount of scrolling, time until target was found, load times, choice times, and each user's subjective rating of the "accessibility" of the information for each task and web site.

Subjects

For homogeneity all subjects were recruited from a single college community. The average number years of web usage was 4.1 years with an average age of 22. The majority used Netscape Navigator. About half had made a purchase over the Web.

The subject population reflects a homogenous population of web savvy users. The distribution in Figure 4 shows the amount of web usage per day. Half of the subjects (15 out of 30) use the web more than one hour a day.

 

Figure 4: Web usage (in time per day) for 30 subjects.

Data

None of the 30 subjects was able to find the correct target information for all 15 tasks within the 5-minute time limit. On average, 12 of the 15 tasks were found by the subjects [range from 7 to 14 correct tasks].

Table 1 shows the data and statistical tests for the participants who found the task information within the 5-minute time limit. For each task, statistically significant differences were detected for the time to find the task information between competitive web sites (p < 0.05 in all comparisons but one) using a repeated measure ANOVA. The same statistical significance was found for comparisons of participants' performance when all data was used (successful and unsuccessful stopped by the 5 minute time limit). In order to provide the comparative rankings of these sites shown in Table 1, the Friedman Test was used for the 3-way comparisons and a Wilcoxon Signed Rank Test was used for the 2-way comparisons.

 

Table 1: Tasks shown for each web site with Mean time until found (for only those tasks that were found within the time limit) and the repeated measure ANOVA values.

Site

Mean Time (s.)

ANOVA F-Value, P-Value

Comparative Rank

Task 1

What is the company's product guarantee?

F[2,28] = 11.6, p < 0.0005

www.llbean.com

140.7

3

www.rei.com

78.6

1

www.sierra-designs.com

88.4

2

Task 2

What is the name of a tent that will sleep 4 and weighs under 12 lbs?

F[2,17] = 16.4, p < 0.05

www.llbean.com

169.3

3

www.rei.com

161.1

2

www.sierra-designs.com

118.6

1

Task 3

What is the price of the cheapest mini van from this company?

F[1,14] = 10.6, p < 0.05

www.honda.com

178.6

2

www.toyota.com

110.1

1

Task 4

When was the company established in the United States?

F[1,6] = 16.3, p < 0.10

www.toyota.com

163.3

can't rank *

www1.daimlerchrysler.com

246.4

can't rank *

Task 5

Who is the CEO of the company?

F[2,20] = 44.8, p < 0.0001

www.dell.com

133.3

2

www.compaq.com

162.7

3

www.apple.com

98.9

1

Task 6

Using the site map, what is the phone number to get more information about a loan for a college student who wants to buy a computer?

F[1,15] = 4.9, p < 0.005

www.dell.com

190.3

2

www.apple.com

128.1

1

 

* Non-parametric tests were not run on data that did not show a significant difference with an ANOVA repeated measures test.

The subjective data for "accessibility" of finding each task (as detailed below) was compared to the time per task. A strong correlation was found (z[15] = 5.0, p < 0.0001) as seen in Figure 5. The subjective data was only collected from subjects who were actually able to find the task information. This subjective data mirrors the time data presented above in relative ranking of the sites.

 Figure 5: A comparison of average time and average subjective ratings

.

The subjective rating scale tracks with the reported time to complete the task

    1. _____ Highly accessible
    2. _____ As accessible as information on the average web site
    3. _____ Less accessible than information on the average web site
    4. _____ Least accessible or inaccessible

 

 

Accessibility Value and Metric

The WebCriteria SiteProfileÔ Accessibility Metric is an accumulation of the accessibility scores for each target page within the spidered portion of the web site explained in http://www.webcriteria.com/our_tech/index.htm. The values reported in Figure 6 are for a single target only, as opposed to the entire Accessibility Metric for an entire site. Thus each value to a target piece of information is called an "Accessibility Value."

 Figure 6: A comparison of users' average time for each task and the WebCriteria Accessibility Value. Note: The Accessibility Value is available for 13 out of the 15 tasks.

 

Of the tasks for which there was user data there, 3 pairs can be compared for Task 1, 3 for Task 2, 1 for Task 3, 1 for Task 4, 3 for Task 5 and 1 for Task 6. This is a total of 12 possible pairs that can be used to indicate if the model is generating similar orderings to the observed times and subjective ratings for these pairs. For these 12 paired comparisons we have Accessibility Values for 10 of them. Of the 10 pairs for which there was user data and a computed Accessibility Value, 8 of the 10 assessments indicated the same order.

Table 2: Paired comparisons of relative orders of average user time versus Accessibility Values.

Paired Comparison

Average time and Accessibility Value are same relative order

Task 1

llbean vs. rei

NO

Task 1

llbean vs. sierra-design

YES

Task 1

rei vs. sierra design

YES

Task 2

llbean vs. rei

?

Task 2

llbean vs. sierra-design

YES *

Task 2

rei vs. sierra design

YES

Task 3

honda vs. toyota

YES

Task 4

toyota vs. chrysler

?

Task 5

dell vs. compaq

YES

Task 5

dell vs. apple

YES

Task 5

compaq vs. apple

YES

Task 6

dell vs. apple

NO

* The Accessibility Value for the page before the LLBean Task 2 target information was 2.3, so the actual Accessibility Value must be concluded to be larger than the Sierra Design Task 2 Accessibility Value.

 

 

Results

Accessibility can be tested and measured. We have established that different sites that are navigated by a homogenous user group result in significantly different user experiences based on the web site design. Results show that users significantly experience different speeds and subjective feelings for access to information tasks on different sites containing similar information. Users report this experience, testers observed the result, and the user time data correlates to it.

Another encouraging result is that the Accessibility Values in 8 out of 10 comparisons aligned with user data. This is a promising and intriguing result, but not yet conclusive.

In addition, we were able to observe user behaviors that we have built into the Max Model. The Max Model is currently being abstracted and used as the basis for metrics in the first release of WebCriteria's SiteProfileÔ . The next step in validating the Max Model and derivative metrics is to verify additional aspects of human web behavior to evolve the Max Model.

 

The use of Max compared to other web usability metrics

Currently there is only one other automated, web usability metric publicly available. The NIST WebSAT tool [7] analyzes HTML on a page by page level looking for violations of currently held "acceptable" or "unacceptable" coding practices as they pertain to usability. As these authors acknowledge in their 1998 paper, they want to begin to look for interactions between pages.

In contrast, the Max Model looks at both within and between page issues as they relate to usability. In addition, the Max modeling approach accounts for the user's behavior towards items on a "viewable page" or only that portion of a web page that can be seen at any one time. Instead of using the web page as the locus of interest, the Max Model uses the human model as the focus and those things perceivable objects to act upon.

 

Future Work with Max

The current embodiment of the Max Model is found in the computational metric devised by WebCriteria. They have abstracted web sites for the purpose of measuring them and have operationalized Max so that an accessibility metric can be computed. In the future, The Max Model, Max, and the derived metrics will undergo further testing, verification, and tuning.

The standard user model, Max, will be continually tuned and updated to include new research findings in the area of web based behavior. For example, eye-tracking studies using web sites may help us uncover better patterns and order of scanning when looking at a new web page.

Future work may include the modeling of others in the Max Model "family" for other types of standard web use. This standard user may also be used to develop other metrics based on how the Max Engine (control program) (see Figure 2) is realized.

Conclusion

Currently, the main conclusion to be drawn from this study is that use data can be consistently and reliably measured about the differences in finding particular tasks on competing web sites. The data is highly correlated (internally consistent) and conclusive that a set of homogenous users presented with the same task will have similar results in terms of time and subjective experience when searching for that task on sites that were designed to support those tasks. In addition, early but encouraging data suggests that these user data match 8 out of 10 Accessibility Values based on the abstraction of the Max Model.

The use of the Max Model to describe the accessibility of information in a web site can not replace traditional web usability evaluation methods. These evaluation methods that include tools such as usability testing, heuristic reviews, and design walkthroughs, cannot be exhaustive. Thus, a tool using the Max Model, a standard web site user, allows for a fast, objective, and repeatable characterization of the use behavior enabled by a web site's design. Measurements based on the Max Model may be used to determine when a more detailed usability study is necessary as well as providing a general characterization of the usability of a web site.

 

References

  1. Card, S., Moran, T., Newell, A. (1983) The Psychology of Human-Computer Interaction Lawrence Erlbaum Associates, Publishers, Hillsdale, New Jersey ISBN 0-89859-243-7
  2. 1931 CIE International Congress on Illumination. Proceedings, International Congress on Illumination, Cambridge. Cambridge University Press.
  3. Young, R. W. (1964), Single-Number Criteria for Room Noise. Journal of the Acoustical Society of America, Vol. 36, No.2, February 1964, pp. 289-295.
  4. Nygren, E. & Allard, A. (1996) Display Design Principles Based on a Model of Visual Search Center for Human-Computer Studies. Uppsala University, Else.Nygren@cmd.uu.se
  5. Trumbo, J. (1998) Spatial Memory and Design: A Conceptual Approach to the Creation and Navigable Space in Multimedia Design. interactions v.4 pp. 26-34
  6. Spool, Scanlon, Schroeder, Snyder, and DeAngelo (1997) Web Site Usability: A Designer's Guide. User Interface Engineering; North Andover, Massachusetts
  7. Scholtz, Laskowski, & Downey (1998) Developing Usability Tools and Techniques for Designing and Testing Web Sites 4th Annual Conference on Human Factors and the Web. http://www.research.att.com/conf/hfweb/proceedings/scholtz/index.html
  8. Nielsen, J. (1996-1999) http://www.useit.com/papers/ A web based collection of papers on Web Usability.
  9. Detweiler & Omanson (1996) Ameritech Web Page User Interface Standards and Design Guidelines http://www.ameritech.com/corporate/testtown/library/standard/index.html
  10. Tullis, T. S. (1984) A Computer-Based Tool for Evaluating Alphanumeric Displays, Proceedings of IFIP INTERACT'84: Human-Computer Interaction, 1984, p. 719-723.

"The Max Model: A Standard Web Site User Model"
<- Back

Thanks to our conference sponsors:
A T and T Laboratories
ORACLE Corporation  
National Association of Securities Dealers, Inc.

Thanks to our conference event sponsor:

Bell Atlantic


Site Created: Dec. 12, 1998
Last Updated: June 10, 1999
Contact hfweb@nist.gov with corrections.