Home | Proceedings


The Effects of Cultural Markers on Web Site Use

Charles Sheppard and Jean Scholtz
Charles.Sheppard@nist.gov,   Jean.Scholtz@nist.gov
National Institute of Standards and Technology(NIST), Gaithersburg, MD, USA

Abstract

Our research involves identifying factors that affect the usability of web sites. In particular, given that many web sites are accessed by substantial numbers of international users, an important question is whether a site design that reflects a user's culture is more effective. In this paper, we describe our initial study which is a pilot test to determine if the absence or presence of cultural markers affect the user's preference or performance. The experiment we performed found no preference ratings differences, but did indicate some performance differences. We discuss the experiment, the results, and some factors that may have influenced the results and outline some ways for correcting these factors in future experiments.

Introduction

Web sites are being developed in many different cultures around the world. The interface designs of these sites seem to be influenced by the culture in which they originate. Barber and Badre [1] performed research in this area, and state that there are prevailing interface design elements and features of a web site within a given culture which can be called cultural markers. They state that such cultural affiliation can be seen in the frequent use of cultural markers such as a national symbol, color, or spatial organization in web site designs. They were able to identify these cultural markers using a three stage process. The first stage involved categorizing hundreds of web sites by country, genre, and language. For the second stage, they performed a detailed inspection of the collected web sites and identified a list of prevalent design elements. Lastly, the cultural markers were checked for emergent patterns within countries, genres, and across regions. It is also hypothesized by Barber and Badre that the inclusion of cultural markers in web site design will improve the usability of the site for individuals from the culture for which the web site was designed. They stress the importance of using cultural markers, and have coined the term culturability to describe this special relationship between culture and usability in web site design. We are interested in devising experiments that test whether a web site is more preferred by subjects when its design is reflective of their culture, that is, a site design that contains their cultural markers.

The research by Barber and Badre is a natural follow-on to our own research [2] and [3] which involves the development of tools to support rapid, remote, and automated evaluations of web sites. Our experience in developing these tools provided insight into how they could be used to validate Barber and Badre’s research. Hence, the NIST WebVIP (Visual Instrumenter Program) became an immediate candidate since it has the functional capability to track the paths generated by user activity of a web site. We contacted Barber and Badre and proposed an experiment using the NIST WebVIP tool to prove their hypothesis that cultural markers can directly impact user performance. Our discussion of the experimental design and the results from performing the experiment are presented in the following sections.

Experimental Design

This section describes our process to determine whether cultural markers can directly impact user performance. The approach that we used was a six-step process. First, we evaluated the community of international web-based designs so that we could select a culture which contrasted with the North American culture. We chose the Middle Eastern culture because it was one of the cultures (identified in Barber and Badre's research) that contained cultural markers that could easily be contrasted with those in the North American culture.

Second, we selected a commercial web site to be used as a template for each culture. The following criteria was used:

The selected site (Edmund's at www.edmunds.com) is one used by individuals who are considering the purchase of a vehicle. It contains information about new and used cars and trucks which is a topic that is familiar to most people regardless of their culture. In addition to satisfying our selection criteria, the Edmund's site has been described in published usability tests and has received good user ratings [4]. This fact added to our confidence in choosing the Edmund's site, and we found its design to be very modular which allowed us to easily take a subset of the site.

Third, we modified the selected commercial site to create both United States and Middle Eastern versions. The original commercial site was scaled down due to the manpower required to insert the appropriate cultural markers in the entire site. Specifically, we removed links to portions of the site. Our tasks were then constructed so that the answers could be located on the limited site. Two instrumented sites, United States and Middle Eastern, were created from the scaled down version. The Middle Eastern version was redesigned to incorporate those features identified to be cultural markers for native Middle Eastern web sites, while the United States version was simply the original, scaled-down version. The cultural markers used in the Middle Eastern site were: fancy text fonts, green colored backgrounds, and such spatial orientations as the placement of menu columns on the right.

Fourth, we designed our experiment. We constructed two sets of five tasks – one set for each site. By design, subjects would attempt to locate requested information on both versions of the web site (United States and Middle Eastern). Question responses would be recorded by the subjects and returned to NIST for compilation and review. The order of the sites that subjects were asked to use was counterbalanced. However, the order of the individual tasks was not counterbalanced or randomized.

Fifth, we contacted and obtained agreement from candidates to participate in our experiment. In all, 10 subjects participated in the experiment. Of those 10, 5 were born in North America [footnote 1] which we designated as Group 1, and 5 were born in a Middle Eastern state (such as, Iraq, Jordan or Saudi Arabia) which we designated as Group 2. Gender was not considered when soliciting subjects for our experiment. Additionally, we did not consider whether the information content of the selected site would be biased in its appeal by gender. We assigned unique numbers to each subject within a group. Because the experiments were to be conducted remotely, subjects were mailed packets of information containing consent forms, demographic questions, instructions, and the tasks.

Table 1 shows demographic information for the two groups of subjects.

 

Questions

North American

Middle Eastern

#

Response

#

Response

Country of Origin

5

0

0

U.S.A

Canada

Mexico

1

1

3

Iraq

Jordan

Saudi Arabia

Primary/Native Language

5

English

5

Arabic

Secondary Language(s)

0

2

2

1

English

Spanish

French

Italian

5

1

0

0

English

Spanish

French

Italian

Years in U.S.

2

2

1

18 to 20 years

21 to 25 years

35 years

3

1

1

4 years

5 years

8 years

How many years have you used the Web

1

1

2

1

< 1 year

1 to 2 years

3 to 4 years

> 4 years

0

3

1

1

< 1 year

3 years

2 years

4 years

How often do you use the Web

2

2

1

Weekly

Daily

Never

1

4

0

Weekly

Daily

Never

Web expertise

2

1

1

1

Very good

Good

Okay

Novice

3

1

1

0

Very good

Good

Okay

Novice

Main web usage

4

5

3

1

3

1

0

Entertainment

Information

Education

News

Travel

Shopping

Work related

0

4

4

5

2

0

1

Entertainment

Information

Education

News

Travel

Shopping

Work related

Web usage success rate

0

5

0

0

Extremely Satisfied

Good

Okay

Mostly okay

2

1

1

1

Extremely Satisfied

Good

Okay

Mostly okay

Modem speed

2

1

1

1

56 Kbytes

Fast (gov. speed)

T1

Not known

3

0

1

1

28.8 Kbytes

Fast (gov. speed)

T1

Not known

Table 1: Subject Demographics

 

Finally, we analyzed and formatted the data. The data was sorted into three categories: the answers to the sets of task questions, the responses to the subjective questions about the sites, and the path information from the log files that was generated by WebVIP. Subjects' answers to the sets of task questions supported some measure of performance. A correct response was credited as success for the subject having performed the correct navigational path in locating the answer page. However, it was the navigation logs that allowed us to assess a truer measure of success. For example, the log file revealed the number of search choices that were made by subjects before locating an answer page as well as confirming whether an answer page was even located. The log files were also helpful in assessing any navigational techniques that were used by subjects that shorten their search time as they became more familiar with the structural layout of the sites. And, the responses to the subjective questions were useful in assessing the subjects' impressions and preferences about the features that were used in designing each site. In the following sections, we further describe the experimental sites, their differences, and similarities. In Task Performance Data, we describe the marker test, and detail the tasks for each site. We also summarize the results in a table format. We describe time and preference data and then discuss some future efforts.

Experimental Sites

The Middle Eastern site was designed to maximize cultural markers uncovered in the research by Barber and Badre. These cultural markers included:

While the graphics in this site were minimized, the normal text headers were replaced by ornamental text. Supporting informational text was presented in horizontal chunks instead of the columnar format used in the original site. Conversely, the columnar format was retained for the United States site. Additionally, the United States site contained menus of links positioned on the left side of the page; whereas, in the Middle Eastern site, menus were positioned on the right side of the page. In addition, green backgrounds and text colors were added to the Middle Eastern site. Figures 1 and 2 are screen shots of the home pages for the Middle Eastern and United States sites, respectively.

 

Figure 1. Home Page for the Middle Eastern Site

 

Figure 2. Home Page for the United States Site

 

Task Performance Data

The tasks for each site were designed to test usage of the various cultural markers. There were two different task versions for a given marker test, that is, one for each site. In each of the five tasks that follows, we hypothesized that the North American subjects would perform better on the United States site and the Middle Eastern subjects would perform better on the Middle Eastern site. The shaded boxes in each Results Table are used to call attention to where we expected to see better results.

Task 1 Marker Test: Selection of a graphical versus textual link.

Version for Middle Eastern site: Is there a rebate for the 1998 Nissan and if so, what is it?

Version for United States site: What are the lease payments for an Acura 3.2 if a $999 down payment is made?

Results of Task 1

 

North American

Middle Eastern

Middle Eastern Site

4/5 correct

5/5 correct

Text link

Graphic link

3/5 used

2/5 used

4/5 used

1/5 used

United States Site

5/5 correct

4/5 correct

Text link

Graphic link

5/5 used

 

3/5 used

2/5 used

 

Discussion

With respect to accuracy of answers, the results of Task 1 were as expected. The Middle Eastern subjects did better in supplying the correct answer on the Middle Eastern site and the North American subjects did better on the United States site. In the case of the incorrect response by the one North American subject on the Middle Eastern site, there was data evidence that showed more uncertainty than a navigation problem in locating the answer page. On the subject's answer sheet, the correct answer was crossed through, and an incorrect answer was entered. In the case of the incorrect response by the one Middle Eastern subject on the United States site, the subject stated on the answer sheet that they could not locate the answer. However, textual links were chosen over graphic links by both groups regardless of the site. We expected this for the Middle Eastern subjects but not for the North American subjects. We suspect that this may have been influenced by the network load. Preference for text links were determined by a subject's first choice in initiating their search for answers.

Task 2 Marker test: Use of menu columns positioned on right (in the Middle Eastern site) or left (in the United States site).

Version for Middle Eastern site: What were the results of the crash tests for the Ford F-150 pickup?

Version for United States site: What is the exterior length of the 1998 Audi A6 sedan?

Results of Task 2

 

North American

Middle Eastern

Middle Eastern Site

5/5 correct

5/5 correct

Right menu col.

1/5 used

1/5 used

United States Site

5/5 correct

3/5 correct

Left menu col.

3/5 used

1/4 used

 

Discussion

The results of Task 2 were as expected. Again, the Middle Eastern subjects were best able to supply correct answers on the Middle Eastern site and the North American subjects performed equally well on both sites in that all subjects responded correctly to the tasks for the respective sites. In the case of the incorrect responses for the Middle Eastern subjects on the United States site, one subject stated on their answer sheet that they could not locate the answer while the other showed confusion about what the answer should be since they located the answer page but responded with the incorrect phrase of "mid-size" instead of the correct numerical length of 192 inches. However, the marker test showed that most subjects did not use the column menus as their first preference when initiating their search (regardless of the column menus' right or left positioning). In many instances, the back arrow key was the first preference (even though the column menu was readily available on the page that they were currently on) and then, a choice of using the column menus was made.

Task 3 Marker test: Use of menu columns positioned on left or right.

Version for Middle Eastern site: Which econo sports cars were tested and which was determined to be the winner?

Version for United States site: In the long term road test in June, 1998 of the VW what problems were encountered?

Results of Task 3

 

North American

Middle Eastern

Middle Eastern Site

2/5 correct

3/5 correct

Right menu col.

2/5 used

None

United States Site

5/5 correct

5/5 correct

Left menu col.

1/5 used

None

 

Discussion

The results of Task 3 were not as expected for the Middle Eastern subjects. Their best performance in giving correct answers was on the United States site instead of the Middle Eastern site. In the case of incorrect responses by the two Middle Eastern subjects on the Middle Eastern site, one subject located the answer page but indicated the wrong econo sports car as the winner; while, the other indicated that they could not find the answer. The results were as expected for the North American subjects. They gave the most correct answers on the United States site. In the case of incorrect responses by the North American subjects on the Middle Eastern site, one subject showed more of a lack of patience to complete the answers to the task than in being able to locate the answer. They located the answer page and identified the participating cars in the road test, but they did not proceed to identify the winner. The second North American subject who responded incorrectly appeared to have not read thoroughly the task. They did not have any problem locating the answer. It was the incompleteness in their answer. They only answered the last part of the task which was identifying the winner. They did not answer the first part of the task which was to identify the other participating car. The third North American subject who responded incorrectly indicated that they could not locate the answer. With regard to the marker test, it was revealed in the path data that the subjects did not use the column menus as their first preference to initiate searching.

Task 4 Marker test: Text link versus menu columns.

Version for Middle Eastern site: When you're buying a new car, what is a telephone number to call to get a competitive insurance quote?

Version for United States site: What is a reasonable amount to figure in for dealer profit when you're buying a new car?

Results of Task 4

 

North American

Middle Eastern

Middle Eastern Site

4/5 correct

5/5 correct

Text link

Right menu col.

2/5 used

3/5 used

1/5 used

4/5 used

United States Site

5/5 correct

5/5 correct

Text link

Left menu col.

3/5 used

2/5 used

1/5 used

4/5 used

 

Discussion

The results of Task 4 were as expected. The Middle Eastern subjects responded correctly in each case on the Middle Eastern site. However, they performed just as well on the United States site. The North American subjects performed best on the United States site where each subject responded correctly. In the case of the one incorrect response on the Middle Eastern site, the North American subject indicated that they could not locate the answer. As for the marker test, we had to work around design contrasts allowing at most, one click of the back arrow key before determining the use of the column menus. This allowance revealed a significant increase in identified use of the column menus (regardless of right or left positioning), especially among the Middle Eastern subjects. Overall, the use of menu columns was preferred over text links.

Task 5 Marker test: Use of menu columns from home page.

Version for Middle Eastern site: Is GEICO one of Edmund's partners?

Version for United States site: Where is the Edmund's job for associate news editor based?

Results of Task 5

 

North American

Middle Eastern

Middle Eastern Site

4/5 correct

3/5 correct

Right menu col.

3/5 used

1/5 used

United States Site

4/5 correct

3/5 correct

Left menu col.

4/5 used

3/5 used

 

Discussion

The results of Task 5 were not as expected. The Middle Eastern subjects performed equally well on both sites. In the case of the two incorrect responses on the Middle Eastern site, one Middle Eastern subject stated that they could not locate the answer page while the other subject's path data indicated that they could not locate the answer page either. Additionally, only one of the three Middle Eastern subjects who answered correctly on the Middle Eastern site actually located the answer page in accordance with their path data. It is suspected that the other two answered correctly as a guest since the answer to the task was a logical yes or no. In the case of incorrect answers on the United States site, one Middle Eastern subject stated that they could not locate the answer while the other located the answer page but showed confusion with the recorded answer. The performance of the North American subjects was unexpected. Like the Middle Eastern subjects, they performed equally well on both sites. The incorrect answers for the North American subjects on both sites was made by the same North American subject, and in each case, the subject could not locate the answer page. On the United States site, use of menu columns continued to be the preferred marker choice to initiate navigation.

Based on the average correct responses, the performances of the subjects from the respective cultures proved to be in agreement with our hypothesis that the North American subjects would perform better on the United States site and the Middle Eastern subjects would perform better on the Middle Eastern site. On the United States site, the North American subjects responded correctly 96% of the time versus on the Middle Eastern site, they responded correctly only 76% of the time. In the case of the Middle Eastern subjects, they gave a correct answer 80% of the time on the United States site versus 85% of the time on the Middle Eastern site. Thus, we see that the North American subjects had a wider dispersion of performance than did the Middle Eastern subjects. This wider dispersion of performance by the North American subjects could be attributed to the fact that they had the least experience in using the web. One North American subject stated that our experiment was their first experience in using the web.

Time Data

We did not collect timing data per task and would like to de-emphasize time due to the variety of line connect speeds that were used by subjects and the possible slowness in load time for the Middle Eastern site as a result of design constraints. Moreover, since the study was conducted remotely we could not guarantee that the subjects had no interruptions during the experiment. However, the time required by the subjects to complete all the tasks is presented in the table below for discussion.

Average Completion Times

 

North American

Middle Eastern

Middle Eastern Site

12 minutes

17 minutes

United States Site

12 minutes

21 minutes

 

Interestingly, the average completion times for the North American subjects were approximately the same for both sites. This might indicate that the ease with which North American subjects were able to locate answers was not affected by the sites' design; whereas, the longer time period spent on the United States site by the Middle Eastern subjects might indicate that they were more comfortable on the Middle Eastern site which supports our hypothesis.

Preference Data

We had also hypothesized that the Middle Eastern subjects would prefer the Middle Eastern site (and rate it higher), while the North American subjects would prefer the United States site. Table 2 shows the average ratings on a scale from 1 to 5 with 1 being the lowest rating and 5 being the highest rating. These ratings were based on the subjects' first impressions. The shaded cells indicate agreement with our hypothesis. Based on a t distribution of 8 degrees of freedom, we did not achieve sufficient levels of certainty that warrant stating them here; however, the data does support that there is more agreement in the case of the North American subjects.

After giving their first impressions of the design features used in each site, the subjects were asked to compare the features of one site against the other and to express a preference. Again, we used a scale of 1 to 5 with 1 meaning that a specific design feature did not make a difference in preferring one site over the other, and 5 meaning that a specific design feature made a great difference in preferring one site over the other. This direct request of asking the subjects to compare one site against the other caused our levels of certainty to go up significantly among the North American subjects. This was especially true for preferences specified for appearance of the United States site over the Middle Eastern site. In this case, the North American subjects rated a certainty level of plus 90%. The results are shown in table 3. The "Likes" columns list the number of subjects who preferred a specific feature of one site over the other for a given group and the "Rate" columns list the average degree to which they liked that feature for that site over the other. The shaded areas emphasize agreement with our hypothesis.

 

 

Category

North American

Middle Eastern

Middle Eastern Site

Unite States Site

Middle Eastern Site

United States Site

Appearance:

Overall.......................

Colors used................

Prefer other colors.....

 

3.0

 

3.8

 

2.6

 

3.8

3.0

4.0

2.8

4.0

2.2

1.6

2.6

2.2

Pictures:

Ones used...................

Number used.............

 

2.6

 

3.6

 

2.8

 

2.8

4.8

4.2

3.4

3.6

Text:

Amount used.............

Font or script used.....

Color of headings.......

 

2.8

 

3.4

 

2.4

 

3.0

3.6

4.2

2.8

2.6

3.0

3.6

2.2

3.6

Organization of Information:

Layout........................

 

3.6

 

3.8

 

2.6

 

3.2

Ease of use:

In performing the task

 

3.4

 

3.8

 

3.6

 

3.4

Navigation:

Through the site.........

 

3.4

 

3.8

 

3.6

 

3.8

Table 2: Average site Ratings of Subjects' First Impressions

(1 = lowest, 5 = highest)

 

 

Category

North American

Middle Eastern

Middle Eastern Site

United States Site

Middle Eastern Site

United States Site

Likes

Rate

Likes

Rate

Likes

Rate

Likes

Rate

Appearance:

Over all.....................

Colors used...............

Site preference..........

 

1

 

4.00

 

4

 

3.50

 

1

 

4.00

 

4

 

3.00

1

1.00

4

3.25

1

5.00

4

3.00

1

1.00

4

2.25

1

1.00

4

2.75

Graphics:

Site preference..........

 

1

 

1.00

 

4

 

3.75

 

4

 

1.25

 

1

 

4.00

Text:

Amount used............

Font or script used...

Color of heading used

 

2

 

2.50

 

3

 

3.00

 

3

 

1.00

 

2

 

3.50

3

1.67

2

3.50

3

1.33

2

3.00

1

1.00

4

3.00

3

2.00

2

3.50

Information:

Organization of.........

 

1

 

3.00

 

4

 

3.50

 

2

 

3.00

 

3

 

3.33

Ease of Use:

Performing tasks.......

 

1

 

4.00

 

4

 

3.25

 

2

 

3.50

 

3

 

4.00

Navigation:

Site preference...........

 

1

 

3.00

 

4

 

3.00

 

1

 

5.00

 

4

 

4.00

Overall Preferences:

Site preference...........

Site appearance..........

 

1

 

2.00

 

4

 

3.50

 

1

 

5.00

 

4

 

3.25

1

1.00

4

3.50

2

4.00

3

3.00

Table 3: Comparison Ratings of Sites' Features

(1 = lowest, 5 = highest)

 

Discussion and Future Work

There were several factors that confounded our experiment. First, we chose to conduct the experiment remotely. The decision was based on the fact that it would be easier to obtain participants since they did would not have to travel to our usability lab. However, this decision became counter productive in designing the Middle Eastern site. In order for us to maintain a consistent character look across platforms for the Middle Eastern site, it became necessary to implement ornamental text as graphic images that made the site considerably slower to download versus the United States site. This may well have influenced subjective ratings of the Middle Eastern site.

Secondly, since the Middle Eastern subjects have lived in the United States on an average of four or more years, we do not have any way of measuring the degree to which they have been influenced by the North American culture. Such influence could possibly have nullified the design attempts to make the site reflective of the Middle Eastern culture. Additionally, the language that was used in the design of our Middle Eastern site was English. This may have biased subjects as well. However, we feel that there was enough evidence of effect that we intend to conduct more experiments, correcting for the confounds in this one. Recall that we did find some performance benefits for the Middle Eastern subjects when using the Middle Eastern site. Subjective preference data showed that both groups heavily favored the North American site. The design of the site did have some influence on subject's use and preferences. Future experiments need to eliminate confounds in this experiment and produce other metrics that will allow more detailed investigation of any effects.

In future experiments, we will refine our approach with more control. Future experiments will be run from a CD to minimize any graphical loading problems over the internet. This will allow the use of more graphical culture markers such as flags, monuments, and area scenes. We will then be able to collect data on time for individual tasks rather than relying on correct responses alone. Another possibility might be to have our subjects start from the same web page for each search, thus allowing us to compare the individual paths that users take to locate information. We will conduct the next set of experiments in several other countries, insuring the existence of a minimum amount of influence from the North American culture. In these future experiments, we would like to have three way comparisons as well – that is, a North American design, a design native to that culture, and another design for another culture. Our goal is to provide design guidelines, and possibly, automated construction of web sites that are more modified for various cultures. Users could choose the version of the site they wanted or it could be automatically displayed upon detecting the access address of the user.

Acknowledgments

We thank Paul Hsiao, Wendy Barber, and Al Badre for their help in the design of the web sites and experiment.

References

  1. Barber, W. and Badre, A., Culturability: The Merging of Culture and Usability, presented at HFWeb '98 (June 5, 1998), http://www.research.att.com/conf/hfweb/.
  2. Scholtz, J., Laskowski, S. and Downey, L., Developing Usability Tools and Techniques for Designing and Testing Web Sites , presented at HFWeb '98 (June 5, 1998), http://www.research.att.com/conf/hfweb/.
  3. Scholtz, J., WebMetrics: A Methodology for Producing Usable Web Sites, Proceedings of the Human Factors and Ergonomics Society, 42nd Annual Meeting, Chicago, October 5-9, 1998, Vol. E, page 1612.
  4. Spool, J. M., Scanlon, T., Schroeder, W., Snyder, C., and DeAngelo, T., Web Site Usability: A Designer’s Guide, User Interface Engineering, 1997.

Footnotes

1. Barber and Badre identified some culture markers that may be particular to a given region, especially when the region shares similarities in language. North America is the cultural region into which the United States is classified; therefore, the choice of a United States site is representative of the North American culture.


"The Effects of Cultural Markers on Web Site Use"
<- Back

Thanks to our conference sponsors:
A T and T Laboratories
ORACLE Corporation  
National Association of Securities Dealers, Inc.

Thanks to our conference event sponsor:

Bell Atlantic


Site Created: Dec. 12, 1998
Last Updated: June 10, 1999
Contact hfweb@nist.gov with corrections.