Pluralistic walkthrough

The pluralistic walkthrough (also called a participatory design review, user-centered walkthrough, storyboarding, table-topping, or group walkthrough) is a usability inspection method used to identify usability issues in a piece of software or website in an effort to create a maximally usable human-computer interface. The method centers on recruiting a group of users, developers and usability professionals to step through a task scenario, discussing usability issues associated with dialog elements involved in the scenario steps. The group of experts used is asked to assume the role of typical users in the testing. The method is prized for its ability to be utilized at the earliest design stages, enabling the resolution of usability issues quickly and early in the design process. The method also allows for the detection of a greater number of usability problems to be found at one time due to the interaction of multiple types of participants (users, developers and usability professionals). This type of usability inspection method has the additional objective of increasing developers’ sensitivity to users’ concerns about the product design.

Procedure

Walk-Through Team

A walk-through team must be assembled prior to the pluralistic walk-through. Three types of participants are included in the walk-through:

representative users
- Users should be representative of the target audience, and are considered the primary participants in the usability evaluation.
human factors (usability) engineers/professionals
- Human factors professionals usually serve as the facilitators and are also there to provide feedback on the design as well as recommend design improvements.
- The role of the facilitator is to guide users through tasks and facilitate collaboration between users and developers.
product developers
- Product developers answer questions about design and suggest solutions to interface problems users have encountered.
- It is best to avoid having a product developer assume the role of facilitator, as they can get defensive to criticism of their product.

Materials

The following materials are needed to conduct a pluralistic walkthrough:

Room large enough to accommodate approximately 6-10 users, 6-10 developers and 2-3 usability engineers
Printed screen-shots (paper prototypes) put together in packets in the same order that the screens would be displayed when users were carrying out the specific tasks. This includes hard copy panels of screens, dialog boxes, menus, etc. presented in order.
Hard copy of the task scenario for each participant. There are several scenarios defined in this document complete with the data to be manipulated for the task. Each participant receives a package that enables him or her to write a response (i.e. the action to take on that panel) directly onto the page. The task descriptions for the participant are short direct statements.
Writing utensils for marking up screen shots and filling out documentation and questionnaires.

Participants are given written instructions and rules at the beginning of the walkthrough session. The rules indicate to all participants (users, designers, usability engineers) to:

Assume the role of the user
To write on the panels the actions they would take in pursuing the task at hand
To write any additional comments about the task
Not flip ahead to other panels until they are told to
To hold discussion on each panel until the facilitator decides to move on

Tasks

Pluralistic walkthroughs are group activities that require the following steps be followed:

Participants are presented with the instructions and the ground rules mentioned above. The task description and scenario package are also distributed.
Next, a product expert (usually a product developer) gives a brief overview of key product concepts and interface features. This overview serves the purpose of stimulating the participants to envision the ultimate final product (software or website), so that the participants would gain the same knowledge and expectations of the ultimate product that product end users are assumed to have.
The usability testing then begins. The scenarios are presented to the panel of participants and they are asked to write down the sequence of actions they would take in attempting to complete the specified task (i.e. moving from one screen to another). They do this individually without conferring amongst each other.
Once everyone has written down their actions independently, the participants discuss the actions that they suggested for that task. They also discuss potential usability problems. The order of communication is usually such that the representative users go first so that they are not influenced by the other panel members and are not deterred from speaking.
After the users have finished, the usability experts present their findings to the group. The developers often explain their rationale behind their design. It is imperative that the developers assume an attitude of welcoming comments that are intended to improve the usability of their product.
The walkthrough facilitator presents the correct answer if the discussion is off course and clarifies any unclear situations.
After each task, the participants are given a brief questionnaire regarding the usability of the interface they have just evaluated.
Then the panel moves on to the next task and round of screens. This process continues until all the scenarios have been evaluated.

Throughout this process, usability problems are identified and classified for future action. The presence of the various types of participants in the group allows for a potential synergy to develop that often leads to creative and collaborative solutions. This allows for a focus on user-centered perspective while also considering the engineering constraints of practical system design.

Characteristics of Pluralistic Walkthrough

Other types of usability assessment methods include: Cognitive Walkthroughs, Interviews, Focus Groups, Remote Testing and Think Aloud Protocol. Pluralistic Walkthroughs share some of the same characteristics with these other methods, especially cognitive walkthroughs, but there are some defining characteristics (Nielsen, 1994):

The main modification, with respect to usability walkthroughs, was to include three types of participants: representative users, product developers, and human factors (usability) professionals.
Hard-copy screens (panels) are presented in the same order in which they would appear online. A task scenario is defined, and participants confront the screens in a linear path, through a series of user interface panels, just as they would during the successful conduct of the specified task online, as the site/software is currently designed.
Participants are all asked to assume the role of the user for whatever user population is being tested. Thus, the developers and the usability professionals are supposed to try to put themselves in the place of the users when making written responses.
The participants write down the action they would take in pursuing the designated task online, before any further discussion is made. Participants are asked to write their responses in as much detail as possible down to the keystroke or other input action level. These written responses allow for some production of quantitative data on user actions that can be of value.
It is only after all participants have written the actions they would take that discussion would begin. The representative users offer their discussion first and discuss each scenario step. Only after the users have exhausted their suggestions do the usability experts and product developers offer their opinions.

Benefits and Limitations

Benefits

There are several benefits that make the pluralistic usability walkthrough a valuable tool.

Early systematic look at a new product, gaining early performance and satisfaction data from users about a product. Can provide early performance and satisfaction data before costly design strategies have been implemented.
Strong focus on user centered design in task analysis, leading to more problems identified at an earlier point in development. This reduces the iterative test-redesign cycle by utilizing immediate feedback and discussion of design problems and possible solutions while users are present.
Synergistic redesign because of the group process involving users, developers and usability engineers. The discussion of the identified problems in a multidisciplinary team will spawn creative, usable and quick solutions.
Valuable quantitative and qualitative data is generated through users’ actions documented by written responses.
Product developers at the session gain appreciation for common user problems, frustrations or concerns regarding the product design. Developers become more sensitive to users’ concerns.

Limitations

There are several limitations to the pluralistic usability walkthrough that affect its usage.

The walkthrough can only progress as quickly as the slowest person on each panel. The walkthrough is a group exercise and, therefore, in order to discuss a task/screen as a group, we must wait for all participants to have written down their responses to the scenario. The session can feel laborious if too slow.
A fairly large group of users, developers and usability experts has to be assembled at the same time. Scheduling could be a problem.
All the possible actions can't be simulated on hard copy. Only one viable path of interest is selected per scenario. This precludes participants from browsing and exploring, behaviors that often lead to additional learning about the user interface.
Product developers might not feel comfortable hearing criticism about their designs.
Only a limited number of scenarios (i.e. paths through the interface) can be explored due to time constraints.
Only a limited amount of recommendations can be discussed due to time constraints.