UWI Usability Lab

UWI Usability Lab

Study Procedures

The usability Laboratory adopts a set number of procedures for every study. These have been based on the widely accepted usability practices and teachings of Jeffrey Rubin (see Rubin, J. (1994) The Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests, John Wiley & Sons, Inc.)

The procedures that the usability Laboratory follows are listed below.

6.1 Developing a Study Plan

The usability Laboratory ensures that a study plan is written before any other stage of the study goes ahead. It forms the basis for planning and communication between the usability Laboratory and its clients. It also ensures that the stages of the study are well organised and mapped out. Our study plans typically consist of the following parts:

a. Overview. This describes the type of study being performed and the main purpose and goals of the study.

b. Problem statements. This describes the hypothetical problems that need to be studied and analysed throughout the study. Depending on the purpose and goal of the study the issues to be resolved may be broad (focusing on all aspects of the product) or specific (targeting individual aspects of the product). The problem statements might include criteria for measuring usability and user acceptance.

c. Participant profile. This describes, as best we are able, the background and characteristics of the real-life users of the product. Sometimes the users’ features will be very specific; other times the features will be very general. The approach used depends on the product under investigation. Describing these features enables us to target the right types of people as participants.

d. Experimental design. The content of this section depends on the study. If the study is less formal, where a rigorous study is not required, then four to six participants are used, each following the same processes during the study. If the study is more formal and rigorous, then we use this section to identify the number of elements to be studied and the number of participants required to adequately test each element.

e. Process. This describes the general process that will be followed during the study and, where applicable, includes tasks that the users are expected to accomplish.

f. Test environment and equipment requirements. This describes the usability Laboratory space, and the video and audio configuration that is required for the study. We discuss room layouts, camera placements and recording configurations. As we have developed the usability Laboratory with flexibility in mind this has proved to be a relatively easy process in most instances.

g. Observer role. This describes the role that the observer/test monitors will play. Depending on the purpose and goals of the study, the observers may have a lot or little interaction with the participants. For exploratory and assessment type studies, we encourage the participants to communicate with the observers about what they are doing, their thoughts on the application and the processes that they are following. By doing this we tend to gather rich data that highlights the users’ thoughts and understandings--something that is not typically collected when using quantitative methods exclusively. For validation-type studies, we encourage the participants to work alone, as if they were in a real-life situation. As this type of study is typically performed towards the end of a development cycle we predominantly search for quantitative--performance (speed and accuracy) and opinion-typ--data.

h. Evaluation measures. This describes the types of data we will seek to collect. Depending on the study, the data may be quantitative and/or qualitative. Often we look for a combination of data.

6.2 Selecting Participants

Based on the participant profile identified within the study plan we then seek to locate relevant participants that represent real-life users of the product. If the participant profile identifies the users as those with general characteristics, then we generally find that the participants can be found from the student body at the Fatih University and surrounding areas. If the participant profile indicates the users should have specific characteristics then we look further a field for the participants. Sometimes, as the users are internal, they are provided by the client.

6.3 Preparing Study Materials

Depending on the type of study and the amount of supporting materials required, we then typically prepare the following study materials:

a. A participant workbook. This contains an overview, a participant’s bill of rights and consent form, an initial (pretest) questionnaire, application use scenarios, prerequisite training materials (if required) and a summary (posttest) questionnaire. Additional materials, such as tutorials and user manuals, can also be created and provided for the participants.

b. A researcher’s manual. This contains (i) a set-up checklist to ensure that all the equipment (electronic and manual) is in place before the study begins; (ii) additional scripting if it is not supplied in the participant workbook; and (iii) a debriefing topics guide.

6.4 Conducting the Study

We have extensive experience conducting studies. Our goal is to run a quality and organised study that produces excellent and usable results. To ensure this, we like to encourage our participants to relax and ‘enjoy’ their experiences even though they may become frustrated with the product. This is important (i) so that the participants act in a way that is natural and (ii) so that they feel relaxed enough to discuss their experiences without undue pressure. We break the sessions up into manageable time slots so that the participants do not become uncomfortable and unable to concentrate. We record each session onto both audio and video mediums for later observation.

6.5 Debriefing the Participants

Based on our observations during the study, we often debrief the participants about specific usability problems they encountered. We also like to ask more general questions e.g. ‘What was the best thing you liked about the application?’, ‘What did you most dislike about the application?’. We find that this usually opens up a barrage of comments from the participants about their experiences with the product. If required, we review each tape again and log observed events for analysis.

6.6 Analysing the Data

During analysis we review the data collected during the study. In most instances, we seek to:

a. Identify user errors and difficulties and identify the source of these errors.

b. Identify and focus on those tasks that did not meet the criterion identified in the problem statement of the study plan.

c. Prioritise the problems by criticality i.e. those that have the greatest impact on the usability of the product.

In some instances, we may also seek to analyse the difference between groups or users and product versions.

6.7 Producing a Report

Throughout the report we like to focus on recommendations that will have the widest impact on the product, provide both short term and long term recommendations, and indicate areas where further study is required. Although dependent on the product under investigation and the type of study performed, our reports generally include:

a. An executive summary. This includes the findings in summary and highlights the most critical problems that need to be solved.

b. An introduction. This includes the background to the study as well as any terminologies and typographic styles used in the report.

c. The study method. This includes problem statements, procedural notes, participant mix, observer roles and evaluation methods.

d. The findings. This is usually divided into global issues relating to the application and individual areas of the application. Global issues typically involve architectural elements such as structure and organisation, navigation and process of use. They might also include such things as the use of colour and information presentation. Individual areas of the application, outside the scope of the global architecture, are also highlighted. For these issues, we try to apply a usability principle to indicate where the product presents problems. For every issue identified, whether global or individual, we provide a design recommendation and include a graphic as an example solution.

e. A conclusion. This typically includes a summary of the design recommendations made throughout the findings section of the report.

UWI Usability Lab Study Procedures

Revised: 11 June 2008