SE365 · Annotated Slides · Lecture 12 — DECIDE, Usability Testing and Experiments

Exam weight Low

Overview

DECIDE framework
Usability testing
Basic experimental design
Field studies Based on material from: Sharp, H., Preece, J. and Rogers, Y. (2019). Interaction Design Beyond Human-Computer Interaction. 5th Ed. Sussex: John Wiley & Sons Ltd.

Exam weight Low

Learning Outcomes

Explain the DECIDE framework
Explain how to do usability testing
Outline the basics of experimental design
Describe how to do field studies

Exam weight Low

Slide 4: 1) DECIDE: a framework to guide evaluation

Exam weight High

DECIDE: Determine the goals, Explore the questions, Choose the approach and methods, Identify practical issues, Decide about ethical issues, Evaluate, interpret and present the data.

TipDECIDE: Determine the goals, Explore the questions, Choose the approach and methods, Identify practical issues, Decide about ethical issues, Evaluate, interpret and present the data.

🎯 How this can come up · 1 question

MCQ

In DECIDE, the "I" stands for:

Exam weight Low

Exam weight Low

Slide 7: 1.3) Choose the evaluation approach methods

Exam weight Low

Exam weight Low

Slide 9: 1.5) Decide about ethical issues

Exam weight Low

Slide 10: 1.6) Evaluate, interpret & present data

Exam weight Low

Exam weight Low

2. Usability testing

Involves recording performance of typical users doing typical tasks
Controlled settings
Users are observed and timed
Data is recorded on video & key presses are logged
The data is used to calculate performance times, and to identify & explain errors
User satisfaction is evaluated using questionnaires & interviews
Field observations may be used to provide contextual understanding

Exam weight Low

Usability testing: measure users’ performance on typical tasks in controlled conditions (time, errors, satisfaction, think-aloud).

2. Usability testing (cont)

Goals & questions focus on how well users perform tasks with the product
Comparison of products or prototypes common
Focus is on time to complete task & number & type of errors
Data collected by video & interaction logging.
Testing is central
User satisfaction questionnaires & interviews provide data about users’ opinions

Exam weight Low

2.1) Experiment & Usability testing

Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more things – i.e., variables
Usability testing is applied experimentation
Developers check that the system is usable by the intended user population for their tasks
Experiments may also be done in usability testing

Exam weight Low

Slide 15: 2.2) Usability testing & Research

Exam weight Low

Slide 16: 2.3) Usability lab with observers watching a user & assistant

Exam weight Low

Slide 17: 2.4) Portable equipment for use in the field

Exam weight Low

Slide 18: Tobii Glasses Mobile Eye- Tracking System

Exam weight Low

Eye tracking shows where users look (heat maps, gaze plots).

Exam weight Low

Remote usability testing: users test from their own location; recorded online.

2.5) Testing conditions

Usability lab or other controlled space
Emphasis on:
- selecting representative users
- developing representative tasks
5-10 users typically selected
Tasks usually last no more than 30 minutes
The test conditions should be the same for every participant
Informed consent form explains procedures and deals with ethical issues

Exam weight Low

2.6) Some type of data

Time to complete a task
Time to complete a task after a specified. time away from the product
Number and type of errors per task
Number of errors per unit of time
Number of navigations to online help or manuals
Number of users making a particular error
Number of users completing task successfully

Exam weight Low

2.7) How many participants is enough for user testing?

The number is a practical issue
Depends on:
- schedule for testing
- availability of participants
- cost of running tests
Typically 5-10 participants
Some experts argue that testing should continue until no new insights are gained

Exam weight High

About 5–12 participants are often enough for a usability test (Nielsen: 5 users find most problems).

🎯 How this can come up · 1 question

MCQ

According to Nielsen, about how many users find most usability problems in a test?

3. Experiments

Test hypothesis
Predict the relationship between two or more variables
Independent variable is manipulated by the researcher
Dependent variable depends on the independent variable
Typical experimental designs have one or two independent variable
Validated statistically & replicable

Exam weight High

Experiments test a hypothesis: independent variable (what you change) → dependent variable (what you measure).

🎯 How this can come up · 1 question

MCQ

In an experiment comparing two menu designs on task time, the menu design is the:

3.1 Experimental design

Different participants (between subjects) - single group of participants is allocated randomly to the experimental conditions
Same participants (within subjects) - all participants appear in both conditions
Matched participants (pairwise) - participants are matched in pairs, e.g., based on expertise, gender, etc

Exam weight Low

3.2 Different, same, matched participant design

Exam weight High

Designs: different participants (between-subjects: no order effects, needs more people), same participants (within-subjects: fewer people, order effects → counterbalance), matched participants.

Exam trapWithin-subjects (same participants) designs suffer from ORDER effects; fix with counterbalancing.

🎯 How this can come up · 1 question

MCQ

Each participant tries both designs. This is a:

4. Field Studies

Field studies are done in natural settings.
“in the wild” is a term for prototypes being used freely in natural settings
Aim to understand what users do naturally and how technology impacts them
Field studies are used in product design to:
- identify opportunities for new technology
- determine design requirements
- decide how best to introduce new technology
- evaluate technology in use

Exam weight Low

Field studies: observe real use in natural settings; less control, more realism.

Key points

Usability testing takes place in controlled usability labs or temporary labs.
Usability testing focuses on performance measures, eg. how long and how many errors are made when completing a set of predefined tasks. Indirect observation (video and keystroke logging), user satisfaction questionnaires and interviews are also collected.
Affordable, remote testing systems are more portable than usability labs. Many also contain mobile eye-tracking and other devices.

Exam weight Low

Key points

Experiments aim to test hypotheses by manipulating certain variables while keeping others constant.
The experimenter controls the independent variable(s) but not the dependent variable(s).
Field studies are evaluation studies that are carried out in natural settings to discover how people interact with technology in the real world.
Field studies that involve the deployment of prototypes or technologies in natural settings may also be referred to as ‘in the wild’ studies.
Sometimes the findings of a field study are unexpected, especially for in the wild studies that explore how novel technologies are used by participants in their own homes, places of work, or outside.

Exam weight Low