SE365 · LECTURE 12 · SLIDE BREAKDOWN

Determine. Explore. Choose.
Identify. Decide. Evaluate.

DECIDE is a checklist for planning an evaluation study. This lecture then separates usability testing - applied experimentation to improve a product - from experiments for research, which test hypotheses to discover new knowledge, and closes with field studies.

DECIDE frameworkUsability testingTesting vs researchExperimental designIndependent & dependent variablesField studies

LEARNING OUTCOMES

Explain the DECIDE framework and apply it to plan an evaluation.
Explain how to do usability testing and what data it produces.
Outline the basics of experimental design.
Distinguish independent from dependent variables.
Describe how to do field studies and why they matter.

01 - THE DECIDE FRAMEWORK

Six steps for planning an evaluation study

A framework to guide evaluation - and a useful checklist for planning any study.

D - Determine the goalsWhat are the high-level goals of the evaluation? Who wants it and why? The goals influence the methods used.
E - Explore the questionsQuestions guide the evaluation, and high-level goals break down into sub-questions.
C - Choose the evaluation methodsThe method influences how data is collected, analyzed and presented.
I - Identify the practical issuesHow to select users, find evaluators, select equipment, stay on budget and stay on schedule.
D - Decide how to deal with the ethical issuesDevelop an informed consent form and honour participants' rights.
E - Evaluate, analyze, interpret and present the dataConsidering reliability, validity, biases, scope and ecological validity.

1. Determine the goals - what they might be

Identify the best metaphor for the design; check that user requirements are met; check for consistency; investigate how technology affects working practices; improve the usability of an existing product.

2. Explore the questions - worked example

The goal of finding out why some customers prefer paper airline tickets to e-tickets breaks into sub-questions: What are customers' attitudes to e-tickets? Are they concerned about security? Is the interface for obtaining them poor?

3. Choose the methods - what follows

Field studies typically involve observation and interviews; involve users in natural settings; do not involve controlled tests; and produce qualitative data.

5. Ethical issues - participants' rights

To know the goals of the study; to know what will happen to the findings; to privacy of personal information; to leave when they wish; and to be treated politely.

MEMORY HOOK

The acronym is the answer sheet: Determine goals, Explore questions, Choose methods, Identify practical issues, Decide on ethics, Evaluate and present. Note the shape - goals narrow into questions, questions select methods, and everything else is execution.

THE ORDER MATTERS

Goals come before questions, and questions come before methods. Choosing the method first - "let's run a usability test" - is the most common planning error, and DECIDE exists to prevent it.

02 - USABILITY TESTING

Recording performance of typical users doing typical tasks

Usability testing involves recording the performance of typical users doing typical tasks in controlled settings. Users are observed and timed.

Data is recorded on video and key presses are logged.
The data is used to calculate performance times, and to identify and explain errors.
User satisfaction is evaluated using questionnaires and interviews.
Field observations may be used to provide contextual understanding.
Goals and questions focus on how well users perform tasks with the product.
Comparison of products or prototypes is common.
The focus is on time to complete a task and the number and type of errors.
Testing is central; questionnaires and interviews provide data about users' opinions.

Testing conditions	Detail
Setting	A usability lab or other controlled space.
Emphasis	Selecting representative users and developing representative tasks.
Participants	5-10 users typically selected.
Task length	Usually no more than 30 minutes.
Consistency	The test conditions should be the same for every participant.
Ethics	An informed consent form explains procedures and deals with ethical issues.

Types of data collected

Time to complete a task. Time to complete a task after a specified time away from the product. Number and type of errors per task. Number of errors per unit of time. Number of navigations to online help or manuals. Number of users making a particular error. Number of users completing the task successfully.

How many participants?

The number is a practical issue, depending on the schedule for testing, the availability of participants and the cost of running tests. Typically 5-10. Some experts argue testing should continue until no new insights are gained.

Equipment and remote testing

Usability labs have observers watching a user and an assistant. Portable equipment is used in the field, including the Tobii Glasses mobile eye-tracking system. Remote usability testing is also supported. Affordable remote testing systems are more portable than usability labs, and many contain mobile eye-tracking and other devices.

MEMORY HOOK

Usability testing measures exactly two families of thing: how long and how wrong. Every data type in the list is a variation on time or errors - plus help lookups, which are a proxy for both.

03 - TESTING VS RESEARCH

Applied experimentation, not science

Experiments test hypotheses to discover new knowledge by investigating the relationship between two or more variables. Usability testing is applied experimentation.

Usability testing	Experiments for research
Improve products.	Discover knowledge.
Few participants.	Many participants.
Results inform design.	Results validated statistically.
Usually not completely replicable.	Must be replicable.
Conditions controlled as much as possible.	Strongly controlled conditions.
Procedure planned.	Experimental design.
Results reported to developers.	Scientific report to the scientific community.

In usability testing, developers check that the system is usable by the intended user population for their tasks. Experiments may also be done within usability testing - the two are not mutually exclusive, they differ in purpose, rigour and audience.

THE SEVEN-ROW TABLE IS THE EXAM QUESTION

This comparison appears almost verbatim on exams. Learn it as seven contrasts - purpose, numbers, validation, replicability, control, planning, audience - rather than as fourteen separate facts.

04 - EXPERIMENTS AND EXPERIMENTAL DESIGN

Variables, and the three participant designs

An experiment tests a hypothesis by predicting the relationship between two or more variables.

Variable	Definition
Independent variable	The one manipulated by the researcher.
Dependent variable	The one that depends on the independent variable.

Typical experimental designs have one or two independent variables.
Results are validated statistically and the study is replicable.

Different participants (between subjects)

A single group of participants is allocated randomly to the experimental conditions - each person sees only one condition.

Same participants (within subjects)

All participants appear in both conditions.

Matched participants (pairwise)

Participants are matched in pairs, for example on expertise or gender, with one of each pair in each condition.

MEMORY HOOK

Independent is what I change; Dependent is what Depends on it. And the three designs: different people, same people, matched people - the whole distinction is who appears in which condition.

05 - FIELD STUDIES

In the wild

Field studies are done in natural settings. In the wild is the term for prototypes being used freely in natural settings.

They aim to understand what users do naturally and how technology impacts them. In product design they are used to:

Identify opportunities for new technology.
Determine design requirements.
Decide how best to introduce new technology.
Evaluate technology in use.

THE UNEXPECTED FINDING

Sometimes the findings of a field study are unexpected, especially for in-the-wild studies exploring how novel technologies are used by participants in their own homes, places of work, or outside. That capacity for surprise is precisely what a controlled lab study cannot deliver, and it is why the two complement each other.

MEMORY HOOK

Field studies do four jobs across the whole lifecycle: find the opportunity, set the requirements, plan the rollout, evaluate the result. They are not only an evaluation method - they bracket the entire process.

MISTAKES STUDENTS USUALLY MAKE

Mistakes students usually make

Each claim below is the wrong answer; the line beneath it is the correction, in the wording this course marks against.

Getting the DECIDE letters out of order or wrong.

Determine the goals, Explore the questions, Choose the evaluation methods, Identify the practical issues, Decide how to deal with ethical issues, Evaluate, analyze, interpret and present the data.

Choosing a method before setting goals.

The goals influence the methods used, and questions guide the evaluation. Method-first planning is the error DECIDE exists to prevent.

Swapping independent and dependent variables.

The independent variable is manipulated by the researcher; the dependent variable depends on it. Typical designs have one or two independent variables.

Confusing between-subjects and within-subjects designs.

Different participants (between subjects): a single group allocated randomly to conditions. Same participants (within subjects): all participants appear in both conditions. Matched (pairwise): participants matched in pairs on expertise, gender and so on.

Treating usability testing as scientific research.

Usability testing improves products, uses few participants, informs design, is usually not completely replicable, controls conditions as far as possible, follows a planned procedure and reports to developers. Research experiments discover knowledge, use many participants, are statistically validated and replicable, are strongly controlled, use experimental design and produce a scientific report.

Giving the wrong participant count.

Typically 5-10 users, with tasks usually lasting no more than 30 minutes. The number is a practical issue driven by schedule, availability and cost; some experts say continue until no new insights are gained.

"Field studies are just uncontrolled usability tests."

They aim to understand what users do naturally and how technology impacts them, and are used to identify opportunities, determine requirements, decide how to introduce technology and evaluate it in use.

CHEAT SHEET

Shortest correct answers

The night-before table: every term in this lecture with the smallest answer that still earns the mark.

Concept	Shortest correct answer
DECIDE	Determine goals, Explore questions, Choose methods, Identify practical issues, Decide on ethics, Evaluate and present.
Participants' rights	Know the goals, know what happens to the findings, privacy of personal information, leave when they wish, be treated politely.
Usability testing	Recording performance of typical users doing typical tasks in controlled settings; users observed and timed.
Data recorded	Video and key-press logs, used to calculate performance times and identify and explain errors.
Testing focus	Time to complete a task, and the number and type of errors.
Participants	Typically 5-10; tasks usually no more than 30 minutes; conditions the same for everyone.
How many is enough	A practical issue of schedule, availability and cost; some experts say continue until no new insights are gained.
Experiment	Tests hypotheses to discover new knowledge by investigating the relationship between two or more variables.
Independent variable	The variable manipulated by the researcher.
Dependent variable	The variable that depends on the independent variable.
Between subjects	Different participants; a single group allocated randomly to the conditions.
Within subjects	Same participants; all appear in both conditions.
Matched pairs	Participants matched in pairs, e.g. on expertise or gender.
Field studies	Done in natural settings to understand what users do naturally and how technology impacts them.
In the wild	Prototypes being used freely in natural settings.
Field study uses	Identify opportunities, determine requirements, decide how to introduce technology, evaluate technology in use.

APPLY IT

Exam-style application

Write your own answer first, then open the model answer. These are the longer-form questions this material generates.

Apply the full DECIDE framework to plan an evaluation of a new hospital medication-ordering screen.

D - Determine the goals: improve the usability of the ordering screen and check that safety requirements are met; the goals come from the clinical safety lead, and they push the study towards performance measurement rather than opinion. E - Explore the questions: can a resident complete a first order unaided? Where do dosing errors occur? Does the confirmation step actually get read? C - Choose the methods: usability testing in a controlled setting for the performance measures, complemented by a short field observation for contextual understanding, since field studies do not involve controlled tests and produce qualitative data. I - Identify practical issues: recruit representative users across residents and consultants, find evaluators, book a quiet room with video and logging, and keep tasks under 30 minutes to fit clinical schedules. D - Decide on ethics: an informed consent form covering the goals, what happens to the findings, privacy of personal information, the right to leave at any time and polite treatment; approval from the appropriate review board. E - Evaluate, analyze, interpret and present: report completion rates, times and error types with explicit statements about reliability, validity, ecological validity, biases and scope - five to ten participants cannot support a population-level safety claim.

A team runs the same twelve participants through two search interfaces, always A then B. Identify the design, the risk, and two fixes.

The design is same participants (within subjects) - all participants appear in both conditions. The risk is that the order itself becomes an uncontrolled variable: by the time participants reach B they know the task, the vocabulary and the dataset, so B benefits from practice regardless of its design, and the measured difference in completion time confounds interface with learning. Fix 1: counterbalance the order, with half the participants doing B then A, which keeps the statistical power of a within-subjects design while cancelling the practice effect. Fix 2: switch to different participants (between subjects), allocating a single group randomly to the two conditions so nobody sees both - at the cost of needing more participants to detect the same difference. A third option is matched pairs, pairing participants on expertise so the groups are comparable without anyone repeating the task.

Explain why a field study can produce findings a usability test never would, and when you would still choose the lab.

Field studies are done in natural settings, and in-the-wild studies let prototypes be used freely; they aim to understand what users do naturally and how technology impacts them. Because there is little or no control, users bring their own goals, interruptions, environments and workarounds, and the deck notes that findings are sometimes unexpected, particularly when novel technologies are used in participants' own homes, workplaces or outdoors. They also serve the whole lifecycle: identifying opportunities for new technology, determining design requirements, deciding how best to introduce it, and evaluating it in use. The lab is still the right choice when the question is comparative and performance-based - which of two designs is faster or produces fewer errors - because usability testing holds the conditions the same for every participant, records video and key presses, and yields comparable times and error counts that a natural setting cannot. The two complement each other rather than compete.

POP QUIZ

Check yourself

6 questions. Every option is explained after submitting, including why the wrong ones are wrong.