SE365 · LECTURE 13 · SLIDE BREAKDOWN

Methods that need
no users present.

Inspection methods use knowledge codified in heuristics, or models that predict users' performance. An expert role-plays the users the product was designed for and identifies potential usability problems using a set of guidelines.

InspectionsNielsen's revised heuristics3-stage processCognitive walkthroughPluralistic walkthroughA/B testing

LEARNING OUTCOMES

Describe the key concepts associated with inspection methods.
Explain how to carry out a heuristic evaluation.
Explain how to carry out cognitive and pluralistic walkthroughs.
Turn design guidelines and golden rules into heuristics.
Describe A/B testing and its ethical dilemmas.

01 - INSPECTIONS

Experts instead of users

Inspection methods typically involve an expert role-playing the users for whom the product is designed, and identifying any potential usability problems by using a set of guidelines. Users are not required to be present.

Experts use their knowledge of users and technology to review software usability.
Expert critiques can be formal or informal.
They can be used at any stage of a design, and can complement user testing.
Heuristic evaluation is a review guided by a set of heuristics.
Walkthroughs involve stepping through a pre-planned scenario, noting potential problems.

WHAT INSPECTIONS CAN BE APPLIED TO

Inspections can be used to evaluate requirements, mockups, functional prototypes, or systems. That breadth is the reason they can be used at any stage - there is nothing they need that only a finished product provides.

02 - HEURISTIC EVALUATION

Nielsen's method, and the revised heuristics

A usability inspection method developed by Jakob Nielsen in the early 1990s, derived empirically from an analysis of 249 usability problems. The heuristics have since been revised for current technology by Nielsen and others - for mobile devices, wearables, virtual worlds and more.

#	Revised heuristic (2014)	The one-line requirement
1	Visibility of system status	The system should always keep users informed about what is going on.
2	Match between system and real world	Follow real-world conventions, making information appear in natural and logical order.
3	User control and freedom	Support undo and redo.
4	Consistency and standards	Follow platform conventions.
5	Error prevention	Present users with a confirmation option before they commit to the action.
6	Recognition rather than recall	Minimize memory load by making objects, actions and options visible.
7	Flexibility and efficiency of use	Cater to both inexperienced and experienced users.
8	Aesthetic and minimalist design	Avoid information that is irrelevant.
9	Help users recognize, diagnose and recover from errors	Error messages should precisely indicate the problem.
10	Help and documentation	It may be necessary to provide help and documentation.

Briefing sessionTell the experts what to do.
Evaluation period of 1-2 hoursEach expert works separately. One pass to get a feel for the product; a second pass to focus on specific features.
Debriefing sessionExperts work together to prioritize the problems.

Budd (2007): heuristics for websites

A narrower set focused on key criteria: clarity; minimize unnecessary complexity and cognitive load; provide users with context; and promote a positive and pleasurable user experience.

Turning guidelines into heuristics (Granollers, 2018)

Ask questions such as: "Does the application include a visible title page, section or site? Does the user always know where they are located? Does the user always know what the system or application is doing? Are the links clearly defined? Can all actions be visualized directly (i.e., no other actions are required)?" Design guidelines and golden rules can be converted into heuristics this way.

Advantages	Problems
Few ethical and practical issues to consider, because users are not involved.	It can be difficult and expensive to find experts.
Can be used at any stage and complements user testing.	It requires knowledge and experience to apply heuristics.
The best experts have knowledge of the application domain and the users.	Trained usability experts are sometimes hard to find and can be expensive.
	You should use multiple experts and aggregate their results.

THE NUMBER-OF-EVALUATORS POINT

The deck's chart on evaluators versus problems found is the reason for the last row: one expert finds only a fraction of the problems, so multiple experts must be used and their results aggregated. That, plus the fact that user testing and heuristic evaluation may reveal different usability problems, is why inspections complement rather than replace testing.

MEMORY HOOK

The three stages are brief → alone → together. Experts work separately during evaluation so they do not anchor on each other, and together at debriefing so the problems get prioritised. Getting that alone/together split the wrong way round is the classic mistake.

03 - COGNITIVE WALKTHROUGHS

Focused on ease of learning

A cognitive walkthrough focuses on ease of learning. The designer presents an aspect of the design and usage scenarios; the expert is told the assumptions about the user population, the context of use and the task details.

The designer presents the design and scenariosAlong with the assumptions about users, context and tasks.
One or more experts walk through the design prototype with the scenarioStep by step, in role.
Experts are guided by 3 questionsAsked at every step.
Experts note problems as they work throughThe output is the list of noted problems.

The three questions

1. Will the correct action be sufficiently evident to the user? 2. Will the user notice that the correct action is available? 3. Will the user associate and interpret the response from the action correctly?

Pluralistic walkthrough

A variation on the cognitive walkthrough theme, performed by a carefully managed team. The panel of experts begins by working separately, then there is a managed discussion leading to agreed decisions. The approach lends itself well to participatory design.

MEMORY HOOK

The three questions map onto the gulfs: questions 1 and 2 are the gulf of execution (can they see what to do, and see that it is available?), question 3 is the gulf of evaluation (can they read what happened?). Two out, one back.

SCOPE OF WALKTHROUGHS

Walkthroughs are focused, so they are suitable for evaluating small parts of a product. Do not propose one to evaluate an entire system.

04 - A/B TESTING

A large-scale experiment

A/B testing offers another way to evaluate a website or an application running on a mobile device, and is often used for evaluating design changes in social media applications.

It is a large-scale experiment.
It compares how two groups of users perform on two versions of a design.
It can involve thousands of users.
It may create ethical dilemmas if users don't know they are part of the test.
Care is needed to ensure that other issues are not affecting users' behaviour.

HOW IT CONNECTS BACK

A/B testing appeared in Lecture 5 as an online method for informing the choice between alternatives, where setting appropriate metrics and choosing user group sets was called nontrivial. Here it returns as a form of large-scale experiment - the same technique seen from the evaluation side rather than the design-choice side.

THE ETHICS LINE

The dilemma is consent. Every other method in this course obtains informed consent, which tells participants why the study is happening, what they will do and what their rights are. A/B test participants typically know none of that, which is exactly why the deck flags it.

05 - KEY POINTS

What this deck wants you to walk away with

Five statements, all directly examinable.

Inspections can be used to evaluate requirements, mockups, functional prototypes, or systems.
User testing and heuristic evaluation may reveal different usability problems.
Design guidelines can be used to develop heuristics.
Walkthroughs are focused, so they are suitable for evaluating small parts of a product.
A/B testing is a form of large-scale experiment.

THE DISCUSSION THE DECK LEAVES OPEN

After the heuristic-evaluation activity the slides ask two questions worth having an answer for: do the heuristics help you focus on a website more intently than you would without them, and - if the activity was done on a smartphone - were all the heuristics relevant and useful? The expected observation is that heuristics act as a structured attention device, and that some general heuristics fit small-screen contexts poorly, which is precisely why the set has been revised for mobile devices, wearables and virtual worlds.

MISTAKES STUDENTS USUALLY MAKE

Mistakes students usually make

Each claim below is the wrong answer; the line beneath it is the correction, in the wording this course marks against.

Having the experts work together during the evaluation period.

In heuristic evaluation each expert works separately during the 1-2 hour evaluation period, taking one pass for a feel and a second pass for specific features. They work together only in the debriefing session, to prioritize problems.

Using a single expert.

The biggest problem list ends with should use multiple experts and aggregate their results, because one evaluator finds only a fraction of the problems.

"Heuristic evaluation replaces user testing."

User testing and heuristic evaluation may reveal different usability problems. Inspections complement user testing; they do not substitute for it.

Getting the three cognitive-walkthrough questions wrong.

1. Will the correct action be sufficiently evident? 2. Will the user notice that the correct action is available? 3. Will the user associate and interpret the response from the action correctly?

Proposing a walkthrough to evaluate a whole system.

Walkthroughs are focused, so they suit small parts of a product.

Forgetting the A/B testing ethical issue.

It may create ethical dilemmas if users don't know they are part of the test, and care is needed to ensure other issues are not affecting behaviour.

Treating the heuristics as fixed since 1990.

They were derived empirically from an analysis of 249 usability problems and have been revised for current technology - mobile devices, wearables, virtual worlds - by Nielsen and others.

CHEAT SHEET

Shortest correct answers

The night-before table: every term in this lecture with the smallest answer that still earns the mark.

Concept	Shortest correct answer
Inspection methods	An expert role-plays the users and identifies potential usability problems using a set of guidelines; no users present.
What inspections evaluate	Requirements, mockups, functional prototypes, or systems - at any stage.
Heuristic evaluation	A usability inspection developed by Nielsen in the early 1990s, derived empirically from an analysis of 249 usability problems.
Three stages	Briefing session; 1-2 hour evaluation with experts working separately (one pass for feel, one for features); debriefing to prioritize problems together.
Advantages	Few ethical and practical issues because users are not involved; usable at any stage; complements user testing.
Problems	Experts are difficult and expensive to find; applying heuristics requires knowledge and experience; multiple experts must be used and results aggregated.
Budd (2007) web heuristics	Clarity; minimize unnecessary complexity and cognitive load; provide users with context; promote a positive and pleasurable user experience.
Cognitive walkthrough	An inspection focused on ease of learning, stepping through a design prototype with a usage scenario.
The 3 questions	Will the correct action be sufficiently evident? Will the user notice it is available? Will the user associate and interpret the response correctly?
Pluralistic walkthrough	A managed team variation: experts begin separately, then a managed discussion leads to agreed decisions; suits participatory design.
Walkthrough scope	Focused, so suitable for evaluating small parts of a product.
A/B testing	A large-scale experiment comparing how two groups perform on two versions of a design; can involve thousands of users.
A/B ethics	May create dilemmas if users do not know they are part of the test.

APPLY IT

Exam-style application

Write your own answer first, then open the model answer. These are the longer-form questions this material generates.

Plan a heuristic evaluation of a mobile banking app, specifying the process, the heuristics and the staffing.

Process - three stages. A briefing session tells the experts what to do, gives them the assumptions about the user population and the context of use, and fixes the heuristic set. An evaluation period of 1-2 hours in which each expert works separately, taking one pass to get a feel for the product and a second pass to focus on specific features. A debriefing session in which the experts work together to prioritize the problems found. Heuristics: use Nielsen's revised 2014 set, but note the deck's own discussion question - not all general heuristics are relevant and useful on a smartphone, which is why revised sets exist for mobile devices; supplement with Budd's web criteria of clarity, minimizing complexity and cognitive load, providing context and promoting a positive experience. Design guidelines can be turned into heuristics with Granollers-style questions such as "does the user always know where they are located?". Staffing: multiple experts, because a single evaluator finds only a fraction of the problems and results must be aggregated; the best experts have knowledge of both the application domain and the users - so recruit at least one with banking domain knowledge. Finally, plan user testing alongside it, because the two may reveal different usability problems.

Run a cognitive walkthrough, in writing, over a single step: a user must change their delivery address before checkout.

Setup: the designer presents the checkout prototype and the scenario, and the expert is told the assumptions - first-time customer, on a phone, in a hurry. Step: locate the address change. Q1 - Will the correct action be sufficiently evident? No: the address appears as plain text with no affordance suggesting it can be edited, so the user has no cue that changing it is even possible here. Q2 - Will the user notice that the correct action is available? No: the Edit control sits below the fold on a phone, so it is not merely unlabelled but unseen - a visibility failure distinct from the affordance failure in Q1. Q3 - Will the user associate and interpret the response correctly? Partially: tapping Edit opens a full-page form with no indication that the checkout is preserved, so the user may believe they have left the purchase. Note the scope: this is one small part of the product, which is exactly what walkthroughs are suitable for.

A product manager wants to skip user testing because two experts already ran a heuristic evaluation. Respond.

The two are complementary, not interchangeable. The deck states plainly that user testing and heuristic evaluation may reveal different usability problems: an expert role-plays the intended user using codified guidelines, which catches violations of known principles, but cannot produce the behaviour of real users pursuing their own goals - the completion rates, task times, error types and unexpected workarounds that usability testing records on video and in key-press logs. Inspections also have their own limits: applying heuristics requires knowledge and experience; the best experts need knowledge of the application domain and the users, and such people are difficult and expensive to find; and because one evaluator finds only a fraction of the problems, multiple experts must be used and their results aggregated - two may not be enough. The honest framing is that the inspection has cheaply removed a class of known problems at any stage of design, and that a small usability test with five to ten representative users on representative tasks is what will now reveal the problems the heuristics could not predict.

POP QUIZ

Check yourself

6 questions. Every option is explained after submitting, including why the wrong ones are wrong.