Interacting with technology
is a cognitive act.
Attention, perception, memory, learning, language and reasoning are not background details - they are the constraints your interface must be built around. This lecture gives you what users can and cannot be expected to do, and a design implication for every one of them.
LEARNING OUTCOMES
- Explain what cognition is and why understanding users matters.
- Describe the six cognitive processes and the design implication of each.
- Explain what mental models are and how to elicit them.
- Describe the gulfs of execution and evaluation.
- Compare information processing, distributed, external and embodied cognition.
Fast thinking, slow thinking
Cognition is thinking, remembering, learning, daydreaming, decision-making, seeing, reading, talking and writing. The lecture classifies it two ways, and both classifications say the same thing in different words.
| Classification | Fast / automatic side | Slow / deliberate side |
|---|---|---|
| Norman (1993) | Experiential cognition - effortless, in-the-moment, perceiving and reacting. | Reflective cognition - thinking, comparing, deciding, requires effort. |
| Kahneman (2011) | Fast thinking - 2+2, your own eye colour. | Slow thinking - 21 x 29, how many months have 31 days, the name of your first school. |
Sort the lecture's own quiz items and the distinction becomes concrete: 2+2 and what colour are your eyes are retrieved instantly; 21 x 29 and how many months have 31 days require you to run a procedure.
- Interacting with technology is cognitive, so cognitive limitations are interface constraints.
- Knowing the processes tells you what users can and cannot be expected to do.
- It identifies and explains the nature and causes of the problems users encounter.
- It supplies theories, modelling tools, guidance and methods that lead to better products.
A-P-M-L-R-P: Attention, Perception, Memory, Learning, Reading-speaking-listening, Problem-solving. Read it as "A Person Might Learn Rather Poorly" - which is exactly the point of the lecture.
Selecting what to concentrate on
Attention is selecting things to concentrate on, at a point in time, from the mass of stimuli around us. Focused and divided attention let us be selective, but limit our ability to keep track of all events.
Tullis (1987): the spacing study
Two hotel-price screens with identical information density (31%). The first took an average of 5.5 seconds to search, the second 3.2 seconds. The difference was spacing: screen one bunched the information together, screen two grouped characters into vertical categories.
Ophir et al. (2009): heavy multitaskers
Heavy multitaskers were more prone to being distracted than infrequent multitaskers, and found it harder to filter irrelevant information.
Lotteridge et al. (2015): the essay study
Heavy multitaskers were easily distracted, but could put that to good use when the distracting sources were relevant to the task. Irrelevant information hurt performance.
The phone-and-driving argument is the lecture's flagship example, and it is examinable in detail because the intuitive answer is wrong:
- Reaction times to external events are longer while talking on the phone (Caird et al., 2018).
- Response time is slower to unexpected events (Briggs et al., 2018), because phone drivers lean on expectations about what will happen next.
- Drivers try to imagine the other person's face, which competes for the processing resources needed to notice what is in front of them.
- Hands-free is not safer - the same cognitive processing happens when talking.
- Talking to a front-seat passenger is less dangerous: both people can stop mid-sentence when they see a hazard, whereas a remote person is not privy to what the driver sees and carries on talking.
Make information salient when it needs attending to. Use colour, ordering, spacing, underlining, sequencing and animation to make things stand out. Avoid cluttering the interface. Avoid using techniques just because the software allows it. Design ways of supporting effective switching and returning to an interface.
The hands-free question is a favourite. The correct answer is no, hands-free is not safer, and the reason is cognitive, not manual - it is the conversation, not the handset, that consumes attention.
How information becomes experience
Perception is how information is acquired from the world and transformed into experiences. The obvious implication is to design representations that are readily perceivable.
Weller (2004) compared two screens: one that separated information with colour contrast and one that used a border. People took less time to locate grouped items - the border won. The counter-argument the slides raise: some argue too much white space on web pages is detrimental, because it makes information hard to find.
- Icons should let users readily distinguish their meaning.
- Bordering and spacing are effective visual ways of grouping information.
- Sounds should be audible and distinguishable.
- Speech output should let users distinguish between the set of spoken words.
- Text should be legible and distinguishable from the background.
- Tactile feedback should allow users to recognise and distinguish different meanings.
- Research proper colour contrast: yellow on black or blue is fine; yellow on green or white is not. Use haptic feedback judiciously.
Perception design reduces to one sentence: make it distinguishable. Text from background, icon from icon, word from word, group from group. Every implication on the slide is that sentence applied to a different sense.
Encode, then retrieve
Memory involves first encoding and then retrieving knowledge. We do not remember everything - memory filters and processes only what is attended to.
- EncodingThe first stage: it determines which information is attended to and how it is interpreted.
- Processing depthThe more attention paid, and the more it is compared with existing knowledge, the more likely it is to be remembered.
- Context bindingWhere and when you encoded something affects whether you can retrieve it later.
- RetrievalRecognition is far easier than recall.
The slides use HCI itself as the example: reflecting on the material, doing exercises, discussing it and writing notes beats passively reading a book, listening to a lecture or watching a video. Depth of processing, not exposure, drives retention.
Context is why the neighbour on the train is unrecognisable for a few seconds: you encoded them in the hallway of your apartment block, and the retrieval cue is missing. And Henkel (2014) found we remember less about objects we have photographed than about objects we simply looked at.
Recognition > recall
Command-based interfaces require recalling a name from hundreds. GUIs give visually-based options - menus and icons - that users only need to browse until they recognise one. History lists, visited URLs, song titles and tabs all exist to support recognition memory.
Visual cues beat arbitrary material
People are very good at remembering the colour of items, the location of objects and marks on an object. They are bad at arbitrary material - birthdays and phone numbers. Try recalling your grandparents' birthdays versus the covers of the last two films you rented.
Miller (1956) showed immediate memory capacity is limited to about seven items. Designers then wrongly infer: seven menu options, seven icons, seven bullets, seven tabs. That is an inappropriate application of the theory - people scan lists, tabs and menu items for the one they want; they do not recall them from memory after briefly seeing them. Sometimes a small number is good, but it depends on the task and the available screen estate.
Personal information management (PIM) is the applied case: vast numbers of documents, images, music files, video clips, emails, attachments and bookmarks, all needing to be saved, named and found again. Naming is the most common encoding method and the hardest to recall at scale.
- Bergman & Whittaker (2016), step 1Decide what stuff to keep.
- Step 2Decide how to organise it when storing.
- Step 3Decide which strategies to use to retrieve it later.
- Most people use folders and naming, with a strong preference for scanning across and within folders rather than searching.
- Search engines only help if you know the name of the file; smart search helps with partial names or first letters (Apple's Spotlight).
- File systems should optimise both recall-directed search and recognition-based scanning.
- Richer encoding helps: colour, flagging, images, flexible text, time stamping.
- SenseCam (Microsoft Research) is a wearable that intermittently takes photos without user intervention; revisiting the images improved memory in people with Alzheimer's.
- Multifactor authentication increases memory load - ZIP code, birthplace, a memorable date, first school - to increase security. Password managers reduce that load to one master password; biometrics may remove passwords entirely.
- Digital forgetting (Sas and Whittaker, 2013): after a break-up, shared photos are emotionally painful. Harvesting and transforming content - turning photos of an ex into an abstract collage - helps with closure.
Do not overload users' memories with complicated procedures. Design interfaces that promote recognition rather than recall. Provide various ways of encoding information - categories, colour, flagging, time stamping.
Learning, language, and reflective cognition
The remaining three processes are shorter on the slides but each carries its own design implications, and the exam treats them as equals.
4. Learning
Two senses: learning how to use an application, and using an application to understand a topic. People find it hard to learn by following instructions in a manual - they prefer to learn by doing. Two types: incidental (recognising faces, what you did today) and intentional (studying for an exam, learning to cook). Intentional is much harder, which is why digital media, animations and VR have been built to support it. Implications: design interfaces that encourage exploration; design interfaces that constrain and guide learners; dynamically link concepts and representations.
5. Reading, speaking & listening
Ease differs by person and mode. Many prefer listening to reading; reading can be quicker than speaking or listening; listening requires less cognitive effort than reading or speaking; dyslexic users have difficulty recognising written words. Applications: speech-recognition systems, speech-output systems (text-to-speech for blind users), and natural-language systems. Implications: keep speech-based menus and instructions short; accentuate the intonation of artificial voices because they are harder to understand than human ones; let users enlarge text.
6. Problem-solving, planning, reasoning & decision-making
All involve reflective cognition - thinking about what to do, what the options are and what the consequences would be. Often conscious, often discussed with others, and often supported by artifacts such as maps, books, pen and paper. Implications: provide extra information and functions for users who want to understand how to do an activity better; use simple computational aids for rapid decision-making on the move.
Gardner and Davis (2013) argue the "app mentality" makes people risk-averse: they rely on a multitude of apps, grow anxious, become unable to decide alone, and resort to looking things up and comparing notes on social media. The slides pose it as an open question, so an exam answer should present it as a debated claim, not a finding.
Mental models
Cognitive frameworks explain and predict user behaviour at the interface, based on theories of behaviour, focusing on the mental processes that take place and on the use of artifacts and representations. The best known is the mental model.
Craik (1943) described mental models as internal constructions of some aspect of the external world that enable predictions to be made. They cover both how to use a system (what to do next) and how the system works (what to do with unfamiliar systems or unexpected situations). They involve both unconscious and conscious processes, and images and analogies get activated.
| Model depth | Meaning | Example |
|---|---|---|
| Deep model | You understand the underlying mechanism. | Knowing how a car engine works. |
| Shallow model | You know the procedure but not the mechanism. | Knowing how to drive a car. |
The thermostat problem (Kempton, 1996)
You come home to a cold house. Do you set the thermostat to maximum or to the temperature you want? Many people set it to maximum, which does not help. The erroneous model is the general valve theory - a "more is more" principle generalised from gas pedals, gas cookers, taps and volume knobs - whereas a thermostat behaves like an on-off switch.
The oven variant
Starving, with an uncooked pizza and an electric oven: do you preheat to 375 as instructed, or turn it higher to warm up faster? Same erroneous valve model, same wrong answer.
Buttons pressed twice
Elevators and pedestrian crossings: people hit the button at least twice because they think it will make the lights change faster or the lift arrive sooner. Norman (1983) - models of interactive devices are poor, incomplete, easily confusable, and based on inappropriate analogies and superstition.
The lecture's ATM exercise is the standard way of exposing how shallow your own model is: how much can you withdraw, what would happen at another machine, what is on the magnetic strip, what happens if you enter the wrong number, why are there pauses between steps, why does the card stay inside, and why do you count the money? Payne (1991) found people frequently resort to analogies, and their accounts varied greatly and were often ad hoc.
Clear and easy-to-use instructions; appropriate tutorials and context-sensitive guidance; online videos and chatbot help windows; transparency so interfaces are intuitive; and affordances that show what actions the interface allows - swiping, clicking, selecting.
The gulfs of execution and evaluation
The gulfs explicate the gaps that exist between the user and the interface.
| Gulf | Direction | The question the user is asking |
|---|---|---|
| Gulf of execution | User → physical system | How do I do what I intend? Is it obvious what to do next? |
| Gulf of evaluation | Physical system → user | Did it work? Can I tell the system's state from what it shows me? |
- Bridging the gulfs reduces the cognitive effort required to perform tasks.
- The gulfs reveal whether an interface increases or decreases cognitive load.
- They also reveal whether it is obvious what to do next.
eXecution goes out, eValuation comes back. Execution is the gap on the way to the machine; evaluation is the gap on the way back to you. Visibility bridges execution; feedback bridges evaluation.
Beyond the head
The remaining frameworks all push in one direction: cognition is not confined to the inside of a single skull.
3. Information processing
Conceptualises human performance in metaphorical terms of information-processing stages. Limitation: it models mental activities that happen exclusively inside the head, and so does not adequately account for how people interact with computers and other devices in the real world.
4. Distributed cognition (Hutchins, 1995)
Concerned with cognitive phenomena across individuals, artifacts, and internal and external representations, described as propagation across representational state. Information is transformed through different media - computers, displays, paper, heads. It examines the distributed problem-solving that takes place, verbal and non-verbal behaviour, coordinating mechanisms such as rules and procedures, the communication as collaborative activity progresses, and how knowledge is shared and accessed.
5. External cognition
Explains how we interact with external representations - maps, notes, diagrams - what the cognitive benefits are, and how they extend our cognition.
6. Embodied interaction
A newer approach in which user interfaces merge seamlessly with the physical world, making use of the physical objects that surround us.
External cognition has two examinable mechanisms:
| Mechanism | What it does | Example |
|---|---|---|
| Externalising to reduce memory load | Offload what must be remembered onto the world. | Diaries, reminders, calendars, notes, shopping lists, to-do lists. Post-its, piles and marked emails also encode priority by where they are placed. They remind us that we need to do something, what to do, and when to do it. |
| Computational offloading | Using a tool together with an external representation to carry out a computation. | 234 x 456 in your head, on paper, or with a calculator. Now try CCXXXIIII x CCCCXXXXXVI - the identical sum in Roman numerals is far harder, proving the representation, not the arithmetic, is doing the work. |
The Roman-numeral sum is the single best exam anchor in this lecture. Same numbers, same answer, radically different difficulty ⇒ the representation carries part of the cognition. That one sentence answers most external-cognition questions.
Mistakes students usually make
Each claim below is the wrong answer; the line beneath it is the correction, in the wording this course marks against.
Shortest correct answers
The night-before table: every term in this lecture with the smallest answer that still earns the mark.
| Concept | Shortest correct answer |
|---|---|
| Cognition | Thinking, remembering, learning, seeing, reading, deciding - split into experiential vs reflective, or fast vs slow. |
| Six processes | Attention, perception, memory, learning, reading-speaking-listening, problem-solving/planning/reasoning/deciding. |
| Attention implication | Make it salient, avoid clutter, support switching and returning. |
| Perception implication | Make everything distinguishable - text, icons, sounds, groups; borders and spacing group well. |
| Memory implication | Recognition rather than recall; do not overload; give many ways to encode. |
| 7 ± 2 | Immediate memory capacity; misapplied when used to cap menu or tab counts. |
| Mental model | Internal construction of part of the external world that enables predictions. |
| Gulf of execution | Distance from the user to the physical system. |
| Gulf of evaluation | Distance from the physical system to the user. |
| Distributed cognition | Cognition propagating across individuals, artifacts and representations. |
| External cognition | How we interact with external representations; includes externalising memory and computational offloading. |
| Embodied interaction | Interfaces merging with the physical world and the objects around us. |
Exam-style application
Write your own answer first, then open the model answer. These are the longer-form questions this material generates.
Your file manager relies entirely on users typing exact file names into a search box. Diagnose the problem with the memory material and propose three fixes.
A hospital operating room adds a fourth real-time monitor. Argue, using attention research, what this does to the clinician and what the design should do instead.
Explain, using two frameworks from this lecture, why a pilot's paper checklist is not merely a memory aid.
Check yourself
6 questions. Every option is explained after submitting, including why the wrong ones are wrong.