HCI used to ask does it work.
Now it asks how does it feel.
Emotional interaction is concerned with how we feel and react when interacting with technologies - what makes us happy, sad, annoyed, anxious, frustrated or motivated, and how that can be designed for, detected, and deliberately used to change behaviour.
LEARNING OUTCOMES
- Explain how emotion relates to the user experience.
- Give examples of interfaces that are both pleasurable and usable.
- Explain visceral, behavioural and reflective design.
- Describe affective computing and emotional AI, and the techniques they use.
- Describe how technologies can be designed to change attitudes and behaviour.
From efficient systems to felt responses
HCI has traditionally been about designing efficient and effective systems. It is now also about designing interactive systems that make people respond in certain ways - to be happy, to be trusting, to learn, to be motivated.
- What makes us happy, sad, annoyed, anxious, frustrated, motivated or delirious - and how to translate that into aspects of the user experience.
- Why people become emotionally attached to certain products, such as virtual pets.
- Whether social robots can reduce loneliness and improve wellbeing.
- How to change human behaviour through emotive feedback.
Emotional intelligence is the starting point: how people express themselves and read each other through facial expressions, body language, gestures and tone of voice. When people are happy they laugh and relax their body posture; when angry they screw up their face. But Baumeister et al. (2007) argue the relationship between emotion and behaviour is more complex than a single cause-and-effect model - a point worth quoting in an exam.
| Emotion type | Character | Example |
|---|---|---|
| Automatic (affect) | Rapid, dissipates quickly. | A fit of anger. |
| Conscious | Develops slowly, takes a long time to go, involves reflection. | Jealousy. |
Should an interface be designed to improve how we feel - and if so, how? Our moods change continuously, so how would the interface keep track and know when to act? Which moods match which kinds of interface? These are posed as open questions; an exam answer should treat them as unresolved design problems, not solved ones.
Ortony, Norman et al. (2005): visceral, behavioural, reflective
The model's central claim is that our emotional state changes how we think, and therefore that design must address three levels at once.
Visceral design
Making products look, feel and sound good. The immediate, pre-conscious reaction.
Behavioural design
About use - this level equates with the traditional values of usability.
Reflective design
About the meaning and personal value of a product - what owning and using it says.
| Emotional state | How thinking changes | Consequence for the interface |
|---|---|---|
| Frightened or angry | Focus narrows; muscles tense and the body sweats. | The user is less tolerant - minor flaws become blocking problems. |
| Happy | Focus widens; the body relaxes. | The user is more likely to overlook minor problems and to be more creative. |
Brilliant colours and wild design attract attention at the visceral level. Affordances of use operate at the behavioural level. Cultural images and graphical elements are designed at the reflective level. One object, three levels - this is the exact analysis the exam expects you to reproduce for a different product.
V-B-R = Look, Use, Mean. Visceral is what you see in the first second, behavioural is what happens in the first minute, reflective is what you still think a year later.
Feedback that carries a feeling
Expressive interfaces provide reassuring feedback that can be both informative and fun - but which can also be intrusive, causing people to get annoyed and even angry.
- Colour, icons, sounds, graphical elements and animations make the look and feel of an interface appealing, and convey an emotional state.
- That in turn affects usability: people will put up with a slow download rate if the end result is appealing and aesthetic.
- Users invent their own expressiveness to compensate for text's lack of it - emoticons (happy, sad, sick, mad, very angry), plus shorthand such as LOL and I 12 CU 2NITE.
| Era | Approach | Example |
|---|---|---|
| 1980s | Emotional, anthropomorphic icons. | The smiling Apple face on reboot; a sad face on a crash. |
| Now | More impersonal but aesthetically pleasing. | The spinning beachball indicating the user must wait. |
The thermostat comparison makes the same point in hardware: the Nest is minimalist and aesthetically pleasing - a round face, a simple dial, a large font and large numbers - where earlier thermostat designs were utilitarian and dull. Identical function; different felt experience.
The seven causes, and how to write an error message
Badly designed interfaces make people frustrated, annoyed or angry. The lecture lists the causes.
- An application does not work properly or crashes.
- A system does not do what the user wants it to do.
- A user's expectations are not met.
- A system does not provide sufficient information for the user to know what to do.
- Error messages pop up that are vague, obtuse or condemning.
- The appearance of an interface is garish, noisy, gimmicky or patronizing.
- The system requires too many steps, only for the user to discover a mistake made earlier and have to start all over again.
Gimmicks are amusing to the designer and not to the user - clicking a link only to find the page is still "under construction". And the slides' own joke about error messages is the sharpest illustration: instead of "The application Word Wonder has unexpectedly quit due to a type 2 error", why not "the application has expectedly quit due to poor coding in the operating system"?
| Shneiderman's error-message guidelines | What it means in practice |
|---|---|
| Avoid terms like FATAL, INVALID, BAD | Do not condemn the user for the system's failure. |
| Reconsider audio warnings | A klaxon adds stress without adding information. |
| Avoid UPPERCASE and long code numbers | Shouting and hex codes are not diagnosis. |
| Messages should be precise rather than vague | Say exactly what went wrong and where. |
| Provide context-sensitive help | Offer the fix at the point of failure. |
Reeves and Nass (1996) argue computers should apologise and emulate human etiquette - "I'm really sorry I crashed. I'll try not to do it again." The slides raise the counter-questions rather than settling them: would users be as forgiving of a computer as of a person, and how sincere would they judge it to be? A friendly image in place of the impersonal 404 is the mild version of the same idea.
Children talk to Alexa as a friend and learn that please and thank you are unnecessary. Would that transfer to real life - "Aunty, get me my drink"? The slides note that parents should still teach manners, that Alexa can be configured to be polite, and ask how much parental control voice assistants should be given, and whether children would find it creepy to be nagged by their friend.
Machines that read feelings
Affective computing (Picard, 1998) is concerned with how to use computers to recognise and express emotions as humans do. Emotional AI aims to automate the measurement of feelings and behaviour, inferring them from facial expressions and voice.
- Involves designing ways for people to communicate their emotional state.
- Uses sensing technologies to measure GSR, facial expressions, gestures and body movement.
- Explores how affect influences personal health.
- Aims to predict a user's emotions and aspects of their behaviour - for example what someone is most likely to buy online when feeling sad, bored or happy.
| Technique | What it measures |
|---|---|
| Cameras | Facial expressions. |
| Biosensors on fingers or palms | Galvanic skin response (GSR). |
| Speech analysis | Affective expression through intonation, pitch and loudness. |
| Accelerometers and motion capture | Body movement and gestures. |
The six core expressions
Sadness, disgust, fear, anger, contempt, joy. These are the six typically measured.
The facial cues AI detects
Presence or absence of smiling, eye widening, brow raising, brow furrowing, raising a cheek, mouth opening, upper-lip raising and wrinkling of the nose - the basis of facial coding software such as Affdex.
How the data gets used
Screw up your face at an ad → disgust. Start smiling → happy. The website adapts its ad, movie storyline or content to match. In a car, a system might detect an angry driver and suggest a deep breath. Eye-tracking, finger pulse, speech, and the words and phrases used when tweeting or posting are analysed too.
The same techniques are used to infer or predict behaviour: a person's suitability for a job, or how they will vote at an election. The slides ask directly whether it is ethical for technology to read your emotions from your face or your tweets. An exam answer should name the inference leap - from expression, to emotion, to a consequential decision - as the point where the ethical problem bites.
Six core expressions: S-D-F-A-C-J - "Sad Dogs Frighten Angry Cats Joyfully". Note that surprise is not on this list; contempt is. That swap is a classic distractor.
Designing to change attitudes and behaviour
Persuasive technologies are interactive computing systems deliberately designed to change people's attitudes and behaviours (Fogg, 2003).
A diversity of techniques is used: pop-up ads, warning messages, reminders, prompts, personalised messages, recommendations, and Amazon 1-click. Collectively these are referred to as nudging.
Virtual pets
Emotional attachment does the persuading. A happy Pokemon makes a child feel good, a sulking one makes them feel bad, and the child changes behaviour to keep it happy. The open question: can technologies that monitor, nag or behave like a human keep people interested in looking after it - and in doing so, fitter themselves?
Tracking devices
Mobile apps that help people monitor and change behaviour - fitness, sleeping, weight. Comparison with online leaderboards and charts shows performance relative to peers and friends. Some apps encourage reflection, which in turn increases wellbeing and happiness.
Sustainable HCI
Designing interventions to reduce energy consumption. The most effective technique is feedback on consumption; simple infographics and emoticons are often most powerful; peer pressure and social norms are also powerful methods.
A large-scale visualisation of a whole street's electricity usage, stencilled in chalk on the road surface. It gave real-time feedback that everyone could see change each day, and reduced electricity consumption by 15%. Remember the number - it is the kind of detail that turns a vague answer into a graded one.
The same persuasive machinery deceives. Phishing uses the web to trick people into parting with personal details - PayPal, eBay and lottery-win letters - letting fraudsters access bank accounts and draw money out. Many vulnerable people fall for it. The art of deception is centuries old, but the internet allows ever more ingenious versions. Persuasion and deception share a mechanism; only the intent differs.
Giving human qualities to things that have none
Anthropomorphism is attributing human-like qualities to inanimate objects such as cars and computers. It is a well-known phenomenon in advertising - dancing butter, drinks and breakfast cereals - and is much exploited in HCI.
- Used to make the user experience more enjoyable and motivating, to make people feel at ease, and to reduce anxiety.
- Furnishing technologies with personalities can make them enjoyable to interact with.
| Situation | Anthropomorphic version | Neutral version |
|---|---|---|
| Welcome message | "Hello Chris! Nice to see you again. Welcome back. Now what were we doing last time? Oh yes, exercise 5. Let's start again." | "User 24, commence exercise 5." |
| Feedback on an error | "Now Chris, that's not right. You can do better than that. Try again." | "Incorrect. Try again." |
The lecture asks whether your preference differs by message type - and it usually does. A warm greeting is welcome; a warm rebuke can read as patronising, which is one of the undesirable UX qualities from Lecture 1.
Reeves and Nass (1996) found that computers which flatter and praise users in educational software have a positive impact - "Your question makes an important and useful distinction. Great job!" Students were more willing to continue with exercises given this kind of feedback.
Increasingly robots are used as companions in the home - remote, domestic, pet and sociable. The slides ask whether it is acceptable for senior people to develop an emotional attachment to a robot such as Zora. Answer it as a trade-off: measurable gains in wellbeing and reduced loneliness, set against deception, dependency and the substitution of human contact.
Mistakes students usually make
Each claim below is the wrong answer; the line beneath it is the correction, in the wording this course marks against.
Shortest correct answers
The night-before table: every term in this lecture with the smallest answer that still earns the mark.
| Concept | Shortest correct answer |
|---|---|
| Emotional interaction | How we feel and react when interacting with technologies. |
| Automatic vs conscious emotion | Rapid and quickly dissipating (anger) vs slow-developing and long-lasting (jealousy). |
| Visceral design | Making products look, feel and sound good. |
| Behavioural design | About use; equates with traditional usability. |
| Reflective design | About meaning and personal value. |
| Expressive interface | Uses colour, icons, sound, graphics and animation to convey an emotional state and give reassuring feedback. |
| Frustration causes | Crashes, unmet expectations, insufficient information, vague or condemning errors, garish appearance, too many steps. |
| Affective computing | Using computers to recognise and express emotions as humans do (Picard, 1998). |
| Emotional AI | Automating the measurement of feelings and behaviour, inferring them from face and voice. |
| Six core expressions | Sadness, disgust, fear, anger, contempt, joy. |
| Persuasive technology | Interactive systems deliberately designed to change attitudes and behaviours (Fogg, 2003); nudging. |
| Tidy Street | Chalk street-level visualisation of electricity use; cut consumption by 15% (Bird and Rogers, 2010). |
| Anthropomorphism | Attributing human-like qualities to inanimate objects. |
Exam-style application
Write your own answer first, then open the model answer. These are the longer-form questions this material generates.
Analyse a smartwatch using Norman's three levels, then say which level a fitness feature must hit to change behaviour.
A checkout page shows "INVALID INPUT - ERROR 500" when a card is declined. Rewrite it and justify each change.
A company wants a chatbot that apologises warmly whenever the system fails. Argue both sides using this lecture.
Check yourself
6 questions. Every option is explained after submitting, including why the wrong ones are wrong.