Straighten out your thinking
before you lay out your widgets.
This is the longest deck of the course. It moves from assumptions and claims, through the conceptual model and interface metaphors, to the five interaction types and twenty interface types - and the recurring question of which one to choose.
LEARNING OUTCOMES
- Explain what is meant by the problem space.
- Explain how to conceptualize interaction.
- Discuss the use of interface metaphors as part of a conceptual model.
- Explain the five interaction types and the twenty interface types.
- Discuss advanced interfaces and natural user interfaces.
Assumptions, claims, and the question nobody asked
Before designing anything you must understand the problem space: what you want to create, what your assumptions are, and whether it will achieve what you hope.
| Term | Definition | Example from the slides |
|---|---|---|
| Assumption | Taking something for granted when it needs further investigation. | "People will want to watch TV while driving." |
| Claim | Stating something to be true when it is still open to question. | "A multimodal style of interaction for controlling GPS - one that involves speaking while driving - is safe." |
The robot waiter case
The proposed benefits: the robot could take orders and entertain customers by conversing with them, and make recommendations for restless children or fussy eaters. But those are just assumptions. The real problem being addressed is: "It is difficult to recruit good wait staff who provide the level of customer service to which we have become accustomed." Naming the real problem changes which solutions are even candidates.
The 3D TV case
There was no existing problem to overcome - it proposed a new way of experiencing TV. The assumption: people would really enjoy the enhanced clarity and colour detail. The claim: people would not mind paying a lot more for a 3D-enabled screen because of the new experience. Both turned out to be false, which is why it is the standing example.
- Are there problems with an existing product or user experience?If so, what exactly are they?
- Why do you think there are problems?Name the cause, not the symptom.
- How might your design ideas overcome these?Connect each idea to a named problem.
- If designing for a new experienceHow do your ideas support, change or extend current ways of doing things?
Orientation - lets the team ask specific questions about how the conceptual model will be understood. Open-mindedness - prevents the team becoming narrowly focused early on. Common ground - lets the team establish a set of commonly agreed terms.
Assumption = taken for granted. Claim = stated as true. An assumption is something you did not notice you believed; a claim is something you said out loud. Both need investigation before they become requirements.
A high-level description of how a system is organized and operates
A conceptual model enables designers to straighten out their thinking before they start laying out their widgets. The best conceptual models appear obvious and simple, and the operations they support are intuitive to use.
- Metaphors and analogies - so people understand what a product is for and how to use it for an activity.
- Concepts people are exposed to through the product - task-domain objects, their attributes and operations.
- Relationships and mappings between these concepts.
- What will the users be doing?Identify the tasks being carried out.
- How will the system support these?Map system function to user task.
- What interface metaphor, if any, is appropriate?Or is none better than a forced one?
- What interaction modes and styles?Always keeping in mind how the user will understand the underlying conceptual model.
The slides ask you to name the components of the conceptual model behind most shopping sites: the shopping cart, proceeding to check-out, 1-click, gift wrapping, and the cash till. Notice that every one of them is borrowed from a physical shop - the conceptual model is a metaphor here.
Familiar knowledge, unfamiliar functionality
An interface metaphor is designed to be similar to a physical entity but also has its own properties - the desktop metaphor, web portals. It can be based on an activity, an object, or a combination of both.
Metaphors exploit users' familiar knowledge to help them understand the unfamiliar. They conjure up the essence of an unfamiliar activity, letting users leverage that to understand more aspects of unfamiliar functionality. A conceptual model instantiated at the interface is a metaphor - the desktop is the standing example.
The card metaphor
Very popular because it has a familiar form factor. Cards can easily be flicked through, sorted and themed; they structure content into meaningful chunks, similarly to how paragraphs chunk related sentences; and their material properties give the appearance of the surface of paper.
The calculator example
Two digital calculators: one designed too literally as a physical calculator, one designed more appropriately for a computer screen. A metaphor copied slavishly imports the old medium's limitations.
| Benefits | Problems |
|---|---|
| Makes learning new systems easier. | Breaks conventional and cultural rules - e.g. a recycle bin placed on a desktop. |
| Helps users understand the underlying conceptual model. | Can constrain designers in how they conceptualize a problem space. |
| Can be innovative and make computing accessible to a greater diversity of users. | Can conflict with design principles. |
| Forces users to understand the system only in terms of the metaphor. | |
| Designers can inadvertently transfer over the bad parts of an existing bad design. | |
| Limits designers' imagination in coming up with new conceptual models. |
The problems list is longer than the benefits list, and the exam usually asks for problems. Remember the recycle bin on the desktop - a bin is not something you keep on your desk - as the one-line proof that metaphors break real-world rules.
Instructing, conversing, manipulating, exploring, responding
An interaction type describes what the user is doing when interacting with a system. An interface type is the kind of interface used to support that mode.
1. Instructing
Users tell the system what to do - tell the time, print a file, save a file - by issuing commands, selecting options, speaking commands, gesturing or pressing buttons. A very common model, underlying word processors, VCRs and vending machines. Main benefit: quick and efficient interaction, good for repetitive actions performed on multiple objects.
2. Conversing
Interacting as if having a conversation with another human. Ranges from simple voice-recognition menu-driven systems to complex natural-language dialogs: timetables, search engines, advice-giving systems, help systems, virtual agents, toys and pet robots. Pro: familiar to novices, making them comfortable, at ease and less scared. Con: misunderstandings arise when the system cannot parse what the user says - voice assistants misunderstanding children is the slides' example.
3. Manipulating
Dragging, selecting, opening, closing and zooming on virtual objects, exploiting users' knowledge of how they move and manipulate in the physical world. Can involve physical controllers (Wii), air gestures (Kinect), or tagged physical objects whose manipulation triggers physical/digital events.
4. Exploring
Moving through virtual or physical environments. Users can explore a virtual 3D environment; physical environments embedded with sensors trigger digital or physical events when they detect someone. Examples include cities, parks, buildings, rooms and datasets that users fly over and zoom into.
5. Responding
The system takes the initiative and alerts the user to something it "thinks" is of interest, by detecting location or presence, or from what it has learned from repeated behaviour. Examples: alerting the user to a nearby coffee bar where friends are meeting, or a fitness tracker notifying a milestone. It is an automatic response with no request made by the user.
Direct Manipulation (DM) is the sharpened form of manipulating. Shneiderman (1983) coined the term, from his fascination with computer games. Its three properties:
- Continuous representation of the objects and actions of interest.
- Physical actions and button pressing instead of issuing commands with complex syntax.
- Rapid, reversible actions with immediate feedback on the object of interest.
| Benefits of DM | Disadvantages of DM |
|---|---|
| Novices learn basic functionality quickly. | Some people take the metaphor of direct manipulation too literally. |
| Experienced users work rapidly across a wide range of tasks, even defining new functions. | Not all tasks can be described by objects, and not all actions can be done directly. |
| Intermittent users retain operational concepts over time. | Some tasks are better achieved by delegating - spell checking is the example. |
| Error messages are rarely needed. | Can become screen-space "gobblers". |
| Users immediately see whether actions further their goals. | Moving a mouse around the screen can be slower than pressing function keys. |
| Users experience less anxiety, gain confidence and mastery, and feel in control. |
Direct manipulation for "doing" tasks - designing, drawing, flying, driving, sizing windows. Issuing instructions for repetitive tasks - spell-checking, file management. Conversation for children, computer-phobic and disabled users and specialised applications such as phone services. Hybrid models are often employed, supporting several ways of doing the same action - but they take longer to learn.
I-C-M-E-R: Instructing, Conversing, Manipulating, Exploring, Responding - "I Can Make Everything Respond". Only the last one starts with the system, which is the distinguishing feature the exam tests.
Command line to mobile
Twenty interface types are listed. The first eight are the classic and mainstream ones, and carry the most detail on the slides.
1. Command-based
Type abbreviations at a prompt (ls) and the system responds. Some commands are hard-wired at the keyboard (delete, enter, undo), others assignable (F11 = print). Efficient, precise and fast, but with a large overhead in learning the command set. Key research questions concern form, name types and structure; consistency is the most important design principle - always use the first letter of the command. Still popular for web scripting, and used in Second Life for visually impaired users.
2a. WIMP
The Xerox Star was the first WIMP, giving rise to GUIs. Windows - scrolled, stretched, overlapped, opened, closed and moved with the mouse. Icons - representing applications, objects, commands and tools, opened when clicked. Menus - lists of options scrolled through and selected. Pointing device - a mouse controlling the cursor as the point of entry to windows, menus and icons.
2b. GUI
The same building blocks, but more varied: colour, 3D, sound, animation, many types of menus, icons and windows, plus new graphical elements - toolbars, docks, rollovers.
3. Multimedia
Combines graphics, text, video, sound and animation within a single interface with various forms of interactivity. Pros: rapid access to multiple representations, better presentation than any medium alone, easier learning, better understanding, more engagement, more pleasure, and encouragement to explore. Con: a tendency to play video clips and animations while skimming the accompanying text and diagrams. The design answer is hands-on interactivities, simulations, quizzes, electronic notebooks and games that must be completed.
4. Virtual reality
Computer-generated graphical simulations providing "the illusion of participation in a synthetic environment rather than external observation of such an environment" (Gigante, 1993). Pros: higher fidelity than multimedia; induces a sense of presence - "a state of consciousness, the psychological sense of being in the virtual environment" (Slater and Wilbur, 1999); provides first- and third-person viewpoints. Cons: head-mounted displays are uncomfortable and can cause motion sickness and disorientation. Used for flying simulators and for overcoming phobias such as spiders and public speaking.
5. Information visualization
Computer-generated interactive graphics of complex data that amplify human cognition, letting users see patterns, trends and anomalies (Card et al., 1999). The aim is to enhance discovery, decision-making and explanation. Techniques include 3D interactive maps that zoom, presenting data via webs, trees, clusters, scatterplots and interconnected nodes.
6. Web
Early sites were largely text with hyperlinks, and the concern was structuring information so users could navigate quickly. Now there is more emphasis on making pages distinctive, striking and pleasurable - the vanilla versus multi-flavour tension between ease of finding something and an aesthetic, enjoyable experience. Veen's three design principles: (1) Where am I? (2) Where can I go? (3) What's here?
7. Consumer electronics & appliances
Everyday devices in the home, public places or cars - washing machines, remotes, photocopiers, printers, navigation systems - and personal devices such as MP3 players, digital clocks and cameras. Used for short periods, so they must be usable with minimal, if any, learning. Design as transient interfaces with short interactions, keep them simple, and consider the trade-off between soft and hard controls.
8. Mobile
Handheld devices used while on the move, now pervasive: restaurants take orders, car rentals check in returns, supermarkets check stock, streets host multi-user gaming, and education supports life-long learning. Challenges: small screens, few keys, restricted controls. Innovations include roller wheels, rocker dials, up/down lips, two- and four-way directional keypads, softkeys and silk-screened buttons. Tricky for those with poor manual dexterity.
Where am I? Where can I go? What's here? Three words each - this is the most quotable item in the web section and the one most likely to appear verbatim.
Speech to brain-computer
The remaining twelve are the post-WIMP interfaces, each with its own research and design issues.
9. Speech
A person talks with a system that has a spoken-language application - timetables, travel planners. Used most for inquiring about specific information (flight times) or performing a transaction (buying a ticket), and by people with disabilities. The most popular current use is call routing with caller-led speech - "I'm having problems with my voice mail". Directed dialogs put the system in control, asking specific questions requiring specific responses; more flexible systems let the user take the initiative ("I'd like to go to Paris next Monday for two weeks") but risk more errors because the caller assumes the system is human. Guided prompts get callers back on track. Design issues include the type of voice actor - male, female, neutral or dialect.
10. Pen
Lightpens and styluses let people write, draw, select and move objects, capitalising on drawing skills honed from childhood. Digital pens such as Anoto combine an ordinary ink pen with a digital camera that records everything written on special paper. Pro: quick and easy annotation of existing documents. Cons: the hand can occlude part of the screen while writing, and there can be lag that feels clunky.
11. Touch
Touch screens such as walk-up kiosks detect presence and location of a touch. Multi-touch supports swiping, flicking, pinching, pushing and tapping. Much faster to scroll through wheels, carousels and bars of thumbnails by finger flicking, but more cumbersome, error-prone and slower to type on a virtual keyboard than a physical one.
12. Air-based gestures
Camera recognition, sensors and computer vision recognise body, arm and hand gestures in a room - Kinect and EyeToy. Movements map onto gaming motions: swinging, bowling, hitting, punching, with players represented as avatars. Research issues: how the computer recognises and delineates gestures (deictic and hand-waving), and whether holding a control device feels more intuitive than controller-free gestures.
13. Haptic
Tactile feedback applying vibration and forces to the body using actuators embedded in clothing, often with motion capture. Can enrich the experience or nudge users to correct an error, and can simulate the sense of touch between remote people. Design issues: where to place actuators, whether to use single or sequenced touches, when to buzz and how intensely, and how the wearer feels it in different contexts.
14. Multimodal
Multiplies how information is experienced using different modalities - touch, sight, sound, speech - to support more flexible, efficient and expressive interaction. The most common combination is speech and vision. Research issues: recognising and analysing speech, gesture and eye gaze; what is gained by combining inputs and outputs; and whether talking and gesturing as humans do with each other is genuinely a natural way to interact with a computer.
15. Shareable
Designed for more than one person, providing multiple inputs and sometimes simultaneous input by co-located groups: large wall displays with pens or gestures, interactive tabletops using fingertips - DiamondTouch, Smart Table, Surface. Advantages: a large interactional space supporting flexible group working; multiple users can point to and touch displayed information, view interactions simultaneously and share a point of reference; more equitable participation than groups sharing one PC. Findings: horizontal surfaces support more turn-taking and collaboration than vertical ones, and larger tabletops do not improve group working - they encourage more division of labour.
16. Tangible
Sensor-based interaction where physical objects such as bricks are coupled with digital representations, so manipulating the object causes a digital effect. Examples: Chromarium cubes (turning them mixes digital colour animations on an adjacent wall), Flow Blocks (embedded numbers and lights vary with how blocks are connected), and Urp (physical building models on a tabletop with tokens for wind and shadow, changing digital shadows over time). Benefits: held in both hands, combined in ways other interfaces cannot, explored by more than one person, placed on top of, beside and inside each other, encouraging different ways of representing a problem space and leading to greater insight, learning, problem-solving, creativity and reflection. Design issue: the coupling should be explicit for learning and can be implicit and unexpected for entertainment.
17. Augmented & mixed reality
Augmented reality superimposes virtual representations on physical devices and objects; mixed reality combines views of the real world with views of a virtual environment. Applications in medicine (virtual X-rays and scans), air traffic control (identifying planes hard to make out), games, flying and exploring. Design issues: what kind of digital augmentation, when and where in the physical environment, how to stand out without distracting from the ongoing task, how to align with real-world objects, and which device to use.
18. Wearable
Began with head- and eyewear-mounted cameras recording what was seen and accessing digital information; since then jewellery, caps, smart fabrics, glasses, shoes and jackets. Applications include automatic diaries, tour guides, cycle indicators and fashion clothing. Google Glass was short-lived. Four design issues: comfort (light, small, unobtrusive, fashionable, preferably hidden), hygiene (can the clothing be washed?), ease of wear (can the electronics be removed and replaced?), and usability (how does the user control embedded devices?).
19. Robots
Four types: remote robots in hazardous settings, domestic robots helping around the house, pet robots as human companions, and sociable robots that work collaboratively with humans and socialise with them as peers. Drones are unmanned aircraft used for entertainment (carrying drinks and food at festivals), agriculture (flying over vineyards and fields to collect data), and tracking poachers in African wildlife parks; flying low and streaming photos that are stitched into maps to determine crop health and harvest timing. Design questions: how humans react to physical versus virtual robots; whether robots should look human or clearly robotic; and whether interaction should be human-like or human-computer-like.
20. Brain-computer
A BCI provides a communication pathway between a person's brain waves and an external device such as a screen cursor. The person is trained to concentrate on the task, and the system works by detecting changes in neural functioning. The slides' example is a woman who is paralyzed selecting letters on screen.
A NUI lets users interact with a computer in the same ways they interact with the physical world - using voice, hands and bodies: speaking to machines, stroking their surfaces, gesturing in the air, dancing on mats that detect feet movements, smiling to get a reaction. The slides then ask the sceptical question directly: how natural are NUIs?
Is multimedia better than tangible for learning? Is speech as effective as command-based? The answer will depend on task, users, context, cost and robustness. Note the trends the slides state: mobile platforms taking over from PCs, speech being used more for commercial services, appliance and vehicle interfaces becoming more important, and shareable and tangible interfaces entering homes, schools, public places and workplaces.
Windows, menus and icons in detail
The GUI section breaks into three sub-topics that each carry their own design issues, and each is examinable separately.
i. Window design
Windows were invented to overcome the physical constraints of a display, enabling more information to be viewed and more tasks performed; scroll bars extend this further. Multiple windows make it hard to find the one you want - listing, tabbing, iconizing, shrinking and thumbnails help. Research issues: window management letting users move fluidly between windows and monitors; how to switch attention without getting distracted; applying spacing, grouping and simplicity; which terms to use for options ("Front" versus "bring to front"); and the finding that mega menus are easier to navigate than drop-downs.
ii. Menu design
Flat list - good for showing many options at once on a small display. Drop-down - shows more options on the same screen, including cascading. Pop-up - appears when a command key is pressed. Contextual - gives access to often-used commands for a particular item. Collapsible - toggles between + and - on a header to expand or contract contents. Mega - all options shown in a 2D drop-down layout.
iii. Icon design
Icons are assumed to be easier to learn and remember than commands, can be compact and variably positioned, and are now pervasive - desktop objects, tools (paintbrush), applications (browser) and operations (cut, paste, next, accept, change). The mapping between representation and referent can be similar (a picture of a file for the object file), analogical (scissors for cut) or arbitrary (an X for delete). The most effective icons are similar ones. Many operations are actions, which are harder to represent, so designers combine objects and symbols capturing the salient part of the action. Text labels help identification for small icon sets; for large sets - photo editing, word processing - use the hover function.
Icon mappings: S-A-A - Similar (file → file), Analogical (scissors → cut), Arbitrary (X → delete). Effectiveness falls in exactly that order, so the first is best and the last must be learned.
Mistakes students usually make
Each claim below is the wrong answer; the line beneath it is the correction, in the wording this course marks against.
Shortest correct answers
The night-before table: every term in this lecture with the smallest answer that still earns the mark.
| Concept | Shortest correct answer |
|---|---|
| Assumption | Taking something for granted when it needs further investigation. |
| Claim | Stating something to be true when it is still open to question. |
| Conceptual model | A high-level description of how a system is organized and operates. |
| Model components | Metaphors and analogies, concepts (task-domain objects, attributes, operations), relationships and mappings. |
| Interface metaphor | An interface designed to be similar to a physical entity but with its own properties, e.g. the desktop. |
| Interaction types (5) | Instructing, conversing, manipulating, exploring, responding. |
| Direct manipulation | Continuous representation, physical actions instead of complex syntax, rapid reversible actions with immediate feedback (Shneiderman, 1983). |
| WIMP | Windows, Icons, Menus, Pointing device - first seen in the Xerox Star. |
| Menu kinds | Flat list, drop-down (cascading), pop-up, contextual, collapsible, mega. |
| Icon mappings | Similar, analogical, arbitrary - similar is most effective. |
| Presence (VR) | The psychological sense of being in the virtual environment (Slater and Wilbur, 1999). |
| Veen's web principles | Where am I? Where can I go? What's here? |
| Robot types (4) | Remote, domestic, pet, sociable. |
| Wearable design issues | Comfort, hygiene, ease of wear, usability. |
| NUI | An interface letting users interact using voice, hands and bodies as they do with the physical world. |
| Which interface? | Depends on task, users, context, cost and robustness. |
Exam-style application
Write your own answer first, then open the model answer. These are the longer-form questions this material generates.
A team proposes an in-car voice assistant that reads out incoming messages while driving. Identify the assumption and the claim, and say what the real problem might be.
Choose an interaction type and an interface type for a museum exhibit teaching children how a river system works, and justify both.
Argue whether a natural user interface is genuinely natural, using this lecture's own material.
Check yourself
6 questions. Every option is explained after submitting, including why the wrong ones are wrong.