Public webpage for sharing information about Dr. Joyner's CS6750 - Human Computer Interaction course in Spring 2022.
[NOISE] Feedback cycles are the way in which people interact with the world, and then get feedback on the results of those interactions. We’ll talk about the ubiquity of those feedback cycles. Then we’ll talk about the gulf of execution, which is the distance between a user’s goals and the execution of the actions required to realize those goals. Then we’ll talk about the Gulf of evaluation, which is the distance between the effects of those actions and the user’s understanding of those results. We’ll discuss seven questions we should ask ourselves when designing feedback cycles for users and we’ll also look at applications of these in multiple areas of our everyday lives.
Feedback cycles are incredibly ubiquitous, whether or not there’s a computational interface involved. Everything from reading to driving a car to interacting with other people could be an example of a feedback cycle in action. They’re how we learn everything, from how to walk to how to solve a Rubik’s cube to how to take the third order partial derivative of a function. I assume, I’ve never done that. We do something, we see the result, and we adjust what we do the next time accordingly. You may have even seen other examples of this before, too. If you’ve taken Ashok’s and mine knowledge-based AI class, we talk about how agents are constantly interacting with, learning from, and affecting the world around them. That’s a feedback cycle. If you’ve taken cyber physical systems course, you’ve seen this this without human involved at all, as a system can autonomously read input and react accordingly. Under some definitions, some people would even call this the artificial intelligence, specifically because it mimics what a human actually does. They act in the world and they evaluate the result. In fact, if you look at some of the definitions of intelligence out there, you’ll find that many people actually define feedback cycles as the hallmark of intelligent behavior. Or they might define intelligence as abilities that must be gained through feedback cycles. Colvin’s definition, for example, involves adjusting to one’s environment, which means acting in it and then evaluating the results. Dearborn’s definition of learning or profiting by experience is exactly this as well. You do something and experience the results, and learn from it. Adaptive behavior in general can be considered an example of a feedback cycle. Behavior means acting in the world. And adapting means processing the results and changing your behavior accordingly. And most generally, Schank’s definition is clearly an ability gained through feedback cycles, getting better over time based on evaluation of the results of one’s actions in the world. And Schank’s general definition, getting better over time, is clearly something that can happen as a result of participation in a feedback cycle. We find that nearly all of HCI can be interpreted in some ways as an application of feedback cycles, whether between a person and a task, a person and an interface, or systems comprised of multiple people and multiple interfaces.
Let’s take a simple example of the gulf of execution. I’m making my lunch, I have my bowl of chili in the microwave. My goal is simple, I want to heat it up. How hard is that? Well, typically when I’ve been cooking in the past, cooking is defined in terms of the amount of time it takes. So, in the context of this system, I specify my intent as to microwave it for one minute. Now what are the actions necessary to do so? I press Time Cook to enter the time-cooking mode, I enter the time, one minute, and I press Start. I didn’t press Start just now, but I would press Start. I specified my intent, microwave for one minute. I specified my actions, pressing the right sequence of buttons, and I executed those actions. Could we make this better? There were a lot of button presses to microwave for just one minute. If we think that’s a common behavior, we might be able to make it simpler. Instead of pressing Time Cook one, zero, zero and Start, I might just press one and wait. Watch. [NOISE] So I’ve narrowed the gulf of execution by shrinking the number of actions required, but I may have enlarged it by making it more difficult to identify the actions required. When I look at the microwave, Time Cook gives me an idea of what that button does. So if I’m a novice at this, I can discover how to accomplish my goal. That’s good for the gulf of execution. It’s easier to look at the button and figure out what to do than to have to go look, read a manual, or anything like that and find out on your own. But once you know that all you have to do is press one, that’s much easier to execute. That’s something nice about this interface, it caters to both novices and experts, there’s a hard and discoverable way and a short and visible way. But let’s rewind all the way back to the goal I set up initially, my goal was to heat up my chili. I specified my intent in terms of the system as microwaving it for one minute. But was that the right thing to do? After one minute, my chili might not be hot enough, this microwave actually has an automatic reheat function that senses the food’s temperature and stops when the time seems right. So the best bridge over the gulf of execution might also involve helping me reframe my intention. Instead of going to microwave for one minute, it might encourage me to reframe this as simply heating until ready and letting the microwave do the rest.
Here are five quick tips for bridging gulfs of execution. Number one, make functions discoverable. Imagine a user is sitting in front of your interface for the very first time. How would they know what they can do? Do they have to read the documentation, take a class? Ideally, the functions of the interface will be discoverable. Meaning that they can find them clearly labeled within the interface. Number two, let the user mess around. You want your user to poke around and discover things, make them feel safe in doing so. Don’t include any actions that can’t be undone. Avoid any buttons that can irreversibly ruin their document or set up. That way the user will feel safe discovering things in your interface. Number three, be consistent with other tools. We all want to try new things and innovate, but we can bridge gulf of execution nicely, by adopting the same standards that many other tools use. Use ctrl c for copy and ctrl v for paste. Use a diskette icon for save even though no one actually uses floppy disks anymore. This makes it easy for users to figure out what to do in your interface. Number four, know your user. The Gulf of Execution has a number of components: Identifying intentions, identifying the actions to take, and taking the actions. For novice users, identifying their intentions and actions are most valuable. So, making commands discoverable through things like menus is preferable. For experts though, actually doing the action is more valuable. That’s why many experts prefer the command line, although it lacks many usability principles targeted at novices, it’s very efficient. Number five, feedforward. We’ve talked about feedback, which is a response to something that the user did. Feedforward is more like feedback on what the user might want to do. It helps the user predict what the result of an action will be. For example, when you pull down the Facebook news feed on your phone, it starts to show the little refresh icon. If you don’t finish pulling down, it doesn’t refresh. That’s feedforward. Information on what will happen if you keep doing what you’re doing. Many of these tips are derived from some of the fundamental principles of design, pioneered by people like Don Norman and Jacob Nielsen, and we’ll cover them more in another lesson.
Let’s take a thermostat, for example. I have a goal to make the room warmer, so I do something to my thermostat with the intention of making the room warmer. What does the system do as a result? Well, it turns the heat on, that would be the successful result of my action. But how do I know that the heat was turned on? Well, maybe I can hear it, I might hear it click on. But that’s a one time kind of thing and it might be quiet. And if I’m mishearing it, I have no way of double checking it. So I’m not sure if I heard it, and I have to go find a vent and put my hand on it and try to feel the heat coming out. And there’s more going on in a heater, it might have worked, but the heater doesn’t immediately turn on for one reason or the other. These are signs of a large gulf of evaluation. Neither the sound or the vent are optimal displays because they’re either hard to reach or possible to miss. Feeling the heat might be easy to interpret, but hearing the heater turn on might not. So either way, I have to do a lot to evaluate whether or not my action was successful. And this is all for a very small piece of feedback. Ideally if I wasn’t successful, we want the system to also tell me why I wasn’t successful so I can evaluate what I did wrong and respond accordingly. There’s a very large gulf of evaluation if there’s no indicator on the actual thermostat. So how can we resolve that? Well, simple. We just mark on the thermostat that the heat is on. That sounds trivial, but nothing in the fundamental design of this system demanded a note like this. It’s only in thinking about the system from the perspective of the user that we find that need. I can let you know as well, this system still isn’t very ideal. For various reasons, it’ll turn the heater on or the air conditioning off even when it hasn’t reached the temperature I put in. And it gives me no indication of why. I can look at the system and evaluate that the temperature is set to lower than the current temperature in the room. But at the same time, I can see that the heater isn’t on. Under those circumstances, I have no way of knowing if the heater’s malfunctioning, if the switch is wrong, or I don’t even know. In this case, it might just be that it’s set to the wrong mode. The mode is visible, but after I remembered to check it, it appears to be malfunctioning. We can imagine an alternative message on the screen indicating the direction of the relationship or something similar that would give some sign that it’s currently set incorrectly.
Here are five quick tips for bridging gulfs of evaluation. Number one; give feedback constantly. Don’t automatically wait for whatever the user did to be processed in the system before giving feedback. Give them feedback that the input was received. Give them feedback on what input was received. Help the user understand where the system is in executing their action by giving feedback at every step of the process. Number two; give feedback immediately. Let the user know they’ve been heard even when you’re not ready to give them a four response yet. If they tap an icon to open an app, there should be immediate feedback just on that tap. That way even if the app takes a while to open, the user knows that the phone recognize their input. That’s why icons briefly gray out when you tap them on your phone. Number three; match the feedback to the action. It might seem like this amount of constant immediate feedback would get annoying and if executed poorly, it really would. Subtle actions should have subtle feedback. Significant actions should have significant feedback. Number four; vary your feedback. It’s often tempting to view our designs as existing solely on a screen, and so we want to give the feedback on the screen. But the screen is where the interaction is taking place, so visual feedback can actually get in the way. Think about how auditory or haptic feedback can be used instead of relying just on visual feedback. Number five; leverage direct manipulation. We talked about this a lot more, but whenever possible, let the user feel like they’re directly manipulating things in the system. Things like dragging stuff around or pulling something to make it larger or smaller, are very intuitive actions because they feel like they’re interacting directly with the content. Use that. Again, we talk far more about this in another lesson, but it’s worth mentioning here as well. By loading these things into your short-term memory several times, we hope to help solidify them in your long-term memory. That relationship is actually something we also talk about elsewhere in this unit.
In Don Norman’s Design of Everyday Things, he provides a different way of looking at the same information. This diagram puts a greater emphasis on what the user is doing at each of these stages. The user is setting a goal, and then planning, and specifying, and performing, and then perceiving, and interpreting, and comparing. This diagram also changes our terminology a bit and with good reason. We see a bridge of execution and a bridge of evaluation. The reason for that is these behaviors specifically bridge the gulf between the goal and the world. They’re what the user has to do to make the world match their goal, and then confirm that the world now does match their goal. Norman uses these stages to introduce seven questions that we should ask ourselves when we’re designing interfaces. First, how easily can one determine the function of the device? In other words, how easily can one tell that this device can accomplish that goal? Second, how easily can one tell what actions are possible to do with the device? Third, how easily can the user determine the mapping from their intent to the actual movements or actions they need to take with the device? So, in planning to figuring out what they can do with specifying to figuring out what they should do. Then finally, how easily can the user actually perform the physical movements associated with that plan? After they’ve done these three stages, something has happened in the world, and they’re ready to see if they can perceive what has changed. So, then we ask, how easily can the user perceive or tell what state the system is in? This is the raw information that they’re getting out of the system. Then how easily can they tell if the system is in the desired state, or how easily can they interpret what they perceived? Then finally, how easily can the user determine the mapping from state to interpretation? In other words, how easily can the user compare what they interpreted as happening to what they wanted to happen? You’ll notice that these match our stages from earlier. Norman also makes it even more clear by describing this specifically in terms of what the user is thinking. The user starts by thinking, “What do I want to do?” Then they consider, “What are the alternatives for accomplishing it?” Then they specify, “What can I do to actually perform one of those alternatives?” Then they ask, “How do I do that?” Once they’ve done it, they ask, “What happened? What does that mean? Is that okay?” I like this phrasing because it makes the distinction between interpreting and comparing a little easier. Interpreting is just about understanding what happened. Comparing involves going back to the original goal that the user had. Norman also further articulates this by breaking the process into phases that span both execution and evaluation. Closest to the world, he has the visceral phase which is the actual physical activity or the raw perceptions. Above that, he has the behavioral layer, which is where we’re actually specifying what behaviors to do or interpreting the outcome of those behaviors. Then at the top level, there’s a reflective phase, which is thinking about the problem and planning our solution, or comparing the results to our original goal. If we change his vocabulary a little bit, we might describe this lowest level as raw reaction, the middle level as deliberation, and the top level as metacognition, thinking about the goal, thinking about the problem solving process, thinking about what we did and what happened. If we rewrite these phases with this vocabulary, we start to get at a diagram you might recognize if you’ve taken knowledge-based AI. If you haven’t, here’s a little teaser for what you’ll see if you do.
[MUSIC] Good design. A phone that quietly clicks every time a letter is successfully pressed to let you know that the press has been received. Bad design. A phone that loudly shouts every letter you type. » P. I. C. Remember small actions get small feedback. The only time you might want your device to yell a confirmation at you is, if you’d just ordered a nuclear launch or something.
Let’s pause for a second, and reflect on the roles of gulfs of execution and gulfs of evaluation in our own lives. So try to think of a time when you’ve encountered a wide gulf of execution, and a wide gulf of evaluation. This doesn’t have to be a computer, it could be any interface. In other words, what was a time when you were interacting with an interface, but couldn’t think of how to accomplish what you wanted to accomplish? What was a time when you were interacting with an interface and couldn’t tell if you’d accomplished what you wanted to accomplish?
It’s not a coincidence that I’m filming this in my basement. This actually happened to me a few weeks ago. The circuit to our basement was tripped, which is where we keep our modem, so our internet was out. Now this is a brand new house and it was the first time we tripped a breaker, so I pulled out my flashlight and I opened the panel. And none of the labels over here clearly corresponded to the breaker I was looking for over here. I ended up trying every single one of them and still it didn’t work. I shut off everything in the house. Why didn’t it work? In reality, there was a reset button on the outlet itself that had to be pressed. The only reason we noticed it was because my wife noticed something out of the corner of her eye turning on and off as I switched these. That was a terribly large gulf of execution. I knew what I wanted to accomplish, I could translate it into the system’s terms easily, reset a breaker. But figuring out the actions to accomplish that goal was very difficult. That’s a large gulf of execution. How was that? » [SOUND] What? Sorry, I wasn’t paying attention. » You weren’t watching? [LAUGH] So I have no way of knowing if that was good or not? Isn’t that a terrible gulf of evaluation? I joke, but a lack of feedback on your performance at a task, whether it be filming, like I’m doing now or doing a project like you’ll do later in our material, presents the same kind of poor gulf of evaluation.
HCI, but nowadays this is basically a computer on wheels. So let’s talk a little bit about how feedback cycles apply here. Let’s start with the ignition. The button that I start my car is right here. Why is it located there?
Before cars had push button starts, this is where you inserted the key to turn on the ignition. Why? I have no idea. But I do know that now, the start button can be placed in any number of different locations. So why do we put it where we’ve always put it? Well, the reason is, that’s where the driver expects it to be placed. We help them across the gulf of execution by designing a system that’s consistent with their expectations about how it should work. It makes it easier for them to translate their intentions into actions. Now, other times we might violate this principle because of some other benefits we hope to gain. But generally speaking, when all else is equal, we want to stay consistent with the way users expect our systems to work.
So we know where the ignition button is. Let’s press it. [NOISE] Do you think the car turned on?
Well, what do we know? We know the car was off. We know this is clearly the power button based on how it’s labeled and where it’s located. And most importantly, when I pressed it we heard kind of a happy confirmation-y sound. So did the car turn on? Actually, it didn’t. To turn this car on you have press the brake petal while pressing the on button. The car doesn’t do a great job of helping us across that goal of execution. There’s no indicator that you’re doing it wrong until you’ve actually already done it wrong. But the car does give us a short gulf of evaluation. If you do it incorrectly, an alert pops up on the dashboard letting you know you need to press the brake pedal and then press the on button. The output presented is easy to interpret. As presented in the context of when you need to know that information, so you kind of understand that it’s a response to what you just did. So here we have some trouble with the gulf of execution but the gulf of evaluation is still pretty short. So now that I see this message I press down the brake pedal, press the on button [SOUND] and now the car is on.
So now that we’ve seen the way this feedback cycle currently works, let’s talk about improving it. How might we make this feedback cycle even better? How might we narrow the gulf of execution and the gulf of evaluation?
So here are a few ideas that I had. We know that the screen can show an alert when you try to turn the car on without pressing the brake pedal down. Why not show that alert as soon as the driver gets in the car every time? That doesn’t widen the gulf of execution for an expert user, but it does narrow it for a novice user, because even a novice can see that alert the first time they get in the car. But what still throws me off to this day is the sound the car makes when you try and turn it on. Watch. [SOUND] Did the car turn on? No. It didn’t that time. [SOUND] Now it turned on. So it plays the same sound initially when you press the button and then plays a different follow up sound to confirm that the car actually turned on. I know why they do this, that one sound just confirms that you pressed the button successfully while the other sound confirms that the car turned on. But for me, I would just as soon have two different sounds confirm whether or not you just pressed the button or whether the car turned on. That way, just the presence of a sound confirms the fact that the button was pressed and the nature of the sound confirms the effect of that press.
I asked you earlier to pick an area of HCI in which you’re interested and reflect on it as you go through this course. Depending on the area you selected, feedback cycles can play a huge number of different roles. In health care, for example, feedback cycles are critical to helping patients manage their symptoms. That relies on the results of certain tests being easy to interpret and evaluate. Feedback cycles are also present in some of the bigger challenges for gesture-based interactions. It can be difficult to get feedback on how system interpreted a certain gesture and why it interpreted it that way. Compare that to touch, where it’s generally very easy to understand where you touched the screen. So, think for a moment about how feedback cycles affect the area you chose to keep in mind.
Lately I’ve encountered another interesting example of feedback cycles in action. You may have actually seen this before as well. They’re the new credit card readers. My wife sells arts and crafts at local events, and so she has these Square readers that can scan credit cards on her phone. One version lets you swipe, and the new version lets you insert the card. So let’s check this out real quick. With the swipe version you just insert the card and pull it through, just like a traditional card reader. The problem is there’s typically no feedback on whether you’re swiping correctly. And what’s more is you can be wrong in both directions. You can be both too fast or too slow. So you may have had a time when you were trying to swipe a credit card on some kind of reader, and you kept doing it more and more slowly and deliberately, thinking that the problem was that you had done it too fast originally. And then you discover that you’ve actually been going too slowly all along and your slowing down was actually counterproductive. There’s no feedback here and the space and acceptable input is bounded on both sides. You have to go above one speed and below another speed. But now credit card readers are moving to this model where you just insert the card. You try, at least. In terms of feedback cycles, in what ways is this actually better?
First, in terms of the gulf of execution, the insertion method is actually physically easier to do. While you can be both too fast and too slow with the sliding method, you can’t push it too far in with the insertion method. So you know if there’s an error, it’s because the card isn’t far enough into the reader. And second, there’s rich feedback with the insertion method. It doesn’t even have to come from the screen telling you that you didn’t do it correctly. You feel the card stop when it’s far enough into the reader. You have immediate physical feedback on whether you’re putting it in the right place, and whether you’ve actually put it far enough in, rather than delayed feedback asking you to try again after some kind of waiting period.
So, using the insertion method is significantly easier. However, the insertion method introduces a new problem. With the sliding method, I never had to actually physically let go of my card, so there was little chance of me walking away without it. With the insertion method, I insert the card and I wait. I’m not used to having to remember to retrieve my card from the card reader. Now this isn’t quite as big a deal with these new portable readers, but for the mounted ones you see in stores it can be far more problematic. So how can we build some feedback into the system to make sure people remember their cards when they walk away?
There are a few things we could do here. We might build some kind of buzzer into the card reader to let the customer know when they can take their card out. That would make sure that they don’t leave without it. ATM machines often do this, actually. They’ll ring a buzzer until the card and the cash are removed. But that’s noisy and potentially irritating. It would mess with the ambiance of a restaurant or something like that. We could do something super complicated, like pair the credit card with a smartphone and ring the phone when it gets too far away from the credit card. But that requires adding some new technology to every single credit card, which could be a pretty big expense. So what about something simpler? Why not force a customer to remove the credit card in order to get the receipt and their goods? Unless they’re going to walk away without what they came to buy, that’ll ensure that they remember their card.
Now notice one last thing about this example. We’ve been discussing how to make the process of sliding or swiping a credit card easier. What’s wrong with that question?
The problem is that we’re not focused on the right task. Our task shouldn’t be to swipe a credit card, or insert a credit card, or anything like that. Our task should be how to most easily pay for purchases. And possibly the easiest way to do that would be to design a system that lets you just tap your phone against the reader, this reader actually does that. That way, we can use the thing that people have on them at all times. Now maybe that’s isn’t the best option for various other reasons, but the important thing is we need to focus on what we’re really trying to accomplish. Not just how we’ve done it in the past. We can make incremental improvements just sliding or swiping or inserting a credit card all we want. But we should always keep our eyes on the underlying task that the user needs to accomplish.
Today, we’ve talked about, arguably, the most fundamental concept of human-computer interaction, feedback cycles. We describe feedback cycles for our purposes as the exchange of input and output between a user and a system to accomplish some goal. We discussed feedback cycles’ incredible ubiquity in other fields and discussions. We talked about gulf of execution, the distance between knowing what they want to accomplish and actually executing the steps necessary to accomplish it. We talked about gulf of evaluation, the distance between making some change in the system and evaluating whether or not the goal was accomplished. We introduced the seven questions we need to ask ourselves to bridge those gulfs. Now that we understand these goals, our next goal is to understand methods for crossing them.