Microsoft : Designing the future with the help of the past with Bill Buxton

Principal Researcher Bill Buxton

Episode 46, October 17, 2018

The ancient Chinese philosopher Confucius famously exhorted his pupils to study the past if they would divine the future. In 2018, we get the same advice from a decidedly more modern, but equally philosophical Bill Buxton, Principal Researcher in the HCI group at Microsoft Research. In addition to his pioneering work in computer science and design, Bill Buxton has spent the past several decades amassing a collection of more than a thousand artifacts that chronicle the history of human computer interaction for the very purpose of informing the future of human computer interaction.

Today, in a wide-ranging interview, Bill Buxton explains why Marcel Proust and TS Eliot can be instructive for computer scientists, why the long nose of innovation is essential to success in technology design, why problem-setting is more important than problem-solving, and why we must remember, as we design our technologies, that every technological decision we make is an ethical decision as well.

Related:

Transcript

Bill Buxton: If you are going to come and make an argument that something is going to have huge impact in the next five years, if you haven't got fifteen years of history of that idea and can trace its evolution and history and so on, then you are probably wrong or you haven't done your homework or you might get your head cut off when you come to this presentation unprepared. Even if you are right, and you don't have that fifteen years, then that's gambling, that's not investment, that's not research. You are just lucky. Design is a repeatable profession.

Host: You're listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I'm your host, Gretchen Huizinga.

Host: The ancient Chinese philosopher Confucius famously exhorted his pupils to study the past if they would divine the future. In 2018, we get the same advice from a decidedly more modern, but equally philosophical Bill Buxton, Principal Researcher in the HCI group at Microsoft Research. In addition to his pioneering work in computer science and design, Bill Buxton has spent the past several decades amassing a collection of more than a thousand artifacts that chronicle the history of human computer interaction for the very purpose of informing the future of human computer interaction.

Today, in a wide-ranging interview, Bill Buxton explains why Marcel Proust and TS Eliot can be instructive for computer scientists, why the long nose of innovation is essential to success in technology design, why problem-setting is more important than problem-solving, and why we must remember, as we design our technologies, that every technological decision we make is an ethical decision as well. That and much more on this episode of the Microsoft Research Podcast.

Host: Bill Buxton, welcome to the podcast.

Bill Buxton: Glad to be here.

Host: So, I'd like to start by asking my guests what gets you up in the morning, but you've already answered that in print, and I quote, 'What gets me up in the morning is to realize what I dream about.' So, now you have to tell us what you dream about.

Bill Buxton: It depends which morning it is, I think. I think there's an embarrassment of riches of things to want to do, and I think that that's one of the best things because you're never at a loss to be motivated. But then the other problem is, you have to make choices as to which one you pursue. You can do anything and everything in your life, you just can't do them all at once. You always want to be falling in love with something that just captured your imagination, but in so doing, you have to retire a previous passion or at least move it to the background because you can't go full-throttle into more than one or two things. One description of what I do for a living is Experience Design. And I'm prone to say Jimmy Hendrix had the greatest wisdom of this, and that's the most profound question, 'Are you experienced?' And if you don't have a breadth, as well as depth, of experience to draw on, how can you be good at Experience Design? Because it's building up this repertoire and curating this repertoire of experiences in your life across the board that is the treasure trove that you can mine in whatever you're trying to do.

Host: Your bio says you are a relentless advocate for innovation, design and the appropriate consideration of human values, capacity and culture in the conception, implementation and use of new products and technologies.' Which is a mouthful. But let's unpack that a little bit. I'm really intrigued by your statement of the 'appropriate consideration.' Tell us what you mean by that in the context of designing new technologies and products.

Bill Buxton: Well, one of my heroes is a historian of technology named Melvin Kranzberg, and he has some laws. But his first law is, 'Technology is not good, it's not bad, but nor is it neutral.' It will be some combination of the two. As soon as you say words like good and bad, that implies you have a moral compass. And the real question is, is that when you are making technological decisions and launching technologies into society, you are, in fact, making an ethical choice, whether you know it or not. And so, maybe you'll do a better job of it and weight more heavily on the positive if you actually know what that moral compass is and that you are, in fact, making an ethical decision. I'm not trying to put too heavy a weight on this in that you are playing God, but you are in fact having impact. But you are also human, so how can you just do the best? You will get some stuff wrong. So, take responsibility to clean up the mess without throwing the baby out with the bath water. And so, it basically says that 'appropriateness' is appropriate to the moral order of place or where it's going to be placed. That's the closest way I can put it.

Host: Let's talk about your job description at Microsoft Research. When you started at MSR, Rick Rashid hired you to, as you say, 'Make design a core part of the Microsoft culture.' So, how did you go about doing what he said? That's about the vaguest job description I can think of, and yet it… it's perfect.

Bill Buxton: Well, actually what he really said was, 'Make a difference and if you are not the best person to figure out how you should do that, you are probably shouldn't have the job.' Then my response was okay, I'm going to try to help contribute to bringing a greater awareness of design to the company and that meant, actually, not trying to design products, but trying to design a culture and change the attitudes and not elevate design to the point where everything is design-led, but where it's an equal partner at the table. In the early days, when I would speak to different teams in the company, in large or small groups, it would be kind of like, don't expect this to come from above, or from management or anything like that, because we are our own culture. We make it. And it's every individual. And if you can actually start to just feel empowered to, within your own, even if it's one other person, you can start to make adjustments along the way you want that can go viral because if we're shifting in a good direction, it will be noticed, and then people will say, well, what's the secret sauce that you're using? And nobody can own this. It can't be about any individual. It's got to be about empowering individuals to form groups and clusters because that's what culture is. It's a mutually agreed upon set of values.

Host: So today, like you, I'm going to use some literary quotes to prompt our discussion on technology research. So, let's start with Marcel Proust. He once said, 'The real voyage of discovery is not in seeking new landscapes, but in having new eyes.' One of the major themes in research is looking for the next big discovery, right?

Bill Buxton: Yes.

Host: How does having new eyes, or different optics, as you've said it, inform the quest for innovation or how should it inform the quest for innovation?

Bill Buxton: So, the net result is that, in some sense, I would describe my job description as being an optician and to find the right lenses. I'll give an example. As you say, the industry is heavily driven by people trying to find the next big thing, whether it's a new gadget or a new application, killer app or a new service. And if you're just graduating from university or design school or whatever, that of course, you want to become a millionaire by the time you're 24 or you're a failure. And so, there's all these pressures. And so, my automatic reaction, I just wrote a two-pager that said the next big thing isn't a thing. And, I said, it's actually a change in relationship amongst the things that are already there and the things that are going to emerge. And when I say relationship amongst those things, it's about the social relationships, things like kinship, introduction, negotiation, approach, departure, all of these things, the moral order. These are all terms that we know about the society of people. But, we aren't used to speaking about in terms of the society of technology. What could you do that would have more impact than if things just worked? If things just worked together seamlessly? And if, in working together, every new thing I added came at a great value in and of itself, but it also added value to everything else I already had, and they to it, and furthermore every new thing I added reduced the complexity, not only of that new thing, but reduced the complexity of everything else in the ecosystem and they it. We realize that hardly anything works well together much less seamlessly. And what we've forgotten, when we come back to the human side, is that the better we get at really making desirable, beautiful, affordable, useful, valuable devices, the worse we're making things. The cumulative complexity of a bunch of desirable, simple, affordable, valuable things is way above the human's capacity to deal with. And that's why you must reduce complexity with everything you add. And that takes a very different approach because it forces you into thinking about an ecosystem. Albert Shum who is part of the 'Canadian Mafia' trying to change design at Microsoft here is a good friend and a fellow cyclist. And he has a nice way of saying it, that in the industry, we spend a whole bunch of time learning how to design houses. The real challenge is building the communities and the city planning and the urban planning and the flow of things. And I think even the changes we've been making over the last year or two have been significant steps on this path. But the challenge in innovation is, how do you go beyond that and say what are the right metrics for our aspirations and where we can be and how soon we should get there? Because only when you find that, can you set appropriate goals that most meet your objectives.

(music plays)

Host: You have become the collector and curator of more than a thousand computer hardware artifacts that chronicle the history of various aspects of human computer interaction. So, tell us about your collection, or collections. How did you get started doing this, what kinds of things have you collected and how hard was it for you to get your hands on some of these things? I've seen the collection. It's crazy!

Bill Buxton: Well, first of all, my name is Bill and I'm not a hoarder. I'm a collector. The one-word answer, it was an accident. Maybe a more informative answer is to say it is a reflection of my process of what I do for a living. I'm always looking for reference material, always scanning, collecting things around, surrounding yourself with them for inspiration for ideas, and to trigger thoughts, and having them sitting there around you, and all of a sudden, some new relationship pops out. When I'm at a loss for a solution to a problem, I go and surround myself with these objects. But over about forty, forty-five years, I've never thrown any of them out. I've kept them all. And so, when anything came out, whether it was a brochure or an article in a magazine, or something like that, I kept it and documented it for future reference, for teaching, for teaching myself and to go back to say, hey, I think I've seen this before. And you can think of them all as prototypes, and really expensive-to-make prototypes, which I could get for practically nothing, sometimes like on eBay, where it's like a really expensive education where somebody else paid the tuition. And they're sitting there, if you want get the benefit of that education, you can. And therefore, when I do start to make something or when anybody in the company does, they can start at a much higher level because they've got these reference objects.

Host: Interesting.

Bill Buxton: And so, the base point-of-departure for any problem I'm looking at is, somebody has already solved this problem and there's something out there that's already done this. So, I'm going prospecting before I go building.

Host: Tell us about the collection. What's in it?

Bill Buxton: Well, the collection is sort of a cross-section of all of the input devices through which people have interacted with digital technology pretty much from the beginning. And so that would include mice and joy sticks and trackballs and trackpads. It is PDAs. It's game controllers, it's foot pedals, head displays. It's uh, smart watches going back to 1978. It's the world's first smart phone. It's the history of portable music players. It's the history of AR and VR technologies going back to a reproduction I made of the very first Stereo Viewer from 1838. And it's also examples to use to serve as the basis for story-telling that illustrate some of the things that are really important about design. I don't think many people in VR know that it is due to virtual reality, in an early form, that led to Yellowstone being made the first national park in the world, not just the United States. Or that the very first stereoscope from 1838 was already looking into a virtual space because photography wasn't invented till the following year. There were no photographs to make stereo images from and they had to be hand-drawn and so when you looked into Wheatstone's original reflective stereo craft, you're looking into hand-drawn lines into a world that never existed.

Host: Wow.

Bill Buxton: I think those things are really interesting because you start to see patterns, if you go through it. But from those patterns, you say, okay, they probably haven't stopped, and so you can extrapolate. So, it's really hard to extrapolate from a point. If I have a line, it's much easier. And so, I have this game, I'll do it with adults as well as children. I'll draw all these different lines and say, 'Continue these lines.' And then I'll put a point. And they have no idea what to do with the point, but all those other things, they can continue because they can see the pattern as things were going. And it doesn't mean the extrapolation is correct, but it gives you your initial bearing for your exploration and usually because there's other things involved, there's probably a couple of lines that come and you'll start, maybe you'll see there's intersections from extrapolations. And you have these ways to visualize. And this gives you a different way to think, accompanied by concrete examples that you can experience to get to the assets at the finest granularity.

Host: So, you referred to something you called the long nose of innovation. I think researchers are familiar with the phrase the long tail. But the long nose is an interesting one. And it's in context of new technologies and how long it takes them to catch on. And you also had said at some point, that our slow rate of progress is maybe not necessarily due to a lack of technology but lack of imagination. How and why do we lack imagination and what can we do? What can researchers do about that?

Bill Buxton: The long nose basically comes back as sort of saying if we look historically at the evolution of technologies, it takes at least twenty years from the first clear articulation of something to the point that it's mature, where let's measure maturity as it's a billion-dollar industry. If you are going to come and make an argument that something is going to have huge impact in the next five years, if you haven't got fifteen years of history of that idea and can trace its evolution and history and so on, then you are probably wrong or you haven't done your homework or you might get your head cut off when you come to this presentation unprepared. Even if you are right, and you don't have that fifteen years, then that's gambling, that's not investment, that's not research. You're just lucky. Design is a repeatable profession. It's not, I get lucky once in a while. And so, if you want to study design and innovation, study the repeat offenders, the ones that can do it over and over. You don't have to wait for the muse to come and drive you. And that's what you learn. And you can only do that if you have process. And the long nose is a key part of that process. Now, for those who doubt, the mouse, which everybody who saw one in 1968, knew it was the right thing. But it wasn't until Windows 95 before everybody had a mouse at their desk. Now, why did it take so long? I first used a mouse in 1971. Now the thing is, you need a perfect wave of things. You had to perfect Windows icons. You had to train the developers how to write this type of graphic user interface. That was a whole new thing from DOS or UNIX. And you needed the processors. You needed graphics processors. You needed the displays to switch to bitmap displays rather than calligraphic displays which dominated back in the time, basically glorified oscilloscopes. Every technology goes the same route. And so, the long nose is basically this reminder of how long it takes. So, it also says the following things and reinforces what I was saying about the combinations about innovation being the aggregation of existing ideas: that everybody thinks that things are moving really quickly and that is not true. We mistake a whole bunch of things moving really slowly, with things moving quickly. It's the difference between amperage and voltage. Any single technology is evolving, statistically speaking, really slowly. But, when you have a number of different things moving slowly, at slightly different paces, but simultaneously and at different stages on the nose, if you start to realize that's what's going on in the overall technological ecosystem, you can see those patterns and then project forward because you can extrapolate from history, and say, here's where you hit the inflection point and that's when things are going to happen. Everything has a perfect storm, and there's methods by using this technique to actually predict when that perfect storm is going to happen. I'll give you a really quick example. I spent my early career, after I switched being a musician, to building digital music synthesizers for live performance. So, I saw the evolution of how digital audio emerged. I went to Silicon Graphics and became Chief Scientist there doing animation systems. But the only act of genius I had, because I wasn't in computer graphics, I was literate, but I wasn't, you know, a specialist in computer graphics. But, I knew that computer graphics was going to follow exactly the same pattern as computer music, but it was multiple orders of magnitude more complex, so it was just shifted further along the timeline. And so, all the planning over the eight and a half years I was there, we kept hitting that right. And the reason we could know exactly what to do and when was because I just was repeating what I had already done in music. And so, all I needed to do was to see that relationship. And I think overall, that type of pattern happens throughout, but you have to know those other areas where you go prospecting. So, the long nose, the notion of history, collecting, sampling and not just going immediately to building. We spend far too much and go far too quickly into problem-solving and don't spend enough time problem-setting. And that's the ultimate skill.

Host: Can you define problem-setting a little more clearly?

Bill Buxton: Problem-setting is basically, it's not enough to get the design right, you've got to design the right thing. And so, if you just leap in and start building something where you've got a solution, you have no idea if that's the best option. There might have been a better way and you didn't take time because you are already behind schedule. But here's the crazy thing. At the beginning of the product cycle, you have a small team just getting going. Your burn rate, in terms of what it's costing you per week in terms of the project and that, is very, very low. So, what you then should be doing is thoroughly exploring a range of different alternatives. Problem-setting, part of that process is this notion of, you cannot give me one idea. You have to learn how to work quickly and give me multiples. That's a technique for this whole issue of, how do you deal with the problem-setting? And by exploring the space first… oh, that's the real problem… Put it this way. You have a bunch of people that talk about user-centered design. And they'll say, you know, go talk to your users and they will tell you what to do. Okay. Would you go to a doctor where you walked in, and the doctor said, okay what's wrong with you, what operation do you need and what drugs should I give you under what dose, right? And that's how some people naively interpret user-centered design, is 'listen to users.' And, no. I'm going to ask you all kinds of questions. But I'm going to take all of those as part of the information that helps me make a diagnosis. And so, where do we collect the symptoms to find out where the real problems are? You're telling me this. I understand the situation. Now, I have to know enough about your industry to ask pertinent questions. And for me, that's what the problem-setting is. The designer, the main equipment is to have that meta-knowledge. And that's where the diverse interests come in, so how do you get that knowledge? But if you don't even know that's the kind of knowledge you need to get, you're not even going to go looking for it.

Host: So, you look at the product development cycles and, even in research, what you're talking about is something that people would have to say, 'Okay, we need to rethink how we work and what we make time for.'

Bill Buxton: So, I'd throw the argument the other way: you can't afford not to do it. So, your cost-per-month on a project, if you put an extra month up front, it costs you almost nothing. And if it comes up with a much better solution that's a fraction of the price and can get it done more quickly and have a much better margin, first of all, you've made up for the lost time by having spent that up front. But let's pretend it still takes the same amount of time. We never have time to do problem-setting and so on sufficiently. We're getting better at it. But we seem to be able to have time to be three months late where we are fully-loaded with the highest burn rate possible, right? I mean, if you're going to take an extra month, do you want to play it where it costs you the most or do you want to do it up front and you get a better product? The other part is, it's not all in one. You don't make all your decisions up front and then go build. The decisions that you make the earliest are the ones that are hardest to change later. So, that's your basic architecture. In the software industry, we don't have architects. What we call an architect, in architectural terms, is actually a structural engineer. And we have no architect that has design perspective at the very beginning. But also, there's this notion that once you've got a representation, like a rendering of what the screens are or some of these other things, that that's the end of the design. There's only two places where there's room for creativity in design. So, the first place for creativity is the heuristics process whereby you innumerate the repertoire of things from which you are going to choose, and then the second is the heuristic you use to eliminate all but one. And it's that inhale/exhale. You start with nothing, you end with one. But you have to go through that whole thing. You would love, afterwards, you know I say, I could have got here right from the beginning. And you could, but you never would have. And that's the biggest mistake. The fastest way to a mediocre product is to make a plan and stick to it.

(music plays)

Host: Let's talk about AI for a minute. Because tech companies are putting a premium on AI talent, uh…

Bill Buxton: Oh, is it important now?

Host: Apparently, people are using the terms gold rush, talent war…

Bill Buxton: Feeding frenzy…

Host: …feeding frenzy. And you've suggested that there's a risk that anyone who's not doing AI, might be marginalized.

Bill Buxton: So, I have to preface that by saying, I think what we can do today in AI is absolutely unbelievable. It's beyond my wildest expectations in what we'd be able to do at this point. It's unbelievably valuable, but it's nevertheless essential but not sufficient. And as I said, you need a perfect storm of a whole bunch of things to get a sustainable system, or an ecosystem in place. And my fear is that if you focus too much on the AI component, that you distort the other requisite skillsets and disciplines that are needed to ensure that AI is successful. Every discipline represented in our company is essential to our success but not sufficient. And the trick is to find the balance. And one of the important elements here is to make a distinction between literacy and expertise. It is essential that everybody in the company has a level of literacy about AI. But it's equally important to have literacy about every one of those disciplines. And that means that AI should be working as hard to gain literacy in the disciplines that are core to its success, as those disciplines are to AI. What happens, if we push so hard on the AI front and we don't make that clear distinction between literacy and expertise, that developers and designers are so focused on AI, that they feel that if they're not going that direction and chasing that really, really hard, that it's a career-limiting move. I think that what is clear is that you may end up with the best AI in the world and still be beaten by somebody who's got only 20% of the AI competence, but they've got way better integration of the AI into their larger ecosystem. Because, like any other technology, it's not good, it's not bad, but nor is it neutral, and it will be a positive and a negative consequence of that technological change.

Host: Another premise in AI research has its underpinnings in what we've referred to as DIKW pyramid where you start with data which supposedly gets refined into information and then to knowledge and culminates in wisdom, which is the ability to make good decisions based on the data you have. And this, of course, has literary roots in T.S. Eliot's, The Rock: 'Where is the life we've lost in living? Where is the wisdom we've lost in knowledge? Where is the knowledge we've lost in information?' Talk about this in the context of this idea that if we have enough data, with machine learning, computational power and sophisticated algorithms, we'll end up with wisdom.

Bill Buxton: Well, first of all, Eliot left off two levels there. So, where's the wisdom we've lost in knowledge, the knowledge we've lost in information, information we've lost in data and the data we've lost in noise. You have to remember noise cancellation. And people talked about a data revolution and so on… No, it's a data explosion. And information technologies? No, it's not. It's only information if it can serve as the basis for informed decision-making. I think it's very, very healthy to have that hierarchy. I think it's extremely valuable to be able to fit things into moving up that food chain. But I think that the role that intelligence plays there, and where intelligence lies, is a sticky thing. And we have to base our expectations of the technology, and therefore have our engineering guided, by a sense of what's possible at any point in time along that path. Now, I know that we were talking in AI about, you know, sensing an ecosystem environment and all this sort of stuff. Well, we have to be realistic about how much of that we can sense at what point in time, and then understand what elements are being neglected and are not simply feasible at this point to deal with and therefore our notion of intelligence is limited. And how do we, at any point in time, make sure we're back-filling those gaps until it can be proven that we've got those other parts reliably taken care of. And again, by looking at the disciplines, doing the analysis, we can look at the timeline and take appropriate action for each thing to make sure that we've got the bases covered with the appropriate technologies for that moment in history and not make colossal mistakes and confuse the target with where we are right now. It comes right back to what I said earlier: it's not just being able to get the vision, it's how do I get there from here?

Host: What would you say to the people that are moving into this arena right now? What should they be thinking? What could their next steps be?

Bill Buxton: In a way, my advice is less concrete in terms of 'learn this, learn that' in terms of some skill. We've said already that the problems we face today require depth. You have to be really good at what you are doing if you want to really have influence. And for me, the only way you can get really, really, really good at something is if you're just so passionately in love with it that it's not work. Now, people say okay, you got to find your passion. Well, the problem is how do you do that? Get into the traffic, because if it's not hitting you wherever you are, then move. But the other part is, by trusting my nose, the stuff that caught my fancy in chasing those things that made no sense, but, in retrospect, were the perfect career moves. Like why would anybody go to university and do computer music when nobody even knew what a computer was? And spend four years doing that? But it was the most brilliant career decision that I never made. It wasn't a career decision. I wanted to be a musician. But I would say, always be bad at something you love. And it doesn't matter if things make sense. That's the other part that's really critical. I purposely rejected any career path for which there was a brochure in the guidance counselor's office in high school. Because it's already full. There's going to be already too many people doing that. And it's not that I'm not competitive, it's just that my main competitive advantage is, I'm not trying to compete in the same race. And if you've got these interests and you become uniquely qualified, you can have the satisfaction you're the best in the world at what you do. You're just the only one. That makes you also the worst. That keeps hubris from taking over. But have the faith that at some point in your life, all that work will be recognized and somebody will need it. There's somebody in the world who needs it. And the question is now to find it. For me, it took me till I was forty. But the time leading up to that was so full of rich experience that it never occurred to me that I wasn't making any money. I was the richest person in the world because I was doing what I love doing.

Host: Bill Buxton, thank you for joining us today.

Bill Buxton: Thank you for having me.

(music plays)

To learn more about Bill Buxton and the latest innovations in human computer interaction, visit Microsoft.com/research