Category Archives: UX Philosophy

More on “Usability Testing” vs “User Testing”

I don’t know how we got there, but use of “user testing” instead of “usability testing” is pretty widespread and I don’t understand it at all. We’re not testing users. We’re testing the usability of the design.

Language matters. We construct reality out of language. Language shapes how we see the world—it limits our perceptions, it tells us what kinds of things exist or don’t exist in the world. The power to name a thing lets us shape what that thing is for and who can use it.

So what we call usability testing matters. However, nothing convinces others like experts, so please do check out this blog post by Dana Chisnell from 2008:

Are you doing “user testing” or “usability testing”?

“User Testing is a Qualitative Activity”: A Common Mistake

Every now and then, I hear people say that “user testing” is a method that only generates qualitative data. There are two problems with this statement.

First, we are not testing users. We are testing the usability of the product in question. It is usability testing, not user testing. In an age where “user experience” has superseded “usability,” we may want an even better name for this method, but it is most definitely not “user testing.”

Second, usability testing is a perfectly good way to gather quantitative data. It just so happens to also be a very good way to get qualitative data.

When you conduct a usability test, you can quantify the number of errors that happen during tasks (and categorize them for even cooler measurements), completion time, whether or not participants completed the test, and you can collect task-level and test-level survey data.

Usability testing generally takes place with small sample sizes (on the order of 6 people or so), and this isn’t a barrier to descriptive statistics. You just have some big confidence intervals around your averages, but you can just acknowledge that in your reporting and decision making and get on with it.

I do understand thinking that quantitative data gathered from usability testing isn’t so helpful if you’re taking some kind of lean approach to product development. You’re moving quickly and treating usability testing as a more generative activity. Maybe it would be worthwhile to draw a distinction between these activities.

Will There Be UX Professionals in the Future?

I’ve been pondering lately where there will even be formal UX roles in 10, 20, or 30 years’ time.

Certainly, I don’t think the concepts of caring about users and designing based on researching those users will go away. And I certainly hope I’m not jobless and starving in a few decades.

It feels like we’re in the middle of a race to see who can own UX the most. Everyone wants to talk about the user experience (to the point where that term has even displaced the more accurate “user interface” in a lot of cases). Developers build user experiences, quality assurance people assure the quality of the user experience, product managers want to swallow the UX skillset whole and incorporate it into their own jobs.

Meanwhile, there’s a constant drumbeat of “you don’t need UX specialists.” All you need to do is read this list of 5 tips on how to conduct interviews, and you’re instantly an expert on user research! What kind of fool worries about getting bad data from bad research? Any research is, after all, better than none. (Protip: No, it isn’t)

All that said, though, I do want everyone to think about how their work affects users. Disseminating these skills is going to make life better for everyone. And when we live in a world where practically everybody has had at least a class on UX and concerned with getting real data about users… what do we need specialists for?

I suppose we’d need specialists for highly specialized problems. In those cases, though, I can’t imagine we’d see the kind of “UX Unicorn” model, where we pretend that a UX expert can also be an expert at visual design and coding. High specialized problems will probably need people that can solve tricky research problems, or who spend all their time thinking about complex design problems. How many specialists could the world possibly need, though?

I’m less worried than curious. I suspect a lot of UX people will end up migrating to product management.

Visual Analytics

The following is a letter I sent in to User Experience Magazine after a recent article on Visual Analytics. Sadly, I don’t think they publish letters to the editor anymore.

I am writing to offer feedback on Bartosz Mozyrko’s recent article, “Visual Analytics: Uncovering the Why in Your Data.”  I am pleased to see the magazine covering data analysis topics.  As the author of Practical Web Analytics for User Experience, I also have a great interest in the subject. I’d like to offer a few comments and concerns.

Mozyrko argues that visualizing web analytics data answers the question of “why” users do what they do, in a way that either plain numbers and/or page-to-page path data do not. I must disagree: Neither kind of data actually reveal the “why” of user behavior. Rather, things like heatmaps and session recordings are just representations of plain, why-less data.

Mozyrko starts by describing how traditional web analytics tools like Google Analytics are insufficient for the task of answering “why,” which is accurate. However, he then argues that visualizations help people digest large amounts of data. While true, understanding large amounts of data still doesn’t solve the problem of “why,” as Mozyrko suggests.

Later, he states that visual analytics focus on what happens within pages, whereas traditional web analytics tools focus on how people move between pages. This statement is an oversimplification. Robust tools like Google Analytics and Adobe Analytics capture a rich amount of data on in-page and between-page behavior. In fact, having these two kinds of data together in one tool offers amazing opportunities for analysis, such as the ability to segment data about in-page behavior based on something the user did elsewhere during their session.

The rest of the article discusses types of visualizations. However, click tracking heatmaps, session replays, and form analyses still do not tell you why users did what they did. Mozyrko never answers the question of how to understand the “why.” He writes “armed with your visual analytics, you now have a complete picture of how users engage with your website or mobile application.” However, the picture is not complete—a complete picture would contain the answer to “why” and Mozyrko does not discuss how to answer that question.

A complete picture of user behavior comes from triangulating different kinds of data. Web analytics data work best when combined with data from things like surveys, usability testing, interviews, field studies, and so on. Interaction with users and putting together quantitative and qualitative data sources are the ways that we get closer to answers the “why” question.

Lastly, Mozyrko has a link in his article to a case study on his company’s website. He is the CEO of a company that makes the kind of tools he discusses in this article. While it makes sense that he would be highly knowledgeable about data visualization tools, this connection along with the inaccuracies of this article make it seem more like a piece of advertising than an article that seriously engages with the question of how to better understand user behavior.

I remain committed to maintaining the highest possible ethical standards for UXPA, and felt obligated to share my concern.  Thank you for your attention, and for your efforts managing this vital publication.

Michael Beasley

Keurig: Bad, Wasteful, Unethical Design

Shortly after I started working at my current job, we replaced the old school drip coffee machines with Keurig machines. For those unfamiliar with them, Keurig makes coffee machines that let you insert a little sealed plastic cup into the machine, press a couple of buttons, and have coffee emerge. Cleanup is as simple as throwing a little gob of plastic into the trash.

The Keurig people have built the skill of making a cup of coffee into the device itself, making it easy for people to get consistent results. Not good results, mind you, but consistent results, which is something that we know people care about. Plus, you get coffee that seems fresh, made just for you! And you get to exert choice over what kind of coffee you put into the machine. You get custom made coffee and the ability to choose (within the constraints of the system, of course). Just make sure you only buy Keurig brand coffee cups. They frown upon you using off-brand coffee.

Of course, Keurig, besides making terrible coffee, is super wasteful. Granted, you waste less coffee using this system. Generally speaking, people have no idea how much coffee to use when they’re brewing it (this is the skill that Keurig builds into the artifact). But you are using a bit of plastic every time you make a cup of coffee and tossing it in the garbage afterward (these cups can’t even be recycled). Keurig coffee is not exactly an ethical choice.

Should a UX person participate in making a product like the Keurig machine? Should we make products that devour natural resources and create unnecessary waste, and terrible coffee to boot? In a perfect world: No. We don’t live in a perfect world, though, and living in society means choosing which compromises we’re going to make.

The discussion of sustainability in UX is thought-provoking but, I would say, not really relevant to most people’s lives. UX people don’t often get to participate in discussions of what to build; we’re usually brought in to help figure out how to build it. Without the ability to influence what gets built, we’re left with two choices: Participate or starve.

So while I’d love to live in a world where Keurig doesn’t exist, I wouldn’t blame any UX person that was involved in building it. If it’s any consolation, though, the design of the Keurig machine is so bad that I can only conclude that there isn’t.

What’s the deal with “user testing?”

While I’m on the subject of language that bothers me, what’s the deal with “user testing?”

People harping about the phrase “user testing” is old news. As far as I can tell, it’s something that a few people care about a lot, and that most people do not even think about. When I hear people outside the game use the phrase, I cringe a little bit; when I hear fellow user experience professionals, particularly experienced ones, use the phrase, it really bothers me.

So, the classic argument: We’re not testing the users. We’re testing the usability of the interface. Or testing whether the user experience fits with what we were designing for. Of course, no one thinks we’re actually testing users, but words matter. How we say things matter. When we have a choice between an easily used ambiguous phrase (“user testing”) and an easily used unambiguous phrase (“usability testing”), let’s go with the one that cannot be mistaken.

“User Experience” Is Not a Thing You Can See

When did people start using the phrase “user experience” when they really mean “design?” As in “I’d like you to show me the mockups of the new user experience” or “that website recently came out with a new user experience.”

I mean, seriously.

In a sense, though, I suppose that in changing the design, the experience of the user is, of course, different. But the user’s experience is a property that emerges from the user’s interaction of a bunch of things that includes the actual design of the artifact. Unless someone has gone to the trouble of not just redesigning the website, but also the user’s browser, computer, environment, senses, and brain, to name a few things, then you can’t really go see the new user experience.

And really, how can you ever “see” a user experience just by looking at a picture or by clicking around on a newly designed website.

The best I can figure is that it’s yet another case of buzzwords gone horribly wrong. There is an increasing sense that “user experience” is important—that it’s important to design for good user experiences, to have user experience professionals kicking around the office. Somehow, maybe, that got reduced to “user experience = design.”

But it’s as annoying as saying “user testing.”

Software and hardware, mind and body

When we designed software or websites for desktop computers, it was easy to think about the user interface in isolation from the hardware the user would actually use. When smart phones broke out into the scene, we were (once again) reminded that the software user interfaces we design exist in physical devices that allow us to interact with the software.

The software is the mind or spirit, and the hardware is the body. When you think about them in isolation, you miss the emergent properties of the whole. As it is with humans, so it is with our digital devices.

As a user experience designer (or whatever the job titles happens to be), it’s all too easy to focus on the spiritual side of the device. This makes sense, because for practical purposes, most of us don’t influence the hardware side. But our software or websites do exist in glass, metal, and plastic bodies and I imagine we will want to be more mindful of this in the years ahead.