Monthly Archives: April 2016

More on “Usability Testing” vs “User Testing”

I don’t know how we got there, but use of “user testing” instead of “usability testing” is pretty widespread and I don’t understand it at all. We’re not testing users. We’re testing the usability of the design.

Language matters. We construct reality out of language. Language shapes how we see the world—it limits our perceptions, it tells us what kinds of things exist or don’t exist in the world. The power to name a thing lets us shape what that thing is for and who can use it.

So what we call usability testing matters. However, nothing convinces others like experts, so please do check out this blog post by Dana Chisnell from 2008:

Are you doing “user testing” or “usability testing”?

“User Testing is a Qualitative Activity”: A Common Mistake

Every now and then, I hear people say that “user testing” is a method that only generates qualitative data. There are two problems with this statement.

First, we are not testing users. We are testing the usability of the product in question. It is usability testing, not user testing. In an age where “user experience” has superseded “usability,” we may want an even better name for this method, but it is most definitely not “user testing.”

Second, usability testing is a perfectly good way to gather quantitative data. It just so happens to also be a very good way to get qualitative data.

When you conduct a usability test, you can quantify the number of errors that happen during tasks (and categorize them for even cooler measurements), completion time, whether or not participants completed the test, and you can collect task-level and test-level survey data.

Usability testing generally takes place with small sample sizes (on the order of 6 people or so), and this isn’t a barrier to descriptive statistics. You just have some big confidence intervals around your averages, but you can just acknowledge that in your reporting and decision making and get on with it.

I do understand thinking that quantitative data gathered from usability testing isn’t so helpful if you’re taking some kind of lean approach to product development. You’re moving quickly and treating usability testing as a more generative activity. Maybe it would be worthwhile to draw a distinction between these activities.