A Fundamental Mind Shift For Usability Testing

UX Strategy with Jared Spool, a Center Centre — UIE newsletter focused on bringing UX to a strategic level inside your organization.

A few days ago, I was watching a user experience expert deliver an interesting talk to a conference of UX professionals. (I do this quite a lot, as I’m always looking for new experts for our All You Can Learn UX video library and future events we might host.)

The presenter was doing an excellent job, making some solid points. I found myself nodding in agreement with every great idea I heard.

Suddenly, the presenter felt the need to impress upon the audience the importance of user research. They said, “You don’t need a big project. A small usability test is all you need. Studies have proven that, with only five to eight users, you’ll find 85% of all the problems in your design.”

My immediate thought: Noooooo! Really?!? It’s 2019. We don’t say this anymore. We never should’ve said this.

This idea, that five to eight users will reveal 85% of all usability problems, is an old myth. It’s not true. It’s never been true.

The origins of the five to eight user myth.

In 1990, Bob Virzi published a paper, Streamlining the Design Process: Running Fewer Subjects, suggesting that for many designs, you could get away with running a usability test with 4 or 5 participants. He had shown a handful of studies where he had ten to fifteen users, but most of the problems showed up in the first four or five. He declared that, if you wanted to fix the obvious problems, that’s all you needed.

In 1993, Jakob Nielsen and Thomas Landauer built on Virzi’s suggestion and tried to refine the mathematical curve, in their seminal paper A mathematical model of the finding of usability problems. Their goal was to find a formula that would tell you how many participants you might need to find 85% of your design’s problems.

Nielsen and Landauer analyzed five usability-testing studies and determined the maximum number of users you needed was eight. They didn’t analyze all usability test studies ever done in the world, they just analyzed five.