Are usability studies experiments?

When I conduct usability studies, I use a laptop and Morae to create a portable and low-cost usability lab. I typically visit the participant on site, so I can look around. I provide a scenario that participants can follow as they try using a new software feature. Morae records their voices, facial expressions, on-screen actions, clicks, and typing. The raw data gets saved in a searchable and graphable format. Afterward, I review the recorded data (I rarely have written notes) and make evidence-based recommendations to the feature’s project team.

For example, in one usability study I realised that the order of the steps was causing confusion and doubt. The three-step wizard would disappear after step, so users could modify their data on the screen. Users thought the task was completed with step 2, so the reappearance of the wizard for step 3 caused confusion. I recommended we change the order of the steps:

Change the order of the steps

Changing the order fixed the “confused users” problem. But was this scientific? Aside from the fact that I’m researching human subjects rather than test tubes, am I following a scientific model? At first glance, the usability test I conducted doesn’t seem to follow the positivist model of what constitutes an experiment:

A positivist model

For example, unlike the test-tube lab experiment of a chemist, my usability test had no control group. Also, my report went beyond factual conclusions to actual recommendations (based on my expert opinion) for the next iteration.

I can argue this both ways.

On the one hand, my usability test does have a control group if I take the next iteration of the product and repeat the usability test with additional participants, to see whether my recommendations solved the problem. I could compare the task-completion rates and the task duration.

A model of usability testing

On the other hand, if I were asked to determine whether the product has an “efficient” workflow or a “great” user experience—which are subjective measures—I’d say a positivist-model experiment is inappropriate. To measure a user’s confusion or satisfaction, I might consider their facial expressions, verbal utterances, and self-reported ratings. This calls for a research design whose epistemology is rooted in post-positivist, ethnomethodological, situated, standpoint, or critical approaches, and has more in common with research done by an ethnographer than by a chemist.

If you liked this, you may also like Epistemology of usability studies.

Your usability advantage

When businesses buy software, rather than choose the software with the lowest purchase price, they ought to consider the total cost of ownership—including the added productivity and enjoyment that usability and user-experience provide.

Every software company will say “our product is usable,” so how can you prove to prospective customer that you’ve really got usability?

Your product has a usability advantage if:

  1. Your development team’s motivation is right. Software meets customer business needs if came out of a design and development process that considers stakeholders beyond the development team.
  2. Incidentally, getting the motivation right is what Five Sketches™ was designed to help development teams to do.

  3. Trials quickly reveal product effectiveness. In a hands-on trial, you want users to try common tasks, figure them out, and say they liked the experience. A good hands-on trial reduces a competitor’s vendor demo to an infomercial.
  4. It’s about information more than data. Data requires cognitive transformation in the user’s head to become information. Information is ready now to support insight and appropriate action.
  5. Change management is minimal. Your mental model is clearly evident and the user experience is pleasant, so resistance to change is lower. Employees will see evidence of leadership rather than another “solution” imposed on them.
  6. Your training teaches skills. Pick one: training that leads users through a maze (an unusable-interface), or training that teaches users smarter ways to work toward their goals.
  7. You have metrics. If you tell customers how long it will take new users to start performing, you show your respect for their total cost of ownership.
  8. You have references. A product reference is as close as a Google search. In a web-2.0 world, your best “reference” could be an engaged, loyal user community.

The first 4½ to 5 points, above, require the Development team’s involvement, and the last few benefit from Dev involvement. Clearly, a usability advantage requires the involvement of other departments, directed by a product manager who works the Marketing, Sales, Support, and Development teams in concert. 🙂

This post was inspired by a Howard Hambrose article in Baseline Magazine, which recommends that IT professionals question software usability before they buy and implement.