Effectiveness of usability evaluation

Do you ever wonder how effective expert reviews and usability tests are? Apparently, they can be pretty good.

Rolf Molich and others have conducted a series of comparative usability evaluation (CUE) studies, in which a number of teams evaluate the same version of a web site or application. The teams chose their own preferred methods—such as an expert review or a usability test. Their reports were then evaluated and compared by a small group of experts.

What the first six CUE studies found

About 30% of reported problems were found by multiple teams. The remaining 70% were found by a single team only. Fortunately, of that 70%, only 12% were serious problems. In one CUE study, the average number of reported problems was 9, so a team would be overlooking 1 or 2 serious problems. The process isn’t perfect, but teams found 80% or more of the serious problems.

Teams that used usability testing found more usability problems than expert reviewers. However, expert reviewers are more productive-they found more issues per hour-as this graph from the fourth CUE study illustrates:

CUE study 4 results

Teams that found the most usability problems (over 15 when the average was 9) needed much more time than the other teams, as illustrated in the above graph. Apparently, finding the last few problems takes up the most time.

The CUE studies do not consider the politics of usability and software development. Are your developers sceptical of usability benefits? Usability studies provide credibility-boosting video as visual evidence. Are your developers using an Agile method? Expert reviews provide quick feedback for each iteration.

To learn more about comparative usability evaluation, read about the findings of all six CUE studies.

Make your project win

Here are two usability stories that are currently underway.
The first project

This weekend I got a Skype call from a usability consultant at a very large firm that is, in turn, working on a project for a very large insurance company. The usability consultant was just assigned, but the team is nearly finished building an online application form for household insurance. He told me:

  • The design is due Friday. Developers will start building it on Monday.
  • how-easy-it-itThe design is restricted by a related piece of the site that’s already being built (offshore).
  • Customers must answer potentially hundreds of questions, and the company wants every answer cumulatively displayed on screen. It’s not clear why, so designing alternatives means shooting in the dark.
  • Customers must provide their complete personal and confidential identification-before they get a quote or decide whether to buy. It’s not clear why.
  • After answering all questions and providing personal information, many customers will be deemed ineligible to buy insurance online, due to answers that sounded innocuous to me.
  • There’s no time to test the design by Friday.
The second project

Today I spoke to a manager at a different company. Their current project is software that makes work schedules. So far, the team has:

  • Asked users for problems in the current product.
  • Weighed the severity and frequency of user problems. The worst problems related to starting a new schedule, figuring out where to enter data, and understanding how the data and settings influence the schedule.
  • The worst problems became top priority in the current project.
  • A major GUI redesign is underway. This includes step-by-step workflow support, drag-and-drop interaction, and replacing data-entry spreadsheets with dialog boxes or task panes.
  • The product manager wants usability testing done on the redesigned GUI, while there’s still time to make changes.

What can you do to ensure your projects are like the one in the second story?