Posts

Posted on

Effectiveness of usability evaluation

Do you ever wonder how effective expert reviews and usability tests are? Apparently, they can be pretty good.

Rolf Molich and others have conducted a series of comparative usability evaluation (CUE) studies, in which a number of teams evaluate the same version of a web site or application. The teams chose their own preferred methods—such as an expert review or a usability test. Their reports were then evaluated and compared by a small group of experts.

What the first six CUE studies found

About 30% of reported problems were found by multiple teams. The remaining 70% were found by a single team only. Fortunately, of that 70%, only 12% were serious problems. In one CUE study, the average number of reported problems was 9, so a team would be overlooking 1 or 2 serious problems. The process isn’t perfect, but teams found 80% or more of the serious problems.

Teams that used usability testing found more usability problems than expert reviewers. However, expert reviewers are more productive-they found more issues per hour-as this graph from the fourth CUE study illustrates:

CUE study 4 results

Teams that found the most usability problems (over 15 when the average was 9) needed much more time than the other teams, as illustrated in the above graph. Apparently, finding the last few problems takes up the most time.

The CUE studies do not consider the politics of usability and software development. Are your developers sceptical of usability benefits? Usability studies provide credibility-boosting video as visual evidence. Are your developers using an Agile method? Expert reviews provide quick feedback for each iteration.

To learn more about comparative usability evaluation, read about the findings of all six CUE studies.

Posted on

Complicated GUI is fixable

According to usability guru Jakob Nielsen, the worst mistakes in GUI design are domain-specific. Usually, he says, applications fail because they:

  • solve the wrong problem.
  • have the wrong features for the right problem.
  • make the right features too complicated to understand.

Nielsen’s last point reminds me of what a product manager once told me: many users of highly specialised software think of themselves as experts, but only few are. His hypothesis? Elaborate sets of features are too numerous or complex to learn fully.

Cookie on a plateOne of my projects involved software for dieticians. The software allowed users to enter a recipe. The software would calculate the nutritive value per portion. Users learned the basic settings for an adequate result. They ignored the extra features that could take into consideration various complex chemical interactions between the recipe ingredients. The extra features—the visual and cognitive complexity—got ignored. Ironically, their very presence increased the likelihood that users would satisfice, or avoid the short-term pain of learning something new. When the product was developed, each extra bit seemed a good idea, and they may also each have helped sell the product. But, good idea or not, those extra features needed to be removed, or hidden from the majority of users, or redesigned.

Resolving the “extra features” problem
  1. If the extra features are superfluous, remove them. Usage data can help identify seldom-used features, and many of our products are capable of collecting usage data, though we currently only collect it after crashes and mainly during Beta testing. However, removing a seldom-used part of an existing feature is a complex decision, and one for the Product Manager to make. The difficulty lies in determining whether a feature would be used more if it were simpler to use. In that case, it may not be superfluous.
  2. If the extra features are used only occasionally by relatively few users, then hide them. The suggested GUI treatment for an occasional-by-few control is to expose it only in the context of a related task. Do not clutter the main application window, menu bar, or the main dialog boxes with controls for occasional-by-few tasks. Hiding the controls for an occasional-by-few task is supported by the Isaacs-Walendowski frequency-commonality grid:
    3.   If the
      feature is…    		 Used
    by many    		 Used
    by few
    Used
    frequently    		 GUI treatment:
    •  Visible.
    •  Few clicks.    		 GUI treatment:
    •  Suggested.
    •  Few clicks.
    Used
    occasionally    		 GUI treatment:
    •  Suggested.
    •  More clicks.    		 GUI treatment:
    •  Hidden.
    •  More clicks.
  3. If the extra feature is to be a core feature, simplify it. I’m talking about a feature that the Product Manager believes would be used frequently or by many if users could figure it out. Burying or hiding such features isn’t the answer. You need to find ways to reduce complexity by designing the interaction well and by organising the GUI well. For this, Five Sketches™ can help.
What are the requirements, really?

All this begs the question: who can tell us which features are the extra features (the features to omit), which ones are occasional-by-few (the features to hide), and which ones are used frequently or by many users (the features on which to focus your biggest design-guns)? Nielsen says “Base your decisions on user research” and then test the early designs with users. He adds:

People don’t want to hear me say that they need to test their UI. And they definitely don’t want to hear that they have to actually move their precious butts to a customer location to watch real people do the work the application is supposed to support. The general idea seems to be that real programmers can’t be let out of their cages. My view is just the opposite: no one should be allowed to work on an application unless they’ve spent a day observing a few end users.  …More.

Conclusion: conduct user research and use what you learn to inform the design.

Posted on

A sketch must be disposible

The strength of sketching is that it’s a fast way to capture ideas.

Since a low-fidelity sketch is fast—pen on paper, as shown—it’s also low cost. And low cost means it’s relatively disposable if it turns out you can’t use that idea.

If you don’t like my first 5 ideas, that’s OK. I can have more ideas, easily and at low cost. And so can you.

A variation on this theme: iteration is also painless. With relatively little invested in a sketch, modifying an idea costs marginally more.

The payoff is that you can quickly saturate the problem space with ideas, before you analyse them. This is a key part of why Five Sketches™ works so well for development teams who are in a hurry to start programming.

It’s important to keep sketches cheap. Here’s a video of a cool sketching tool that, if used as a design aid, would greatly increase the project risk. That’s because this cool tool is expensive to install, expensive to learn (it requires training) and expensive to use (it allows only 1 user at a time). All this will reduce the number of sketches in the problem space—and it’s risky to design without considering all options.

The Assist Sketch Understanding System (ASUS) design environment: a computer interprets a sketch and then simulates a cart rolling down a hill.
Posted on

Make your project win

Here are two usability stories that are currently underway.
The first project

This weekend I got a Skype call from a usability consultant at a very large firm that is, in turn, working on a project for a very large insurance company. The usability consultant was just assigned, but the team is nearly finished building an online application form for household insurance. He told me:

  • The design is due Friday. Developers will start building it on Monday.
  • how-easy-it-itThe design is restricted by a related piece of the site that’s already being built (offshore).
  • Customers must answer potentially hundreds of questions, and the company wants every answer cumulatively displayed on screen. It’s not clear why, so designing alternatives means shooting in the dark.
  • Customers must provide their complete personal and confidential identification-before they get a quote or decide whether to buy. It’s not clear why.
  • After answering all questions and providing personal information, many customers will be deemed ineligible to buy insurance online, due to answers that sounded innocuous to me.
  • There’s no time to test the design by Friday.
The second project

Today I spoke to a manager at a different company. Their current project is software that makes work schedules. So far, the team has:

  • Asked users for problems in the current product.
  • Weighed the severity and frequency of user problems. The worst problems related to starting a new schedule, figuring out where to enter data, and understanding how the data and settings influence the schedule.
  • The worst problems became top priority in the current project.
  • A major GUI redesign is underway. This includes step-by-step workflow support, drag-and-drop interaction, and replacing data-entry spreadsheets with dialog boxes or task panes.
  • The product manager wants usability testing done on the redesigned GUI, while there’s still time to make changes.

What can you do to ensure your projects are like the one in the second story?

Posted on

Scott Berkun on saying “No”

While poring over the Vista UX Guidelines for something that was eluding me, I came across a chapter from the Scott Berkun book, The Art of Project Management (which has since been retitled to Making things happen). Here it is, courtesy of Microsoft Developer Network (MSDN):

Scott Berkun has also inspired me to compare Five Sketches™ to his Creativity Hacks ideas.

Posted on

Design Studio vs. Five Sketches™

I came across a software-design approach similar to Five Sketches™. It’s called Design Studio, and was developed in the USA. I was intrigued to learn about Design Studio. When separate teams develop a similar response to the same challenge, it validates both solutions. In this case, the challenge is to support software developers as they design the mental models, interaction, and GUI for their products.

Video no longer availableDesign Studio was presented at a 2007 conference, and you can watch it on video: but the BrightCove video is no longer available to watch.

If you have information about Design Studio, please comment, below. Found it! Jeff White and Jim Ungar presented their Design Studio method at the IxDA 2008 conference, in a presentation titled User Interface Design in an Agile Environment: Enter the Design Studio.

Posted on

Choose: usability or features?

I was talking to a B2B product manager who told me “The industry we target sees little difference between our product and our competitors.” Their plan is to differentiate their product from its competitors. My question: to make your software different from that of the competition, should you mainly add new functionality or mainly improve the usability?

LeapfroggingBob Holt addresses this question in his article, Death by 1000 cuts. He asks: “As the worlds of our customers and our own business models continue to change and evolve, should we be changing the balance between improving the usability of our current products and adding new functionality?” Holt answers his own question: “I say absolutely yes. After all, it wouldn’t it be a shame if our revenues bled out through a thousand little cuts while we are rushing around trying to build the next Big Thing?”

But the same product manager I mentioned above also told me: “I firmly believe that you cannot win only by addressing usability. You must also look for the future of the market—that is, you won’t innovate past your current box—to avoid getting leapfrogged.”

Posted on

Creativity hacks vs. Five Sketches™

My Vancouver colleagues and I had the chance to hear Scott Berkun speak, last week, at an event hosted by Vancouver User Experience. I sat in the front row with Sharon and Sam. Colin and Ken sat in the back.

Berkun shared his “creativity hacks”—a term he uses to describe the tips or advice that you can follow to foster your creativity.

I listened carefully for new ideas, and found that many of Berkun’s tips are already present—in some form—in the Five-Sketches™. There’s a lot of resonance between the various ways to cultivate creativity.

Creativity hacks

Five Sketches™
Document your ideas whenever you have an idea, in any medium.Try journaling to recognise how creative you are. You can be as creative as you want in your personal journal. It’s a “safe space” that is yours alone. We keep all sketches, but they are not private. Instead, they become documentation for your Canadian SR&ED tax-credit program. A portion of the design work may be eligible for a tax credit from Canada Revenue Agency. Journaling is a good habit, but not required by Five Sketches™.
Escape to give yourself the opportunity to hear your ideas. Escape while running, doing housework, driving, showering, suntanning—any rote or mindless activity. A problem-solving brain goes through three phases: understanding, incubation, elaboration. Escape promotes incubation. Design participants do the initial round of sketching alone, in advance, which means you can do it outside your usual workplace.
Invert the problem. Solve the opposite problem. For example, if you have to design a shopping-mall directory, ask: “What’s the worst that a mall directory could be?”

  • It lies.
  • It moves (changes location).
  • It smells or stinks.
  • It’s hidden.

Later, invert each of those ideas to find the desired attributes.

Inverting the problem is a “game” that leads to productive solutions by imposing an unusual constraint on the problem.

 Five Sketches™ changes your usual work setting by using music, food, sketching, and short story-telling to draw people out of their day-to-day routine.
Partner. Pairings force more new ideas to surface than on your own. Partner with people who have divergent skill sets. If you cannot find a partner, find a rival to compete with, as Michaelangelo did with Leonardo da Vinci. For great creative work to happen, seek out creative abrasion: some sort of tension between management and the team, or within the team, to encourage some interplay or competition. Five Sketches™ partners three or four people on a team.
Fail. Commit to taking risks to such an extent that you fail some of the time. If you’re not failing, you’re not doing anything sufficiently difficult. Fail in prototypes, in your journal (which is a “safe space” to fail), and so on. According to Berkun, Ernest Hemingway said: “The first draft of anything is shit.” Not every ideation-design project is an immediate, stellar success. Some design meetings uncover additional problem statements or the need for additional information that was overlooked during the pre-sketch preparation.
Plan for roadblocks. Politics are incredibly frustrating to creative people. Ask yourself why your last three ideas failed. Berkun listed the common reasons: You couldn’t convince the right people. You lost motivation. You ran out of time. Your team lead is an idiot. You gave up. Someone stole your idea. Commit to overcoming obstacles. If you are a manager, make these delegatable tasks. Recommendation: find out what people want, then try to give it to them. Take responsibility for your idea and how it will be perceived. Our ordered approach keeps Five Sketches™ organised. A flow chart helps the team decide when to use Five Sketches™, and who influences the product at which stage. The team conducts retrospectives, talk about the roadblocks, and continue to find ways to make the Five-Sketches™ method better for the individual team.
Switch modes. Ideas can best be discovered:

  • Visually.
  • Verbally.
  • Audibly.
  • Physically.

If you are stuck, find a new way to represent the problem, using a different mode of communication.

 Sketching, with markers, on paper. For really complex problems: Lichert-scale ratings of different solutions. Affinity diagrams help with the nonlinear representation of ideas. Card sorting helps us physically move ideas around the space. Skits and short story-telling helps to clarify a scenario. Customers or user personas, and photos of their workplaces, bring users into the design process.
Do something new/different/unusual. Go to a conference in a field outside your domain. Go to a part of the bookstore that you don’t normally go to—architecture? Chemistry? How would those disciplines approach problems similar to yours? Berkun took an improvisation (acting) course, and learned these rules:

  • No half-assing. Give your all.
  • No apologies.
  • Make the other guy look good.
  • Say “Yes, and…” instead of “No, but…”. This is the most important of these 4 rules, because you add something.
 In the re-sketching or mash-up stage, you “add something, and add something.” Sketching happens on paper with markers—not a developer’s usual medium. It asks you to come up with five different ideas, and makes room for “silly” or “impossible” ideas.

Want to know more? Check out Scott Berkun’s blog posts about creativity, or read about Five Sketches™ on this site.