product management – Usability research and analysis

26 March 200926 March 2009

Usability testing distant users

When a product’s users are scarce and widely dispersed, and your travel budget is limited, usability testing can be a challenge.

Remote testing from North America was part of the answer, for me. I’ve never used UserVue because the users I needed to reach were in Africa, Australia, South America, and Asia—continents that UserVue doesn’t reach. Even within North America, UserVue didn’t address the biggest problems I faced:

My study participants commonly face restrictive IT policies, so they cannot install our pre-release product and prerequisites.
I need to prevent study participants from risking their data by using it with a pre-release product.
There’s no way to force an uninstall after the usability test. Who else will see our pre-release?

Instead, I blended a solution of my own with Morae, Skype, Virtual Audio Cable, and GoToMeeting. I Testing that's really remote used GoToMeeting to share my desktop, which addresses all three of the problems listed above. I used Skype to get video and audio. I used Virtual Audio Cable to redirect the incoming voice from Skype to Morae Recorder’s microphone channel. Morae recorded everything except the PIP video. It worked. However, my studies were sometimes limited by poor Internet bandwidth to the isolated locations of my study participants.

Amateur facilitators. I realise this is controversial among usability practitioners, but beggars in North America can’t be choosers about how they conduct usability tests on other continents. I developed a one-hour training session for the team of travelling product managers. Training included a short PowerPoint presentation about the concepts, followed by use of Morae Recorder with webcam and headset while role-playing in pairs. The main points I had to get across:

Between study participants, reset the sample data and program defaults.
When you’re ready to start recording, first check that the video and audio are in focus and recording.
While you facilitate, do not lead the user. Instead, try paraphrasing and active listening (by which I mean vernacular elicitation). Remember that you’re not training the users, so task failure is acceptable, and useful to us.

I had a fair bit of influence over the quality of the research, since I developed the welcome script and test scenarios, provided the sample data, and analysed the Morae recordings once they arrived in North America. Due to poor Internet bandwidth to the isolated areas of my study participants, the product managers had to ship me the Morae recordings on DVD, by courier.

It worked. I also believe that amateur facilitation gave the product managers an additional opportunity to learn about customers.

25 March 200927 March 2009

Why products stay pre-chasm

I’ve spent some time working with legacy products—software for which the core code was written before “usability” was a term developers had heard of, back when developers were still called programmers.

I remember my first conversation—held last century—with a developer about his users and the usability of his legacy product. The product-adoption curve, with the chasm I used Geoffrey Moore’s book, Crossing the chasm, to introduce the idea that there are different types of users. The chasm (illustrated) shows five groups of users. The area under the curve represents the quantity of potential users, in the order in which they will typically adopt the product. Users who are technology enthusiasts and visionaries either enjoy or don’t mind being on the bleeding edge because of the benefits they get from using the product, according to Moore. Usability isn’t one of the typical benefits to the left of the chasm, where new products are first introduced. Wider adoption follows only after the product is made to cross the chasm—but this takes a concerted effort on the part of the development team.

What delays a product from crossing the chasm?

Revenues that are sufficient to let the company putter along.
Team members who themselves are tech enthusiasts or visionaries. This occurs, for example, when a nutritionist who also knows how to program develops software for nutritionists.
Team members who have infrequent exposure to newer user experiences. This could include someone who still uses Microsoft Office 2003, eschews social-networking applications on the web, or uses Linux.
Product managers whose roles are weakly defined or absent, making it more likely that new features get developed for the technical challenge of it, rather than for the user need or the business strategy.
Sales reps who ask for more functions in the product in order to make a sale—and a development culture that goes along with this.
Managers whose vision and strategic plan leave out user experience and usability.
Team members who have been on the team for decades.
Organisations that don’t use business cases to help them decide where to apply business resources.
Organisations with poor change-management practices—because moving a product across the chasm is more likely dramatic and disruptive than smooth and gradual.
Designers who are weak or absent in the process, so that “design” happens on the fly, during development.

If you recognise your work environment or yourself in this list, do you want to change things? If you do, but don’t know how, what actions can you take to learn the answer?

12 March 200913 March 2009

Users are not used to it

For several years, I did usability testing on CAD-style software that was full of legacy code, some of which preceded Windows 98.

Some of that legacy code dealt with CAD objects that displayed on screen. To work with these objects, users had a choice of menu commands and toolbar buttons, supplemented by dialog boxes. For example, to move an object, users could not simply click and drag it; they would choose a command, click the object, and then, in a dialog box, enter the distance to move the object.

That’s the way CAD programs worked when that legacy code was originally written.

Over the years, during my usability testing of various features, I noticed a growing trend toward direct manipulation. That is, to work with an object, users would try to click it or drag it. They would do this without thinking. Even long-time users, faced with a new feature (studies from 2005-2006), would try direct manipulation first:

100% of the test subjects clicked a cube, trying to select it.
100% of the test subjects dragged a point or line, trying to move it.
100% of the test subjects clicked in the window, trying to create a point.
100% dragged across points, trying to select them.

But the new features were built on the legacy code, so had the command-driven interaction style. A simple click on the object was usually a dead end for users.

And the users would say: “Darn,” and then look for another vector—another pathway—to complete the task.

The reasons we didn’t provide direct manipulation:

“Our users are used to the way it is now.” Clearly, usability-test results negated that argument. Users are not so accustomed to old-style interaction, because their first instinct for new CAD tasks in an existing product was direct manipulation.
“There’s not enough bang for the buck” because the opportunity cost (the cost of skipping other possible projects) was deemed too great. It’s hard to argue with this, as a usability analyst. The company opted for more features, and may have increased its risk of being leapfrogged by the competition, as discussed in an earlier blog.

11 March 200911 March 2009

Your usability advantage

When businesses buy software, rather than choose the software with the lowest purchase price, they ought to consider the total cost of ownership—including the added productivity and enjoyment that usability and user-experience provide.

Every software company will say “our product is usable,” so how can you prove to prospective customer that you’ve really got usability?

Your product has a usability advantage if:

Your development team’s motivation is right. Software meets customer business needs if came out of a design and development process that considers stakeholders beyond the development team.

Incidentally, getting the motivation right is what Five Sketches™ was designed to help development teams to do.

Trials quickly reveal product effectiveness. In a hands-on trial, you want users to try common tasks, figure them out, and say they liked the experience. A good hands-on trial reduces a competitor’s vendor demo to an infomercial.
It’s about information more than data. Data requires cognitive transformation in the user’s head to become information. Information is ready now to support insight and appropriate action.
Change management is minimal. Your mental model is clearly evident and the user experience is pleasant, so resistance to change is lower. Employees will see evidence of leadership rather than another “solution” imposed on them.
Your training teaches skills. Pick one: training that leads users through a maze (an unusable-interface), or training that teaches users smarter ways to work toward their goals.
You have metrics. If you tell customers how long it will take new users to start performing, you show your respect for their total cost of ownership.
You have references. A product reference is as close as a Google search. In a web-2.0 world, your best “reference” could be an engaged, loyal user community.

The first 4½ to 5 points, above, require the Development team’s involvement, and the last few benefit from Dev involvement. Clearly, a usability advantage requires the involvement of other departments, directed by a product manager who works the Marketing, Sales, Support, and Development teams in concert. :)

This post was inspired by a Howard Hambrose article in Baseline Magazine, which recommends that IT professionals question software usability before they buy and implement.

28 February 200820 March 2009

Make your project win

Here are two usability stories that are currently underway.

The first project

This weekend I got a Skype call from a usability consultant at a very large firm that is, in turn, working on a project for a very large insurance company. The usability consultant was just assigned, but the team is nearly finished building an online application form for household insurance. He told me:

The design is due Friday. Developers will start building it on Monday.
The design is restricted by a related piece of the site that’s already being built (offshore).
Customers must answer potentially hundreds of questions, and the company wants every answer cumulatively displayed on screen. It’s not clear why, so designing alternatives means shooting in the dark.
Customers must provide their complete personal and confidential identification-before they get a quote or decide whether to buy. It’s not clear why.
After answering all questions and providing personal information, many customers will be deemed ineligible to buy insurance online, due to answers that sounded innocuous to me.
There’s no time to test the design by Friday.

The second project

Today I spoke to a manager at a different company. Their current project is software that makes work schedules. So far, the team has:

Asked users for problems in the current product.
Weighed the severity and frequency of user problems. The worst problems related to starting a new schedule, figuring out where to enter data, and understanding how the data and settings influence the schedule.
The worst problems became top priority in the current project.
A major GUI redesign is underway. This includes step-by-step workflow support, drag-and-drop interaction, and replacing data-entry spreadsheets with dialog boxes or task panes.
The product manager wants usability testing done on the redesigned GUI, while there’s still time to make changes.

What can you do to ensure your projects are like the one in the second story?

28 January 200822 September 2009

Choose: usability or features?

I was talking to a B2B product manager who told me “The industry we target sees little difference between our product and our competitors.” Their plan is to differentiate their product from its competitors. My question: to make your software different from that of the competition, should you mainly add new functionality or mainly improve the usability?

Leapfrogging Bob Holt addresses this question in his article, Death by 1000 cuts. He asks: “As the worlds of our customers and our own business models continue to change and evolve, should we be changing the balance between improving the usability of our current products and adding new functionality?” Holt answers his own question: “I say absolutely yes. After all, it wouldn’t it be a shame if our revenues bled out through a thousand little cuts while we are rushing around trying to build the next Big Thing?”

But the same product manager I mentioned above also told me: “I firmly believe that you cannot win only by addressing usability. You must also look for the future of the market—that is, you won’t innovate past your current box—to avoid getting leapfrogged.”