Tag: process

Posted on

How to test earlier

Involving users throughout the software-development cycle is touted as a way to ensure project success. Does usability testing count as user contact? You bet! But since most companies test their products later in the process, when it’s difficult to react meaningfully to the user feedback, here are two ways to get your testing done sooner.

Prioritise. Help the Development team rank the importance of the individual programming tasks, and then schedule the important tasks to complete early.

  • Prioritise and schedule the tasksIf a feature must be present in order to have meaningful interaction, then develop it sooner.

    • For example, email software that doesn’t let you compose the message is meaningless. To get meaningful feedback from users, they need to be able to type an e-mail.

    Developers often want to start with the technologically risky tasks. Addressing that risk early is good, but it must be balanced against the risk of a product that’s less usable or unusable.

  • If a feature need not be present or need not be working fully in order to have meaningful interaction, then provide hard-coded actions in the interim, and add those features later.

    • For example, if the email software lets users change the message priority from Standard to Important, hard-code it for the usability test so the priority is always Standard.

  • If a less meaningful feature must to be tested because of its importance to the business strategy, then develop it sooner.

    • For example, email software that lets users record a video may be strategically important for the company, though users aren’t expected to adopt it widely until most laptops ship with built-in cameras.

Schedule. For each feature to be tested, get the Development team to allocate time to respond to usability recommendations, and then ensure this time is neither reallocated to problem tasks, nor used up during the initial development effort of the to-be-tested features. Engage the developers by:

  • Sharing the scenarios in advance.
  • Updating them on your efforts to recruit usability-study participants.
  • After developers incorporate your recommendations, retesting and then reporting improvements in user performance.

Development planning that prioritises programming tasks based on the need to test, and then allows time in the schedule to respond to recommendations, is more likely to result in usable, successful products.

Posted on

Ideas are disposable

Having a good design idea is not an amazing event. Sketching worksPeople have good design ideas all the time. What is amazing is having lots of good design ideas, so that they can be combined and iterated into the best possible design.

Here’s why ideas are disposable:

  1. Sketching lets you capture lots of ideas, quickly and inexpensively.
  2. The more ideas you try for, the sooner you learn that it’s easy to come up with five or more ideas.
  3. The easier it gets to have lots of ideas, the sooner you realise the value of any one idea is minimal.

Minimal value = disposable.

This is one of the keys to the success of Five Sketches™. Once you learn how easy ideas are to have, you can saturate the design space, iterate the design, and produce excellent design outcomes at will.

Posted on

Design and engineering culture

I’m sure Douglas Bowman’s blog last week was widely read. His post was a kind of public exit interview, titled Goodbye Google.

Goodbye GoogleAs Bowman left Google, he pointed out the pro-engineering bias in its approach to problem solving—including problems of design. Two of several examples he gives:

[…] a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. […]

Bowman would like to see more weight put on design principles. His blog includes a link to Wikipedia’s article on design elements and principles, which lists:

• unity • harmony • contrast
• balance • repetition, rhythm, pattern • variety, alternation
• emphasis, dominance, focal point • proportion, scale • functionality
• attraction • artistic unity • genuineness in media and form
• proximity • colour    

I don’t think design principles are beyond an engineer. I do think engineers need to be taught how—and when—to think about these details.

As we discovered during the development of Five Sketches™, this kind of detail is often outside the comfort zone of an engineer. However, I can affirm that even engineers who initially produce work with little design insight or creativity have managed to astonish me with amazing design results within a year. And that’s after only occasional participation in Five Sketches™ design sessions.

So, yes, engineers can learn to participate in design, with success and predictability, though I would caution that Five Sketches™ works for engineers because it was developed for and with them, to meet their needs.

At Google, to bring about such a cultural change, Bowman would have needed the unwavering support of at least one senior executive. Even then, as Bowman himself acknowledged in his goodbye message, a company as large as Google does not change direction easily.

If you liked this post, read about how a company’s use of social media influences its corporate culture.

Posted on

User mismatch: discard data?

When you’re researching users, every once in a while you come across one that’s an anomaly. You must decide whether to exclude their data points in the set or whether to adjust your model of the users.

Let me tell you about one such user. I’ll call him Bob (not his real name). I met Bob during a day of back-to-back usability tests of a specialised software package. The software has two categories of users:

  • What type of user is this?Professionals who interpret data and use it in their design work.
  • Technicians who mainly enter and check data. A senior technician may do some of the work of a professional user.

When Bob walked in, I went through the usual routine: welcome; sign this disclaimer; tell me about the work you do. Bob’s initial answers identified him as a professional user. Once on the computer, though, Bob was unable to complete the first step of the test scenario. Rather than try to solve the problem, he sat back, folded his hands, and said: “I don’t know how to use this.” Since Bob was unwilling to try the software, I instead had a conversation (actually an unstructured interview) with him. Here’s what I learned:

My observation For this product, this is…
• Bob received no formal product training. He was taught by his colleagues. Typical of more than half the professionals. 
• Bob has a university degree that is only indirectly related to his job.  Atypical of professionals.
• He’s young (graduated 3 years ago). Atypical of professionals, but desirable because many of his peers are expected to retire in under a decade.
• Bob moved to his current town because his spouse got a job there. He would be unwilling to move to another town for work. Atypical. Professionals in this industry typically often work in remote locations for high pay.
•  Bob is risk averse. Typical of the professionals.
•  He is easily discouraged, and isn’t inclined to troubleshoot. Atypical. Professionals take responsibility for driving their troubleshooting needs.
• Bob completes the same task once or several times a day, with updated data each time. Atypical of professionals. This is typical of technicians. 

I decided to discard Bob’s data from the set.

The last two observations are characteristic of a rote user. Some professionals are rote users because they don’t know the language of the user interface, but this did not apply to Bob. There was a clear mismatch between the work that Bob said he does and both his lack of curiosity and non-performance in the usability lab. These usability tests took place before the 2008 economic downturn, when professionals in Bob’s industry were hard to find, so I quietly wondered whether hiring Bob had been a desperation move on the part of his employer.

If Bob had been an new/emerging type of user, discarding Bob’s data would have been a mistake. Imagine if Bob had been part of a new group of users:

  • What would be the design implications?
  • What would be the business implications?
  • Would we need another user persona to represent users like Bob?
Posted on

Which user involvement works

User-centred design (UCD) advocated involving users in the design process. Have you wondered what form that user involvement could take, and which forms lead to the most successful outcomes?

I recently came across data that Mark Keil published a while ago. He surveyed software companies and correlated project outcomes with the type of user access that designers and developers had.

Type of contact with users Effectiveness
  For custom software projects
  Facilitate teams, hold structured workshop with users, or use joint-application development (JAD). ██████████
  Expose users to a UI prototype or early version to uncover any UI issues. ██████
  Expose users to a prototype or early version to discover the system requirements. ████
  Hold one-on-one semi-structured or open-ended interviews with users. ████
  Test the product internally (acceptance testing rather than QA testing for bugs) to uncover new requirements. ██
  Use an intermediary to define user goals and needs, and to convey them to designers and developers. ██
  Collect user questions, requirements, and problems indirectly, by e-mail or online locations. █
  For packaged or mass-market software projects
  Listen to live/synchronous phone support, tech-support, or help-desk calls. ████████
  Hold one-on-one semi-structured or open-ended interviews with users. ██████
  Expose users to a UI prototype or early version to uncover any UI issues. ████
  Convene a group of users, from time to time, to discuss usage and improvements. ████
  Expose users to a prototype or early version to discover the system requirements. ██
  Test the product internally (acceptance testing rather than QA testing for bugs) to uncover new requirements. ██
  Consult marketing and sales people who regularly meet with and listen to customers. ██
  At trade shows, show a mock-up or prototype to users and get their feedback. █
Not reported as effective in this 1995 source
  Conduct a (text) survey of a sample of users.
  Conduct a usability test to “tape and measure” users in a formal usability lab. (This study precedes such products as TechSmith Morae.)
  Observe users for an extended period, or conduct ethnographic research.
  Conduct focus groups to discuss the software.

Although Keil’s article includes quantitative data, his samples are small. I opted to show only the relative usefulness of various methods. My descriptions, above, are long because the original article uses 1995 terms that have shifted in meaning. I believe some of the categories now overlap, due to changes in technology and method. For example, getting users to try a prototype of the UI in order to uncover UI issues sounds like the early usability testing that I do with TechSmith Morae, yet the 1995 results gave these activities very different effectiveness ratings.

For details, see the academic article by Mark Keil (Customer-developer links in software development). Educational publishers typically require a fee for access.

These methods also relate to research I’m doing on epistemology of usability analysis.

Posted on

Internet Explorer leapfrogs Firefox?

Previously, I wrote about GUI—when to copy it and when to design it. When your competition has something better, I recommended you design, to leapfrog your competitor. Here’s an example of two competing web browsers:

Click to enlarge

At first glance, the new Internet Explorer 8 address bar looks like a copy of Firefox’s existing awesome bar, but click the image for an enlarged view. You’ll see that:

  • The on-the-fly suggestions are grouped as History and Favourites.
  • Each group lists only five items, by default.
  • To remove an item (think stale links and mistyped URLs), highlight the item and then click the  ×  that appears.

Compared to Internet Explorer 7, Firefox had a better address bar. And just as clearly, the additional features of the Internet Explorer 8 address bar are an attempt to leapfrog Firefox. After the public has used IE8 for three months, it’ll be interesting to hear whether users think Microsoft succeeded.

Read the related post, GUI: Copy it or design it.

Posted on

Epistemology of usability studies

Currently, I’m conducting research on usability analysis and on how Morae software might influence that. My research gaze is rather academic, in that I’m especially interested in the epistemology of usability analysis.

One of my self-imposed challenges is to make my research relevant to usability practitioners. I’m a practitioner and CUA myself, and I have little time for academic exercises because I work where the rubber hits the road. This blog post outlines what I’m up to.

At Simon Fraser University, I learned that epistemological approaches have different assumptions about what is knowable. On one side (below, left), it’s about numbers, rates, percentages, graphs, grids, tables, proving absolute truths. On the other side, (below, right) it’s about seeking objectivity while knowing that it’s impossible because everything has a cultural context. The epistemology you choose, when doing research, depends on what you believe. And the epistemology dictates what methods you use, and how you report your results.

You can be
certain of
what you know.		 You cannot be
objective about
what you know.

Let’s look at some examples.

Study 1 fits with the view (above, left) that “you can be certain of what you know.” I plan and conduct a quantitative study to measure the time it takes a series of users to complete two common tasks in a software package: upgrading to the latest version of the software, and activating the software. I make appointments with users. In my workplace, I give each user a scenario and a computer. I observe them and time them as they complete the tasks by using the software package. My hope is that statistical analysis will give me results that I can report, including the average time on task with error bars, as the graph (right) illustrates.

Study 2 fits with the view (above, right) that “you cannot be objective about what you know” because all research takes place within a context. To lessen the impact of conducting research, I contact users to ask if I can study their workplace. I observe each user for a day. My hope is to analyse the materials and interaction that I’ve observed in context—complete with typical interruptions, distractions, and stimuli. Since a new software version has just been released, my hope is that I’ll get to observe them as they upgrade. I’ll report any usability issues, interaction-design hurdles, and unmet needs that I observe.

The above are compilations of studies I conducted.

  • Study 1 revealed several misunderstandings and installation problems, including a user who abandoned the installation process because he believed it was complete. I was able to report the task success rate and have the install wizard fixed.
  • Study 2 revealed that users write numbers on paper and then re-enter them elsewhere, which had not been observed when users visited our site for usability testing. One user told me: “I never install  the latest version because the updates can be unstable,” and another said: “I only upgrade if there’s a fix for a feature I use” to avoid unexpected new defects. I was able to report the paper-based workaround and the users’ feelings about quality, for product managers to reflect in future requirements.

Clearly, there’s more than one way to conduct research, and not every method fits every team. That’s an idea that can be explored at length.

This has me wondering: which method fits what, when, where? Is there a relationship between a team’s development process and the approach to user research (epistemology) that it’s willing to embrace? …between its corporate usability maturity and the approach?

Those are two of the lines of inquiry in my research at Simon Fraser University.

If you liked this post, you may also like Are usability studies experiments?

Posted on

Generative design vs. Five Sketches™

© Leah Buley, from her presentation on SlideShareLeah Buley talked about generative design at the South by Southwest Interactive conference today [in March 2009]. Buley feels design methods are lacking in the set of professional tools we use for software development: “We don’t have so many good, reliable, repeatable design techniques.” I agree with her.

Buley tells how, in her first design session at Adaptive Path, she was handed a pen and paper, and told to sketch. The point was “to crank out a lot of ideas in a short time.” That’s what generative design is all about: saturating the design space with ideas. Design programs teach generative design, but there’s no generative design taught in programs for developers, QA staff, technical communicators, product management, or marketing.

There are already several design methods for software development teams to choose from, including Buley’s wonderful grab bag. Others I know of are Five Sketches™ (obviously) and Design Studio, both of which focus on the complete process of producing a design, start to finish, with all its challenges. Microsoft’s Bill Buxton told the UPA 2007 conference that he insists on generative design, and I’d love to see that in action. Each method differs slightly, but they all work because….

Why do they all work? Because of generative design. Generative design addresses what Buley calls her “dirty secret.” She freely confesses, about her design work before Adaptive Path: “I had very little confidence that what I was presenting as the design was in fact the one, optimal solution to the problem.” My experience teaching Five Sketches™ tells me that once you’ve participated in a generative-design process, you’ll know that you can have that confidence.

A related question: when will computer science programs teach Basic Design Methods to developers?

Posted on

Your usability advantage

When businesses buy software, rather than choose the software with the lowest purchase price, they ought to consider the total cost of ownership—including the added productivity and enjoyment that usability and user-experience provide.

Every software company will say “our product is usable,” so how can you prove to prospective customer that you’ve really got usability?

Your product has a usability advantage if:

  1. Your development team’s motivation is right. Software meets customer business needs if came out of a design and development process that considers stakeholders beyond the development team.

    2. Incidentally, getting the motivation right is what Five Sketches™ was designed to help development teams to do.

  2. Trials quickly reveal product effectiveness. In a hands-on trial, you want users to try common tasks, figure them out, and say they liked the experience. A good hands-on trial reduces a competitor’s vendor demo to an infomercial.
  3. It’s about information more than data. Data requires cognitive transformation in the user’s head to become information. Information is ready now to support insight and appropriate action.
  4. Change management is minimal. Your mental model is clearly evident and the user experience is pleasant, so resistance to change is lower. Employees will see evidence of leadership rather than another “solution” imposed on them.
  5. Your training teaches skills. Pick one: training that leads users through a maze (an unusable-interface), or training that teaches users smarter ways to work toward their goals.
  6. You have metrics. If you tell customers how long it will take new users to start performing, you show your respect for their total cost of ownership.
  7. You have references. A product reference is as close as a Google search. In a web-2.0 world, your best “reference” could be an engaged, loyal user community.

The first 4½ to 5 points, above, require the Development team’s involvement, and the last few benefit from Dev involvement. Clearly, a usability advantage requires the involvement of other departments, directed by a product manager who works the Marketing, Sales, Support, and Development teams in concert. :)

This post was inspired by a Howard Hambrose article in Baseline Magazine, which recommends that IT professionals question software usability before they buy and implement.

Posted on

Designing *with* developers

Today, Joel Spolsky blogged about development process and design. He makes a couple of points I agree with. As an example, he says that developers don’t know how to do everything. He says it first by describing his lack of skills in an early job at Microsoft, and later by describing the lack of skills in very experienced developers:

Joel SpolskyYour garden-variety super-smart programmer is going to come up with a completely baffling user interface that makes perfect sense IF YOU’RE A VULCAN (cf. git). The best programmers are notoriously brilliant, and have some trouble imagining what it must be like not to be able to memorize 16 one-letter command line arguments. These programmers then have a tendency to get attached to their first ideas, especially when they’ve already written the code.

At this point, reading Spolsky’s blog, I’m like the eager school kid with his hand up: “Oh, oh, I know the answer!” Five Sketches™ was specifically developed to prevent the people-attached-to-their-first-ideas problem and to ensure collaboration between developers and other members of the team.

Software products are developed by teams, but, as Spolsky goes on to say—about developers—some team members have more power those who work alongside them, because developers control the code. He tells this story:

Joel Spolsky       A programmer asks me to intervene in some debate he is having with a [non-developer peer; in this story it’s a Program Manager who performs a design function].
       “Who is going to write the code?” I asked.
       “I am….”
       “OK, who checks things into source control?”
       “Me, I guess, ….”
       “So what’s the problem, exactly?” I asked. “You have absolute control over the state of each and every bit in the final product. What else do you need? A tiara?”

Spolsky’s solution is to rely on the ability of others [in this case, the Program Manager] to convince the developer to make necessary changes. But requiring stakeholders and colleagues to engage in persuasion is risky; the variables include not only technical skill and experience, but also credibility, and rhetoric, persuasion, and interpersonal communication skills of both parties.

I recall when we first developed our ideation-design process, on an interdisciplinary team. We had Sharon to debrief each of us, to find out what had worked and what hadn’t, and then we addressed the shortcomings and frustrations in the process. This is how we developed Five Sketches™. And this is why it works … with developers … and with people from QA, Marketing, Tech-Comm, and so on.

By the way, you can read Joel Spolsky’s entire blog post here.