User-centred design (UCD) advocated involving users in the design process. Have you wondered what form that user involvement could take, and which forms lead to the most successful outcomes?
I recently came across data that Mark Keil published a while ago. He surveyed software companies and correlated project outcomes with the type of user access that designers and developers had.
| Type of contact with users | Effectiveness | |
| For custom software projects | ||
| Facilitate teams, hold structured workshop with users, or use joint-application development (JAD). | ██████████ | |
| Expose users to a UI prototype or early version to uncover any UI issues. | ██████ | |
| Expose users to a prototype or early version to discover the system requirements. | ████ | |
| Hold one-on-one semi-structured or open-ended interviews with users. | ████ | |
| Test the product internally (acceptance testing rather than QA testing for bugs) to uncover new requirements. | ██ | |
| Use an intermediary to define user goals and needs, and to convey them to designers and developers. | ██ | |
| Collect user questions, requirements, and problems indirectly, by e-mail or online locations. | █ | |
| For packaged or mass-market software projects | ||
| Listen to live/synchronous phone support, tech-support, or help-desk calls. | ████████ | |
| Hold one-on-one semi-structured or open-ended interviews with users. | ██████ | |
| Expose users to a UI prototype or early version to uncover any UI issues. | ████ | |
| Convene a group of users, from time to time, to discuss usage and improvements. | ████ | |
| Expose users to a prototype or early version to discover the system requirements. | ██ | |
| Test the product internally (acceptance testing rather than QA testing for bugs) to uncover new requirements. | ██ | |
| Consult marketing and sales people who regularly meet with and listen to customers. | ██ | |
| At trade shows, show a mock-up or prototype to users and get their feedback. | █ | |
| Not reported as effective in this 1995 source | ||
| Conduct a (text) survey of a sample of users. | ||
| Conduct a usability test to “tape and measure” users in a formal usability lab. (This study precedes such products as TechSmith Morae.) | ||
| Observe users for an extended period, or conduct ethnographic research. | ||
| Conduct focus groups to discuss the software. | ||
Although Keil’s article includes quantitative data, his samples are small. I opted to show only the relative usefulness of various methods. My descriptions, above, are long because the original article uses 1995 terms that have shifted in meaning. I believe some of the categories now overlap, due to changes in technology and method. For example, getting users to try a prototype of the UI in order to uncover UI issues sounds like the early usability testing that I do with TechSmith Morae, yet the 1995 results gave these activities very different effectiveness ratings.
For details, see the academic article by Mark Keil (Customer-developer links in software development). Educational publishers typically require a fee for access.
These methods also relate to research I’m doing on epistemology of usability analysis.