"The Hiring Problem" : Academic HCI vs. the Real World of Practice

"The Hiring Problem" :Academic HCI vs. the Real World of Practice Lynn Cherny (a talk given at UW in 2004, while I was at Adobe) Lynn Cherny

Contents • My background • Industry Today • Usability and Design • We need HCI Executives! • The Professionals: Contractors vs. In-House folks • Interviewing for HCI folks • Example job ad and interview contents • But is it the real job or the advertised job? • HCI in Education and Industry Preparation • Please, Teach the Design/HCI Students … [A Big List!!] • And Teach… • Your Research Students • Possible Lessons for Faculty Lynn Cherny

My background • HCI Research at AT&T Labs • Excite, Senior UI designer • TiVo, UI designer & then manager • Axance (consulting), Dir of Methodology • Adobe, Senior UI Designer for cross-product issues • The Mathworks (current), Principal Usability Specialist Lynn Cherny

Industry Today: Usability vs. Design • Usability Engineering: Evaluation, field work, lab work  often highly-educated voices stuck in the lab ghetto. • Interface Designers: A variety of backgrounds, doing a lot of different jobs; not all from HCI. • Strong’s NSF report (1995): researchers, research professionals, converts from “soft disciplines,” ergonomists, converts from computer disciplines, graphic designers… Lynn Cherny

Industry Today: We Need Designers and HCI Executives “The rise in the dependence of HCI on usability labs is basically a regression. Design is where the action is.” -- Stu Card, CHI 2002 “We do not create anything of substance: we are critics. The Design profession flourishes because they do things, they create. We must become designers.” – Don Norman, CHI 2002 “If you want to be the low status, low on the totem pole person in your company, then yeah, rejoice in the fact that you are hiring user testers. User testing is not where the action is. The action is with those people who decide what product to build in the first place. That isn't the user tester community, but it should be the CHI community.” –Don Norman, interactions (2000) Design skills will help break the glass ceiling on promotion for HCI professionals. We want more designers (and executives) coming from solid HCI backgrounds. Lynn Cherny

Industry Today: Who are the “Professionals”? • Mantei and Hewett provided a high-level analysis into the professional categories at work in HCI: • “researcher,” • “profesionally oriented researcher,” • “research oriented professional,” • “professional” • What I see: • Consultants: The senior folks you see and hear from most (from CHI, UPA, etc.) • In-House permanent staff (the “professional” category): • Go to fewer conferences • Are much younger and very often somewhat alienated from the “profession” – nothing new, heard it all before, nothing will benefit me there… • Don’t belong to ACM/Sigchi, don’t read the articles (or write) • Don’t know the theory, quite often • Do far less original design work than you’d expect!  Let’s assume these folks are your students Lynn Cherny

Interviewing: Example Job Ad Responsibilities: Works with multidisciplinary software development teams to help understand user needs, specify usability requirements, and verify requirements and needs are met. Conducts user research, including contextual inquiry. Facilitates user-based requirements discovery and user-centered design. Plans and conducts lab and field usability tests. Analyzes, reports, and presents results and recommendations to development team and product management. Qualifications & Experience:· 4-Year degree or Graduate degree in Computer Science, Software Engineering, Cognitive Psychology, Human Factors, or related field. · Practical knowledge and experience in usability engineering principles and methods and familiarity with various prototyping tools and techniques. · Strong communication, interpersonal, and organizational skills.· Familiarity with chemistry and the pharmaceutical industry helpful. Lynn Cherny

Interviewing: Hiring a UI Designer • Traditional questions: “describe a difficult team interaction around a design and what you did to resolve it” • Portfolio work review (how much is theirs?) • Design critique • Design problem – whiteboard • General vocabulary assessment: “UCD,” “prototype,” “flowchart,” “usability,” “iterative”…  It is famously difficult to hire “good designers.” The question is, do we really need to worry about this? Lynn Cherny

Interviewing: But is it for the Advertised Job or the “Real” Job? • “Evangelism” often comes up in ads/interviews – a warning sign of dysfunction • Note that many designers’ portfolios don’t show “their work” – but are they really trying to mislead you? • Not everyone knows what the real job consists of (at hiring and performance eval time) • We interview in a self-deluded state – how we want it to be (or to evolve) vs. how it really is; hence we hire on the wrong criteria sometimes. • Exactly how important is design talent for success? Possibly less so than a complete skill set for getting design DONE. Case Study Example: Hiring Sally, a “good designer” Lynn Cherny

HCI and Education for Practice • SigCHI was 68 people in 1982, over 6000 in 2002; well-known conference rejection rate  and many satellite conferences • Much interest internally in status of the profession and HCI education: • Strong’s NSF Report (1995), • Hewett, T. T. et al Curricula in HCI (1992), • Perlman & Gasen’s The ACM SIGCHI HCI Education Survey (1993), etc. • multiple panels at CHI… Lynn Cherny

HCI and Education: Industry Preparation? A known problem. It’s difficult to address needs of industry in HCI curricula because: • HCI itself is expanding, parts are ill-defined, and we don’t have a good understanding of what some topics have to offer industry, • “Industry” is a collective term for a wide range of different activities done by people whose needs vary considerably • When you talk to people in industry they usually don’t know their own needs and they certainly don’t know what their colleagues needs are, many don’t even know what most of their colleagues do. --Jenny Preece, CHI 94 on “Is HCI Education Getting a Passing Grade from Industry?” Lynn Cherny

HCI and Education: Supposed Activities of Practice (Strong’s NSF Report, 1995) Lynn Cherny

HCI and Education: Reality of Practice From Strong’s NSF Report in interactions (1995): “Many HCI practitioners experience identity conflicts within the first few years of entering an applied position. In general, they must reorient their views of how they work, what questions they ask, and what about their work is valued.” • Can we improve their experience by better preparation? (Internships aren’t sufficient.) Lynn Cherny

Teach them… The Language of Business & Marketing instead of HCI & UCD Don: Why do you think [marketing has] so much of a role determining the products? It's not because they're brighter, and it's not because they have any more truths. It's because they know how to play the game better. What I suggest is that it's time we learn how to play the game. Janice: The advantage that marketing has is that it's a well-known concept. Executives have typically gone to business school where marketing is well-known. One of the problems that we have is that we don't have usability engineering and HCI in most curricula for business and engineering. Don: One reason we don't is that we don't talk the language of business. Don Norman and Janice Rohn, Interactions, Volume 7, Number 3 (2000), Pages 36-60 Lynn Cherny

Teach Them… Product Design instead of Interface Design, If You Can! Think about the big picture: building and selling a product, not doing just the UI design • From market research, write a business plan • Identify the crucial factors to distinguish your product in the marketplace  Consider that usability alone doesn’t motivate intent to purchase. Study some business cases and learn what makes something sell – this will give perspective and credibility and reduce their anxiety tremendously! Lynn Cherny

Teach Them… Usability Methods Produce designers who know usability evaluation methods: • We’ve won the battle of the usability evaluation at most companies – this is a needed job skill now. • But teach them to make strong recommendations from user problems, that are credible, grounded, realistic. • All field methods, surveys, focus groups, usability test design, basic statistics, interviewing, content analysis… • Researchers who want to make an impact on product definition should learn how to “disguise” themselves as designers till they can achieve the impact they want. Lynn Cherny

Teach them… How toDetermine Appropriate Study Methods Propose the appropriate diagnosis method to get at useful data for a design problem • What usability technique will get useful data? • What type of data will be wanted/understood? • What arguments are needed for it? • Guard against the naive proposals: “Let’s just ask users on the beta list.” Consulting is therapy for dysfunctional companies. Lynn Cherny

Teach Them… To Characterize their Customers Marketing thinks in “segments” – it’s our job to define and identify who will really be using the products. • Be able to get the data, or make a reasonable guess and review with team. The design should be tied to the user/customer profiles, this will save much pain! • Don’t go through the motions; if it’s not in a form that’s obviously useful, you’ve done something wrong. Lynn Cherny

Teach them… To Identify and Characterize Problems To identify and summarize the key problems to solve – in any domain! • By what criteria • According to what evidence • Identify the causes of the problem that require action– break it down usefully. Stated concisely: “Inconsistent color” is a user complaint – but the operational problem is not in the same terms.  Skill that requires strong analysis and abstraction skills, backed up with excellent communication ability. Teach them how to THINK through an issue! Lynn Cherny

Teach them… To Review and Critique Designs To critique designs and find improvements, both minor and major.  Most practical UI design is incremental improvement; and new work is always relative to existing products in some way. Don’t skip this step, it seeds new ideas. Design test at interviews often checks on this: Lynn Cherny

Teach Them… Different Kinds of “Design” Expose them to many kinds of design: visual, industrial, IA (web), interaction, object-oriented software design, patterns, functional specification, chart/table, document layout, experimental design… • They all have different types of deliverables – consider a project with several of them, and make them distinguish their contributions and boundaries/limits • Not everyone is equally good at each, but learning limits now is better.  Try brainstorming and use methods to unlock creativity Lynn Cherny

Teach them… Design for Multiple Devices Consider design for • Mobile devices • Desktop apps • Social/community • Web • TV • Physical objects • Games (several types) Lynn Cherny

Teach Them… To do Fast Low-Fi Design To produce fast preliminary designs – multiple concepts per problem • With callouts/text explaining concisely the differences • Produce multiple formats of low fidelity: workflow diagrams, task breakdowns, wireframes, sketches…  Accept that others will “help” with this and that’s useful, not a threat. Lynn Cherny

Teach Them… ToPrepare Multiple Deliverables To prepare and explain value of all typical design stage deliverables: • Workflows, flowcharts, wireframes, storyboards, high fidelity mockups • Write specifications in increasing detail, focusing on the important details at each stage Lynn Cherny

Teach them… To Recognize the Value of Multiple Levels of Abstraction • Describe the user’s mental model and how it relates to the UI in taskflow/vocabulary • Flowcharts of the UI experience vs. the storyboard; taskflow vs. the data flow… Lynn Cherny

Teach them… ToPresent Their Design Work Make them present their designs to diverse groups: business, engineering… • With appropriate justification of thought process • With strategic defenses prepared • Orally, but with supporting materials • Make them respond to naive questions and criticisms of any type  Consider asking them to present someone else’s work – it will strengthen the arguments, and require the author to distill their points; and this won’t provoke so defensive a response to critique from the naïve. Lynn Cherny

Teach them… To Find and Interpret Previous Research Research literature can be useful, but requires interpretation, summarization, digestion. If they can’t do this, the gulf between practice and the academy will grow. • Consider book reports – oral recaps to audiences who are naïve; make it interesting and relevant to them! • Sometimes “previous work” is not reviewed research – don’t forget about online white papers, etc. • Identify ways to work research strategically and tactically into design work • Make them use the ACM digital library!! Lynn Cherny

Teach them… To Relate In-House Research to Design To summarize the important conclusions from research/evaluations and how they relate to their design decisions • Throughout the design process, at all stages! • This will be a cover-your-ass strategy (for HCI as a profession) but will also pay off in a better design, if we’re really doing the right things. • Gap between field research and design is still huge! Be an advocate for usability methods! Lynn Cherny

Teach them… Justify without Defensiveness • They must justify their work gracefully. • Everyone thinks she’s a designer; every decision will be questioned no matter how minor.  Note that bad designers can’t necessarily recognize good designs. (The question is not will this happen, it’s how insane will it make them.) Lynn Cherny

Teach them… Software Development Lifecycle Models To pay attention to engineering lifecycle concerns, and learn how to argue for and adapt UCD to work with them (Rapid Development, Agile and XP, RUP, boxcar…) Strong (1995): HCI practice can help to inform and improve life cycle activities. Within such a life cycle, HCI emphasizes iteration and concrete communication. HCI research has demonstrated that straightforward activity categories-such as “analysis,” “design,” and “evaluation’‘ are not separable in practice. … HCI practitioners can help show the weaknesses of the waterfall model, but not without risk. The risk is that the somewhat marginalized field of HCI practice may lose some of its credibility and influence if it is perceived within organizations as opposing their received wisdom. Lynn Cherny

Teach Them… How to Interview Have them interview each other for jobs, and ask each other portfolio questions – challenge each other with design tests • Review resumes with them • Their peers will be hiring them eventually! Lynn Cherny

Teach them… To Drive Projects • They should assume that nothing will happen between high level marketing requirements and engineering implementation; they’re the connector: Organize meetings, identify stakeholders, investigate technical issues, do research if needed, design and get it reviewed.  But drive decisions! Design MRD/Biz plan Implementation Lynn Cherny

Teach them… Most Design Happens in Committees The challenge is how to have an influence over the committee – how to prepare the right materials and arguments.  Come in having thought about it harder than anyone else and expecting to present ideas in some concise form. Lynn Cherny

Teach Them… Public Speaking Require presentations – confidence, poise, clarity, extemporaneous speech Start with article/book reports, move on to design presentations Lynn Cherny

Teach them… To Study Their Own Organization They must be ethnographers of the workplace, to understand the best methods of success and tailor their message for their audience. • How do people get promoted? What gets rewarded? What’s teamwork? Who’s in charge? • How can you document the impact of your own contribution? Lynn Cherny

Teach them… To be Solid Experts but not Shrill Evangelists Teach them to justify in plain language, but not to be pushy. Evangelism is dangerous and marginalizes over the long term. Siegel and Dray (interactions, Aug 2003): Actively demonstrate your identification with concerns that exceed your discipline and with the company’s larger goals. Whenever possible, speak a common language rather than the language of your discipline. All your recommendations should be clearly supported not by reference to the principles or belief system of your discipline, but by clear evidence of what is best for the company in a practical way. Consider reframing what UCD is about, in ways that show its links to larger, shared concerns—due diligence, product planning, quality, reduced risk, efficiency, and innovation—and to marketing and business goals like customer satisfaction and retention, and, ultimately, profitability. Lynn Cherny

Teach them… Organizational Planning Make them plan for team projects: • How will the team resolve debates? • How will the team prioritize features vs. schedule and resources? • Track action items and evolution of work? • How do you plan to estimate work across team and disciplines? Scope work? Lynn Cherny

Teach them… To Know and Evaluate Books on Design Engineers regularly ask for references on how to do design. (See Jeff Johnson’s panel at CHI 2002 on books on design) • Consider Cooper’s books, Johnson’s, Mayhew, Mullet and Sano, Norman, Tufte, etc. • Books on requirements analysis that include usability research are useful recommendations too Lynn Cherny

Teach them… The Difference Between Peers and Resources There are two kinds of people in organizations—there are peers and there are resources. Resources are like usability consultants—we go out and we hire them. We'll hire a consultant or we'll have a little section that does usability and think of it as a service organization. We call upon them when we need them to do their thing, and then we go off and do the important stuff. That's very different than peers, where a peer is somebody I talk to and discuss my problems with, and who helps to decide upon the course of action. As you get higher and higher in the organization, this becomes more of an issue. The executive staff talks to the executive staff, and they have beneath them all this organization, which are their resources that they deploy. But the big decisions are being made among peers. And it's really important to advance in the world to be thought of as peers. …Our usability labs are resources that we call in when we think we have some trouble with usability. We go and spend a few hours and we worry about the budget. But we don't take it seriously, we only take peers seriously. And you only get to be peers if you speak the right language and if you're making a contribution at the level the company cares about...which is profitability. --Don Norman, interactions (2000) Lynn Cherny

And Teach Your Research Students: Ben Shneiderman, CHI 2002: “We must recognize that nothing is so practical as a good theory and that theory thrives when challenged by practice. Our goals should include development of predictive, explanatory, and generative theories that systematically support the next generation of innovations.”  Express the implications of your research clearly for your colleagues in practice or it will have limited impact. Lynn Cherny

Possible Lessons for Faculty… • Stay in touch with practice, not just via consulting and conferences • Build departmental bridges, to business schools in particular • Consider offering co-taught “Product Innovation” and “Product Management” courses to cross-pollinate • Encourage good software engineering education and practice as well: concern for architecture, teamwork, documentation, test plans… • Evaluate and improve your own teaching with feedback from former students in professional practice Lynn Cherny

Some Takeaways • We have to produce Supermen, Know-it-alls – obviously difficult. • Not everyone will be a “good designer,” but this may be less important than many of the other skills used on the job • You can’t teach everything – in a pinch, sacrifice some of the theory (move to advanced classes) and instead teach how to do research/where to find it, and how to read it. Plus the practical items I described! • Usability engineering (of the evaluation sort) is clearly a valuable and important skillset, BUT: By getting more design into this practice, the door to true and deep impact will open wide. Lynn Cherny

Thanks for listening! Lynn Cherny arnicas@gmail.com www.ghostweather.com Lynn Cherny

"The Hiring Problem" : Academic HCI vs. the Real World of Practice