Measuring “ROI” for your usability dollar and hour

1. Measuring “ROI” for your usability dollar and hour How justification of investment has made a come back after the heady days of “irrational exuberance,” a.k.a “e-hype”

2. Introduction Defining ROI is a slippery project at best. A lot of people talk about it and a lot of people say they calculate it, but the term seems to morph depending on who is using it. This is my introduction to ROI (Return on Investment).

3. Acronyms-a-Go-Go: Terminology • ROI (Return on the Internet) • ROI (Relying on Instinct) • ROE (Return on Equity) • ROE (Return on Expectations) • SIB (Staying in Business) • ROR (Return on Relationship) • CRM (Customer Relationship Management)

4. Return On Investment: It’s a math thing. • ROI - (Return on Investment) Net Profit divided by Net Worth. A financial ratio indicating the degree of profitability. • return on investment (ROI)the ratio of net profit after income tax, over owner’s equity. Usually expressed as a percentage.

5. Huh? • ROI is an established formula applied to different circumstances. • Assessing ROI http://www.businessweek.com/magazine/content/01_44/b3755609.htm • For example, the next three sites exemplify different uses for ROI calculations: http://bwnt.businessweek.com/roi/enter.asp http://xproi.forbes.com/ http://business.broadcast.com/roi/marketing_roi.asp

6. Old School / New Priorities: it’s a crash thing • ROI is an old business tool. During the economic boom of the turn of the century, evaluation methods got thrown out the window. If it was an IT application, it was given the go-ahead. After the crash, business leaders turned back to established practices: • “Ever since the dotcom bust and economic slowdown, IT organizations have latched on to all manner of "ROI" metrics to justify their technology investments.” (Lewis, David. Koller, Mike.) • “Conventional wisdom holds that in a slowdown, CEOs, CIOs and shareholders alike are watching every IT penny that goes out the door and making sure an equal or greater number of pennies flows back in.” (Joachim, David.)

7. To ROI or not to ROI? That is the question. • “’A lot of what we're doing in IT is being done for the first time, so original estimates and predictions are a dartboard,’ says John Zarb, CIO at Libbey Inc.” (Hayes, Mary. Chabrow, Eric, et. al.) • While ROI (Return on Investment) is established protocol, there are still plenty of people who would rather rationalize their way out of doing the math. • “’I'm not going to say our finance department isn't quite stringent on trying to have us quantify the payback,’ says Parkview's Mourey. ‘But we don't want to commit to that. The time we spend measuring that is less important than the time spent redesigning process, training people and improving care. IT is just a tool.’” (Haugh, Richard.)

8. To ROI or not to ROI? That is the question. (cont.) • As demonstrated with the ROI calculators in the earlier slide, ROI is interpreted differently for each different project. That leads to frustration in some business managers and a sense that the exercise is a waste of time. • “Jupiter Media Metrix says most companies use ROI metrics that are inconsistent from one project to another, making it ‘nearly impossible to correctly choose which projects should be funded and which should be killed’”(Lewis, David. Koller, Mike.)

9. To ROI or not to ROI? That is the question. (cont.) • Others find that there are other factors involved in measuring success. • “Christus Health, Irving, Texas, measures its return on investment using traditional financial benchmarks, but has added other indicators: employee job satisfaction, accessibility of clinical information by physicians whenever and wherever they want, patient satisfaction score. Such fuzzy measures may drive CFOs crazy, but they're here to stay.” (Haugh, Richard.)

10. Net Honesty • “New research and anecdotal evidence suggest that managers may be fudging the numbers-or at least evaluating their projects less than rigorously.” • "I would never have believed there were so many liars," says Howard Rubin, a Meta Group research fellow. "They don't even know how to measure their profitability." Carol Baroudi, an analyst at consulting firm Baroudi & Associates, calls such results "shockingly positive." (Lewis, David. Koller, Mike.)

11. Net Honesty (cont.) • “’If you ask the project manager to measure their own ROI, you can't trust the results,’ said Dave Taylor, a Jupiter analyst.” (Lewis, David. Koller, Mike.) • “IT managers may be playing loose with their return on investment numbers to get pet projects off the ground.” (Preston, Robert.)

12. Best Practices • An ROI evaluation, once one decides to implement it, should follow Best Practices. • “Accurate ROI Requires Impartiality” (Lewis, David. Koller, Mike.) • “Respectable ROI (is) qualitative and quantitative.” (Preston, Robert.) • “Companies should set some concrete expectations for IT investments before approving them.” (Hayes, Mary. Chabrow, Eric, et. al.)

13. Best Practices (cont.) • “’Oversight is another way to ensure that projects bring returns,’ says Allan Woods, CIO at Pittsburgh-based Mellon Financial Corp.” (Solomon, Melissa.) • “For those companies that do ROI studies, Jupiter recommends tapping outside parties to get objective results. Or they can use internal people with expertise in policy, finance, technology and business processes.” (Lewis, David. Koller, Mike.)

14. Best Practices (cont.) • “ROI relies on each department's functioning within the total organization.” (InternetWeek) • “One way to ensure e-business ROI is to break projects into manageable chunks, with milestones to be achieved before for the next phase can proceed. This approach ensures that each phase delivers productivity, cost or revenue improvements, even if future elements are stopped in their tracks.” (Smith, Tom.) • “ROI calculation has to pass the toughest of all tests, the ‘CFO test.’” (Joachim, David.)

15. It pays to foster customer loyalty: ROR • A twist on the old ROI evaluation is a new concept called ROR (Return on Relationship). The theory is that long term planning involves keeping a customer base, not just acquiring one. • “ROR allows businesses to maximize profit by turning customer data into business strategies, and customer relationships into equity.” (Amdocs.com) • “To be successful, businesses must deliver superior customer service, building a seamless and consistent brand experience across all touch points.” (Amdocs.com) • touch points are defined as any place a customer come into contact with the company; i.e. website, advertisement, store, etc.

16. It pays to foster customer loyalty: ROR (cont.) • “To protect their investment in customers, savvy businesses concentrate on the time value of customers. They focus on the potential dollar value of repeat and referral business both over time (through the customer's shifts in age, spending patterns, and other demographic changes) and over a broad range of products and services. To keep customers coming back, these companies are shifting from transaction-based customer interactions to relationship-based customer interactions.” (Amdocs.com)

17. e-Loyalty: courting the customer • The buzz word on the street is e-Loyalty: incorporating the ideas of ROR into e-commerce. • e-Loyalty consists of pleasing the customer to the extent that not only will he or she come back to your site again, but that they will always come back to your site. • “In the rush to build Internet businesses, many executives mistakenly concentrate all their attention on attracting customers rather than retaining them.” http://www.loyaltyrules.com/loyaltyrules/index.html

18. e-Loyalty: courting the customer (cont.) • “Customer loyalty is an economic necessity: acquiring customers on the Internet is very expensive, and unless customers stick around and make lots of repeat purchases, profits will remain elusive.” http://www.loyaltyrules.com/loyaltyrules/index.html • “Loyalty is starting to mean something deeper: a focus on mutually beneficial relationships.” http://www.loyaltyrules.com/loyaltyrules/index.html

19. Use it or Lose it:ROI and Usability • ROI is essential to justifying Usability projects. • There is very little on the Web about the correlation between the effort that goes into rationalizing a Usability lab’s existence and corporate down-sizing. • While it is generally understood that the Usability folks are among the first to go when layoffs are on the horizon, It is not necessarily talked about in the literature. It is a taboo subject. • Therefore, it is necessary to state here that a Usability professional needs to stockpile ammunition against his/her extinction, of which information part should be ROI data.

20. Conclusion Determining whether or not to quantify profitability is perhaps a more difficult task than you might have predicted. IMHO, there is a fine line between financial prognosticators and those gazing into a crystal ball. What we do know, is that business can change rapidly without notice. Prepare for change.

21. References • Web resources • http://www.amdocs.com • http://business.broadcast.com/roi/marketing_roi.asp • http://www.businessplans.org/Glossary.html • http://www.businessweek.com/magazine/content/01_44/b3755609.htm • http://bwnt.businessweek.com/roi/enter.asp • http://www.loyaltyrules.com/loyaltyrules/index.html • http://www.smallbiz.nsw.gov.au/textonly/resources/tools/glossary.html#R • http://xproi.forbes.com/

22. References, (cont.) • Print Resources • Haugh, Richard. “ROI: Return On Investment? Or Relying On Instinct?” H&HN: Hospitals & Health Networks, Jan2002, Vol. 76 Issue 1, p38, 3p • Hayes, Mary; Chabrow, Eric; Khirallah, Diane Rezendes; Maselli, Jennifer; Heun, Christopher T. “Making Sure ROI Measures Up.” InformationWeek, 8/6/2001 Issue 849, p34, 6p, 1 graph, 3c • Joachim, David. “ROI Isn't Only About Crunching Numbers.” InternetWeek, 06/18/2001 Issue 866, p24, 1/3p, • 1p, 1c

23. References, (cont.) • Print Resources, cont. • Lewis, David; Koller, Mike. “ROI: Little More Than Lip Service. (cover story)” InternetWeek, 10/1/2001 Issue 880, p1, 2p, 3 charts, 1c • Preston, Robert. “ROI Has Merit, But Don't Succumb To Hype.” InternetWeek, 10/1/2001 Issue 880, p9, 1/2p • Smith, Tom. “An Approach To ROI That's Worth Heeding.” InternetWeek, 04/16/2001 Issue 857, p18, 1/3p • Solomon, Melissa. “ROI: It's About People, Not Numbers.” Computerworld, 1/14/2002, Vol. 36 Issue 3, p26