Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With

1. Incentives and Program Evaluation – How to Get What You Want and Measure It Properly Jodi N. Beggs Economists Do It With Models Presented to Carlson Marketing July 23, 2010 www.economistsdoitwithmodels.com

2. Two parts: Consumer behavior and incentives Data analysis and program evaluation

3. Part 1: Incentives (aka “Incentives Work…Except When They Don’t”) • When do they work? • Routine effort, direct cause and effect

4. More noise -> weaker incentives • Incentives affect both effort and worker selection • Safelite Auto Glass: • Piece-rate compensation -> Productivity increases by 44% • Half of increase due to increased productivity • Other half due to employee self-selection Source: Edward Lazear and Paul Oyer, “Personnel Economics,” Handbook of Organizational Economics, forthcoming.

5. Part 1: Incentives (aka “Incentives Work…Except When They Don’t”) • When do they work? • Routine effort, direct cause and effect • But, you get exactly what you pay for… • Non-linear incentives and timing

6. Nonlinear incentives (eg. quotas) give workers an incentive to play a timing game • If a worker has made her quota, there is an incentive to push sales to the future • Alternatively, there is an incentive to just not try for the rest of the year • If a worker has no hope of making her quota, there is an incentive to push sales future • Shirking is also a concern here • If a worker is on the cusp of making the quota, there is an incentive to hurry sales • This often corresponds with discounts that may not be in the firm’s best interest • This behavior can even happen with linear incentives if bonuses are paid yearly, quarterly, etc. Source: Paul Oyer, “Fiscal Year Ends and Nonlinear Incentive Contracts: The Effect on Business Seasonality,” The Quarterly Journal of Economics, February 1998.

7. Part 1: Incentives (aka “Incentives Work…Except When They Don’t”) • When do they work? • Routine effort, direct cause and effect • But, you get exactly what you pay for… • Non-linear incentives and timing • Distortion of multi-dimensional tasks • “Goldbricking” and quota restriction

8. A sociologist’s summer research on a factory floor: • Compensation: greater of 85 cents per day or piece rate for work turned in • “Gravy” jobs and “stinker” jobs • Piece-rate payout of 9 cents to $1.66 per day • Bimodal distribution of piece-rate earnings • One group at 45 to 54 cents (and getting the day rate of 85 cents) • One group at $1.25 to $1.34 • What is going on here? Source: Donald Roy, “Quota Restriction and Goldbricking in a Machine Shop,” The American Journal of Sociology, March 1952.

9. Part 1: Incentives (aka “Incentives Work…Except When They Don’t”) • When do they work? • Routine effort, direct cause and effect • But, you get exactly what you pay for… • Non-linear incentives and timing • Distortion of multi-dimensional tasks • “Goldbricking” and quota restriction • Gaming the system in general

10. I Can’t Imagine This Is Profitable For Amazon… Source: http://www.filleritem.com

11. Part 1: Incentives, continued • Incentive strategy and framing also matters • Framing

12. “To increase transaction compliance, marketers sometimes temporally reframe the cost of a product from an aggregate one-time expense to a series of small ongoing expenses, often in spite of the fact that the physical payments remain aggregated.” • “Pennies-a-Day” strategy • How does this affect incentives? Source: John T. Gourville, “Pennies-a-Day: The Effect of Temporal Reframing on Transaction Evaluation,” The Journal of Consumer Research, March 1998.

13. Observation: People dislike losses more than they like equivalent gains • Loss aversion and reference-dependent utility • How can managers and markets use this observation? • Long-term effects

14. Part 1: Incentives, continued • Incentive strategy and framing also matters • Framing • Guaranteed versus probabilistic rewards

15. In general, people are risk averse…but consider the following choices: • $1 or a 1/2 chance of $2 • $1 or a 1/3 chance of $3 • $1 or a 1/5 chance of $5 • $1 or a 1/10 chance of $10 • $1 or a 1/100 chance of $100 • $1 or a 1/1000 chance of $1000 • $1 or a 1/1,000,000 chance of $1,000,000 • Not so clear, is it?  • This can sometimes be leveraged when designing incentives • The Publisher’s Clearing House seems to have figured this out

16. Part 1: Incentives, continued • Incentive strategy and framing also matters • Framing • Guaranteed versus probabilistic rewards • Tournament incentives

17. Tournament incentives are payouts based on rank order (relative performance) • Incentives get stronger as spread between payouts for adjacent places increases • However, can have “underwater option” problem for low and average performers

18. Part 1: Incentives, continued • Incentive strategy and framing also matters • Framing • Guaranteed versus probabilistic rewards • Tournament incentives Follow up: In-kind incentives can sometimes be powerful, contrary to what traditional economic theory would predict. This is likely because people tend to artificially restrict their consumption as a means of self-control, so a gift that the person likes but wouldn’t buy herself can be more satisfying than the equivalent amount of cash. For more, see “Mental Accounting Matters” by Richard Thaler.

19. Part 1: Incentives, continued • When don’t incentives work? • Work involving cognition and/or creativity • Tunnel vision and suppression of intrinsic motivation

20. Output-based incentives can inhibit “out-of-the-box” thinking • Eg. evidence shows that commissioned art is viewed as less creative • Output-based incentives can also inhibit intrinsic or moral incentives and have perverse effects • Day care center – fine for late parents actually increased tardiness • NIS 10 per child for delays over 10 minutes • Fine added to monthly bill • Increase didn’t go away once fine was abolished • “Pay enough or don’t pay at all” Source: Daniel Pink, Drive; Uri Gneezy and Aldo Rustichini, “A Fine Is a Price,” The Journal of Legal Studies, January 2000; Uri Gneezy and Aldo Rustichini, “Pay Enough or Don’t Pay at All,” The Quarterly Journal of Economics, August 2000.

21. Part 1: Incentives, continued • When don’t incentives work? • Work involving cognition and/or creativity • Tunnel vision and suppression of intrinsic motivation • Non-actionable output-based incentives

22. Roland Fryer: “In stark contrast to simple economic models, our results suggest that student incentives increase achievement when the rewards are given for inputs to the educational production function, but incentives tied to output are not as effective. Relative to achievement-increasing education reforms of the past few decades, student incentives based on inputs produce similar gains in achievement at lower costs. Qualitative data suggest that incentives for inputs may be more effective because students do not know the educational production function, and thus have little clue how to turn their excitement about rewards into achievement. • Angrist and Lavy find that paying for passing has a greater impact on girls’ performance than on boys’ performance Source: Roland Fryer, Jr., “Financial Incentives and Student Achievement: Evidence from Randomized Trials,” working paper, 2010; Joshua Angrist and Victor Lavy, “The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial,” The American Economic Review, 2009.

23. Part 1: Incentives, continued • What can a manager or marketer do instead? • Autonomy, mastery, purpose

24. Showing Mastery Via Badges: Huffington Post: Foursquare: Badges can be powerful in a community setting when the badges are public, noticeable and respected. It also doesn’t hurt if the badges are difficult (but not too difficult) to obtain.

25. “Endowed progress effect” • People provided with artificial advancement toward a goal exhibit greater persistence toward reaching the goal • Endow people with 2 completed steps out of 10 rather than 0 completed out of 8 and they are more likely to complete all steps • Reframing from “not undertaken” to “undertaken but incomplete” • Also decreases completion time • Effect depends on perceptions of task completion rather than misunderstanding of sunk costs • Attainability also matters Source: Joseph C. Nunes and Xavier Dreze, “The Endowed Progress Effect: How Artificial Advancement Increases Effort,” The Journal of Consumer Research, March 2006.

26. Part 1: Incentives, continued • What can a manager or marketer do instead? • Autonomy, mastery, purpose • Status

27. Status in loyalty programs: • Increasing the number of elites in the top tier dilutes perceptions of status • Adding a subordinate tier enhances status • Tiers below the second tier do not affect those at the top but can make those in the tier immediately above feel more elite • Those who do not qualify for status prefer hierarchies with multiple tiers • Status-laden labels (eg. gold and silver) on their own signal an increasingly selective hierarchy Source: Xavier Dreze and Joseph C. Nunes , “Feeling Superior: The Impact of Loyalty Program Structure on Consumers’ Perceptions of Status,” The Journal of Consumer Research, April 2009.

28. Part 2: Program Evaluation (aka “Showing Causality Is Harder Than • it Looks”) • The fundamental program evaluation problem

29. The “Post Hoc, Ergo Propter Hoc” Fallacy in a Nutshell: Source: http://xkcd.com/552/

30. “The Least Surprising Correlation of All Time” “Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids' SAT scores.” -- N. Gregory Mankiw Source: http://gregmankiw.blogspot.com/2009/08/least-surprising-correlation-of-all.html

31. Part 2: Program Evaluation (aka “Showing Causality Is Harder Than • it Looks”) • The fundamental program evaluation problem • Why life should be like a middle school science project or a clinical trial • How do researchers get around this problem? Natural experiments!

32. The effect of babies on lifetime earnings • Why not just compare earnings of those who had kids at 24 to those who had kids at 25? • Two more valid comparisons: • Women who had children at 24 to women who miscarried at 24 and had children at 25 • Women who had children at 24 by accident to women who had children at 25 • Findings: • 1 year delayed childbirth for women in 20’s = 10% higher lifetime earnings • Result goes away once women reach their 30’s Source: Amalia R. Miller, “The Effects of Motherhood Timing on Career Path,” working paper, 2005.

33. What is the effect of veteran status on earnings? • Researchers can control for what is observable- age, education, gender, race, etc. • But, can’t just compare “identical” military and non-military people • There is some difference that made some of them choose the military and others not, and this difference could be correlated with earning potential • Vietnam draft lottery allowed researchers to circumvent this selection bias • Findings: • Based on data from Vietnam War veterans, veterans earned 15% less than equivalent non-veterans • Finding limited to white males and Vietnam War, so may not be generalizable Source: Joshua D. Angrist, “Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records,” The American Economic Review, June 1990.

34. Part 2: Program Evaluation, continued • Other things to consider in program evaluation • Hawthorne Effect and Mere-Measurement Effect • Impact of feedback on subsequent behavior and loyalty • Salience and attention (pilot program effect) • Participant selection • The Counterfactual

35. For more information, go to http://www.economistsdoitwithmodels.com/Carlson