Experimental Design: Part II

1. Experimental Design: Part II MAR 6648: Marketing Research February 8, 2010

2. Overview • Let’s review experiments! • What can experiments do that other techniques can’t? • How do experiments get implemented in marketing?

3. An experiment

4. An experiment? • The owner of two McDonalds franchises here in Gainesville wants to see if transactions run more quickly if he uses both drive-thru windows or only one. He picks one restaurant to use both windows at all times for a month, and the other he has closed at all times for a month. He finds that the drive-thru that uses both windows has notably faster service times.

5. An experiment?

6. An experiment?

7. A Case Study in Causal Investigation • Broken Windows Theory (Kelling & Wilson, 1982) • “If a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken” • Implication: People see the building as the sort of place where it is OK to break windows • Implication #2: If you fix the window, people see this as a place where people behave nicely • Broad sweeping implication: If you eliminate petty crimes, people will stop perpetrating major crimes

8. A Case Study in Causal Investigation • Primary concern: The theory is only interesting if it is causal. Consider the alternatives: • 1. Reverse causation: When you reduce major crimes, people are also less likely to commit minor crimes (e.g., if I stop people from stealing cars, I also stop them from breaking windows) • 2. Third variable: If I remove all the criminals from circulation, there will be fewer broken windows AND fewer stolen cars • Now, given the methodologies we have thus far, how would you try to evaluate the theory?

9. A Case Study in Causal Investigation • Possibilities: • Qualitative Data • Survey Design • Observational Research • Archival Research/Data Mining

10. Archival Analysis: • Crime in New York City Primary Conclusion: “… an increase in the size of the police force generates a decrease in robberies and burglaries.” Corman and Mocan, 2000

11. A Case Study in Causal Investigation • Possibilities: • Qualitative Data • Survey Design • Observational Research • Archival Research/Data Mining • Experimental Design • Experimentation is the conscious manipulation of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured. • The variable being manipulated is called the independent variable (a.k.a. cause). • The variable being measured is called the dependent variable (a.k.a. effect). • Elimination of other possible causal factors: i.e., the research design should rule out the other factors (exogenous variables) as potentially causal ones. • This is typically done through random assignment to condition

12. Experimental Design: Example 1 • Independent Variable • “Policing Disorder” vs. Control • Dependent Variable • Service calls for five serious crimes

13. Control Policing Disorder Braga and Bond, 2008

14. Results But what limitations do we see in this?

15. Experimental Design Experimenters attached a paper flyer to each bicycle and recorded whether or not people dropped the flyer or took it with them. Control Disorderly Setting Keizer, Lindenberg, & Steg, 2008

16. Results

17. Further Experiments But what are the shortcomings of this design?

18. Key Points • Experimentation is necessary to infer causality • A poorly designed experiment will not allow you to infer causality • A good experiment should achieve: • High internal validity through appropriate choice of experimental design. • High external validity by keeping the experimental setting as close to the real marketing environment as possible. • There is a trade-off here… • A good control group is often a key requirement of a good experimental design.

19. How should experimentation be used in marketing? Focus is on detecting causal relationships between variables ? New Customer Service Program Customer Satisfaction

20. Causal Research in Marketing • Many examples of the need for causal effects in Marketing • 4P’s alone... • How do we actually identify causal effects? Key tool: Experiments

21. How to Run Experiments? • How do we actually run experiments? • In the sciences there is a long history of lab experiments • In a lab it is relatively easy to control external conditions that might affect the validity of the experiment • What is the situation in Marketing?

22. Experiments in Marketing • Usually take the form of a comparison between a test and (at least one) control group • Experiments are frequently run as field experiments

23. Example: Price Experiment • A catalog retailer selling women’s apparel • Conducted price experiment to estimate demand curves • One control group and 4 test groups • Each group consisted of 15,000 randomly sampled customers

24. Version 1 Version 2 Version 3 Version 4 Version 5

25. The Concept of Validity • Internal Validity: • Refers to the ability of the experiment to unambiguously show a cause and effect relationship, i.e., to what extent can we attribute the effect that was observed to the experimental variable and not other factors? • External Validity: • Refers to the extent to which the results of the experiment can be generalized from the experimental environment to the environment of the decision maker; i.e., the real world • There is a trade-off between internal and external validity, from a managerial perspective.

26. Test Markets • Many uses: • Controlled introduction of a new product • Change of pricing strategy • Change of product design • Choose representative markets • Often as long as one year • Expensive – but highly informative • Three Types: • Simulated Test Markets • Controlled Test Marketing • Sell-In test Marketing

27. Simulated Test Marketing (STM) • (a.k.a Laboratory Test Markets) • Simulates an actual test market to estimate • initial purchase rates • ultimate repeat purchase rates • Advantages: Compared to true test markets… • Fast (3 months) • Relatively cheap ($250,000) • Flexible • Impressive accuracy rates • Limitations • assumes preference data and purchase/repurchase decisions are valid predictors of what would actually happen in the market place • convenience sample • attrition • Despite this, laboratory test markets are one of the biggest success stories in market research

28. Example: Typical STM Procedure Step 1: Recruit Qualified Shoppers Step 2: Background Questions Familiarity, Preferences, Usage • Step 4: Simulated Shopping • Respondents given money • Invited into mock/real store • Where they may buy any item Step 3: Screening of Ads Ad for Target Product + Others • Step 6: Reinterview • Contacted after few weeks • Product attitude, usage, satisfaction • Repurchase (REPURCHASE RATE) • Intended & Actual • Step 5: Debriefing • Choices recorded (TRIAL RATE) • Reasons for (non) purchase • Non buyers given free sample

29. Controlled Test Marketing • Cities for distribution is prearranged • Purchases of a panel of consumers are monitored through scanner data • Example • IRI BehaviorScan • 3,000 households in 7 cities • ID card presented to supermarket • In-store conditions - price, promotion, displays - controlled and monitored • Device on TV allows channel selection to be monitored and ads to be substituted

30. Sell-In Test Marketing • Cities where product is sold just as it would be in a national launch • Must gain distribution space • Issues • selecting the test cities • implementing and controlling the test • timing • evaluative measures • costs

31. Six U.S. Common Test Markets

32. Downsides to Test Markets?

33. Other Downsides?

35. Key Points • Experimentation is very useful in marketing to determine true causal effects of marketing decisions • Test markets are the primary example, although smaller scale experiments are common • Test markets represent a trade off between internal and external validity • They have other downsides, too

36. Summary • Experiments are awesome, right? • They can demonstrate causality, which makes them useful tools for marketers • When you design an experiment, keep in mind the balance you are looking for with regard to internal and external validity • Test markets give you a lot of external validity, which can be good and bad