Experimental Design & Analysis

1. Experimental Design & Analysis Making a Compelling Case April 24, 2007 DOCTORAL SEMINAR, SPRING SEMESTER 2007

2. Outline • Making a compelling case • Writing and exposition • Data and evidence

3. Writing the Empirical Journal Article • Journal articles have a specific form • Introduction • Literature review • Hypotheses • Experimental overview • Experimental studies • Participants • Methods and procedures • Results • Discussion • General discussion • Limitations and future research

4. Writing the Empirical Journal Article • Introduction • Motivation • Practical problem • Resolve contradictions in literature • Extend others’ work • Literature review • Marshal findings from literature that suggest your hypothesis is plausible, sensible, inevitable • Avoid exhaustive listing of articles you haven’t read • Tap diverse literatures for “kernels” that can be re-interpreted in a way that supports your story

5. Writing the Empirical Journal Article • Hypotheses • Conceptual development • Articulation at abstract, construct level • Experimental overview • How will you test your hypotheses? • What pattern of results can be interpreted as support for your hypotheses? • How do selected variables relate to constructs?

6. Writing the Empirical Journal Article • Experimental studies • Goal • Participants • Include pertinent information – sex, age, recruitment procedures, incentive for participation, other variables that may be related to design • Methods and procedures • Written such that reader can understand participants’ experience • Results • Manipulation checks, process measures, blocking • Methods of analysis, excluded data, covariates • Report results in words, then report statistics • Discussion • What are the inferences taken from data? • What questions must yet be answered?

7. Writing the Empirical Journal Article • General discussion • What are the inferences taken from data based on all the experiments? • What is contribution on larger scale? • How do findings shed light on prior work? • Does contribution fit with current understanding of theory? • Can data speak speculatively to other questions?

8. Writing the Empirical Journal Article • Limitations and future research • Weaknesses? • Boundary conditions • Suggestions for other ways to test findings • Other independent variables • Other dependent variables • Other situational contexts • Other samples

9. Writing the Empirical Journal Article • Pulling it together • Start from an outline • Build the argument in a linear fashion • Topic sentence in each paragraph must advance the argument • Be parsimonious • Be vigilant about transitioning between conceptual and operational

10. Instead of “Show” “Confirm” “Convincing” “Prove” “Justify” “Model is correct” “The data is …” “Subjects” Use “Suggest” “Corroborate” “Supportive” “Can be interpreted” “Assess” “Model is plausible” “The data are …” “Research participants” Writing the Empirical Journal Article

11. Using Data to Build Support • Multiple studies • Building support for theory • Convergent evidence • Mediation vs. moderation • Ruling out alternative accounts • Enriching theory • A bigger story • Abstraction

12. Independent variables Measured vs. manipulated variables Chronic states (person) vs. situational states Dependent variables Attitude Likert scales Scale measures Rank order Behavior Choice Purchase Conscious vs. nonconscious measures Cognitive responses (thoughts) data Using Data to Build Support

13. Using Data to Build Support • Torturing the data • Plot data • Analyze data multiple ways • Look at multiple dependent variables • Outliers? • Covariates

14. Theory + Effects • Winer suggests that theoretical studies should be tested in settings that have external validity • Calder and Tybout believe theory tests and effects applications should not be confused

15. Theory + Effects • Which should dominate research at business schools – theory or effects research?

16. Wearout occurs Evaluation Number of Repetitions Theory + Effects

17. Experts Evaluation Novices Number of Repetitions Theory + Effects