QUANTITATIVE RESEARCH METHODS

1. QUANTITATIVE RESEARCH METHODS Irina Shklovski

2. Quantitative Research Methods • Include a wide variety of laboratory and non-laboratory procedures • Involve measurement…

5. Quantitative Research Methods • Measurement • Populations and Sampling • Random Assignment • Generalizability

6. Quantitative Research Methods • Measurement • Populations and Sampling • Random Assignment • Generalizability • Time • Cross-sectional studies & single experiments • Longitudinal studies & repeated measures

7. Quantitative Research Methods • Method • Experiments & Quasi-experiments • Behavioral Measures • Questionnaires & Surveys • Social Network Analysis • Archival and Meta-Analysis

8. What we will talk about today • Measurement • Population & Sampling • Random Assignment • Generalizability • Method • Experiments & Quasi-experiments • Questionnaires & Surveys

9. Measurement – Sampling • Specify your population of concern • Sampling • Selecting respondents from population of concern • Random sampling • Systematic selection • Stratified sampling • Convenience sampling • Snowball sampling

10. Sampling Biases • Non-response bias • Be persistent • Offer incentives and rewards • Make it look important • Volunteer bias • Some people volunteer reliably more than others for a variety of tasks

11. Random assignment • Different from random sampling • Mostly used for experiments or quazi-experiments • Protects against unsuspected sources of bias • Does NOT guarantee to balance out the differences between participants • Chance is LUMPY

12. Generalizability • How do you know that what you found in your research study is, in fact, a general trend? • Does A really, always cause B? • If A happens, is B really as likely to happen as you claim? Always? Under certain conditions?

13. Association vs. Causality Thanks toSara Kiesler for these graphs!

14. Experiments & Quasi-experiments • ex·per·i·ment • Pronunciation: \ik-ˈsper-ə-mənt also -ˈspir-\ • Function: noun • Etymology: Middle English, from Anglo-French esperiment, from Latin experimentum, from experiri • Date: 14th century • An operation or procedure carried out under controlled conditions to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law

15. Experiments & Quasi-experiments • Key feature common to all experiments: • To deliberately vary something in order to discover what happens to something else later • To seek the effects of presumed causes

16. An Experiment is • A controlled empirical test of a hypothesis. • Hypotheses include: • A causes B • A is bigger, faster, better than B • A changes more than B when we do X • Two requirements: • Independent variable that can be manipulated • Dependent variable that can be measured

17. Experiments in Research • Comparing one design or process to another • Deciding on the importance of a particular feature in a user interface • Evaluating a technology or a social intervention in a controlled environment • Finding out what really causes an effect • Finding out if an effect really exists

18. Remember • Experiments explore the effects of things that can be MANIPULATED • (but there is a caveat)

19. Types of Experiments • Randomized – units/participants assigned to receive treatment or alternative condition randomly • Quazi – no random assignment • Natural – contrasting a naturally occurring event (i.e. disaster) with a comparison condition

20. If your study involves experiments • Experimental design:Shadish W.R., Cook T.D. & Campbell P.T. (2002) Experimental and Quasi-Experimental Design for Generalized Causal Inference. Boston, Mass: Houghton Mifflin • Experimental data analysis:Bruning, J. L. & Kintz, B. L. (1997). Computational handbook of statistics (4th ed.). New York: Longman.

21. Questionnaires & Surveys • Self-report measures • Questionnaires & surveys • Interviews • Diaries • Types • Structured • Open-ended

22. Questionnaires & Surveys • Advantages • Sample large populations (cheap on materials & effort) • Efficiently ask a lot of questions • Disadvantages • Self-report is fallible • Response biases are unavoidable

23. Response biases • Relying on people’s memory of events & behaviors • Emotional states can “prime” memory • Recency effects • Routines are deceiving • Social desirability • Solution: none that are simple • Yea-saying • Solution: vary the direction of response alternatives

24. General Survey Biases • Sampling – are respondents representative of population of interest? How were they selected? • Coverage – do all persons in the population have an equal change of getting selected? • Measurement – question wording & ordering can obstruct interpretation • Non-response – people who respond differ from those that do not

25. Design is KEY • Format – booklet, printed vertical, one-sided • Question ordering – earlier questions can prime answers to later questions • Page layout – group similar items & use consistent fonts and response categories • Pre-testing – conduct think-alouds with volunteers demographically similar to expected participants

26. Common Problems • Avoid complicated & double-barrel questions • Complexity increases errors & non-response • Navigation is paramount – make sure the survey is EASY to follow • Open-ended questions • The size of the field allotted will determine the number of words • Incentive is key • BUT amount differences have little impact

27. If your study involves surveys • Designing surveys:Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys : the tailored design method (3rd ed.). Hoboken, N.J.: Wiley & Sons.Fowler, F. J. (1995). Improving survey questions : design and evaluation. Thousand Oaks: Sage Publications. • Analyzing data:Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

28. So… what? • Difference between quantitative methods is in the questions they can answer • There are a LOT of methods and even more statistical techniques • Regardless of the method, if it’s not an experiment, you CAN NOT prove causation

29. Things we did NOT talk about • Reliability assessments • Validity assessments • Statistical analysis of data • Interpretation of results