1 / 37

RESEARCH DESIGN & SURVEYS

RESEARCH DESIGN & SURVEYS. TOPICS. Sampling issues The Instrument (surveys) Bias Non experimental research design Experimental vs. quasi-experimental research designs Research designs with limited power to assess cause and effect

bittner
Download Presentation

RESEARCH DESIGN & SURVEYS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RESEARCH DESIGN & SURVEYS

  2. TOPICS • Sampling issues • The Instrument (surveys) • Bias • Non experimental research design • Experimental vs. quasi-experimental research designs • Research designs with limited power to assess cause and effect • Research design with more power to assess cause and effect

  3. Sampling Issues • The population • The Universe • Census • True parameters • The sampling Frame • PROBABILISTIC • Random • Stratified • NON PROBABILISTIC • Convenient • Snowballing, etc. The Sample Does the sample represent the entire population? Can we make any inference?

  4. Making Inferences • Taking only a sample of data from a population becomes necessary as a feasibility/cost issue. • However, even when we seem to have a full population, treating as such means we also assume there is no error in the data coding. • Hence, some believe we should always use inferential statistical methods when working with surveys or other quantitative studies.

  5. Avoiding Biased samples • A sample is valid when it reflects the population from which it is drawn. • The central limit theorem has been proven to show that the larger the sample relative to the population from which it is drawn, the more it will reflect the parameter of the population. • Hence, by definition larger samples are more likely to be accurate than smaller samples.

  6. Surveying Methods The sample The instrument (aggregate opinion ) The data gathering method

  7. Bias • Systematic bias occurs, and is a huge problem, when bias is incurred directly from the manner in which a survey is collected. • For example, would a survey evaluating perceptions of Democrats be unbiased if conducted exclusively outside a joint meeting of Christian Evangelicals and the NRA?

  8. Bias • Sampling error may also occur, and if minimal not a problem, to the degree that we can not be sure whether we fully captured the parameter of a population. • Nonsampling error results from many other problems of research design, such as poorly worded questions or non-responses of certain subjects. There is always potential error in every empirical project.

  9. Data Data Processing Codifying data The raw data

  10. Data Analysis and Findings Communication Data Analysis Findings Communication

  11. Sample Distributions • Though covered more thoroughly in a few weeks, a sample distribution would be the way a statistic would be distributed if drawn from many samples of a population. • Essentially, this is hypothetical and a best guess of what the population parameters may be if we cannot fully know that in reality.

  12. Interviews and Questioning • Environment, demeanor, neutrality, are all necessary to gain samples of data with the least bias. • Focus groups in particular require structure through questions in the hope of excellent qualitative data without leading those who respond to certain answers, although structure may vary: Group interview or Interactive focus group. • The more the researcher needs specific answers to questions, the more they need to structure the session.

  13. Questions • Questionnaires require much skill in order to avoid error. • Avoiding biases of those collecting the information in types of questions. • Avoid value-laden or biased words that act as queues or triggers for certain political/social groups. • Use clear standard language but also make sure your audience understands the words. Example, asking high schools students whether they are taught civics.

  14. Structure of Questions • Open v. Closed. As with focus groups, the more you structure answer options the more you can have answers to specific questions. Open-ended questions are good for a full range of respondent views.

  15. Additional Problems to be aware of • Instrumentation: How the instrument (e.g. survey) is conducted. For example, a survey through internet will exclude those without access to such technology. • Selection: Selection effects based on who is and who isn’t included in the experiment or survey. Sampling and instrumentation problems.

  16. Additional problems of survey instrumentation or experiments A rival hypothesis is an alternative explanation that may invalidate what we think as cause and effect. There are a few examples: • History: Events that happen in the outside world during the experiment. Ask opinions about civil liberties after 9/11 compared to before, when 9/11 is not the focus of the study. • Maturation: Acquisition and processing of information by subjects is not constant through time. Could be a real problem for some panel studies.

  17. Additional Problems to be aware of • Rivalry between treatment and control group: How aware are the subjects of the experiment. In a survey, this is called the Effect, where respondents try to guess at or please the person asking the questions. • A combination (interaction) between the above threats to validity. (e.g. instrumentation-history)

  18. Research Design • I want to study the relationship between level of development and income distribution in developing countries. • I need a hypothesis • I need a strategy (research design) to undertake hypothesis testing • I need to be aware that there is not a perfect strategy (research design) but I need to do the best possible research.

  19. The hypothesis • The economic development theory suggests that as countries get richer income distribution worsens but eventually income distribution improves. HIGH GAP DIST. LOW LOW HIGH LEVEL OF DEVELOPMENT

  20. Non experimental research CROSS SECTIONAL TREND LONGITUDINAL COHORT PANEL Source: Sampieri et al. 1998:192

  21. NON EXPERIMENTAL RESEARCH • CROSS SECTIONAL: The researcher would take a sample of countries with different levels of development (low, medium, high) and analyze the match between the theory and facts. • This approach is the weakest to determine causality, but strong in generalizing.

  22. GINI CONCENTRATION RATIOS GINI=0 TOTAL EQUALITY GINI =1 TOTAL INEQUALITY

  23. NON EXPERIMENTAL RESEARCH • LONGITUDINAL: The researcher would select a country or countries that have move from lower to higher levels of development (eg. USA, Canada, etc.) and determine whether income distribution over time followed the trend suggested by the theory • Causality can be infer but lacks detail about the true causes as well as weak on generalization.

  24. LONGITUDINAL • TENDENCY: EVOLUTION OF INCOME DISTRIBUTION OVER TIME IN MEXICO. • COHORT: EVOLUTION OF INCOME DISTRIBUTION OVER TIME OF MEXICAN WOMEN BORN AFTER 1960. • PANEL: EVOLUTION OF INCOME DISTRIBUTION OF THE SAME GROUP OF WOMEN OVER TIME (MARIA, SUSAN, BRENDA, JOAN, GLORIA & ISABEL)

  25. CASE (S) STUDY • The researcher will take a case (s) study and analyze in depth what factors (policies, institutions, political systems, culture, etc.) are associated with improving income distribution in a given country. • Why is income distribution better in Chile than in Mexico and Brazil despite having the same level of development? • Improves details and understanding of causes and effect but lacks generalization.

  26. Experimental vs. Quasi-experimental Research Designs • Experimental research design: The researcher has control over the experiment in terms of sample selection, treatment, environment, etc. • Experimental designs are typical in psychology, medicine, education, etc. • Quasi-experiments: The researcher does not have control over the experiment, rather the experiment occurs in a “natural” setting. • Quasi-experimental design are typical in economics, sociology, public administration, urban planning, political sciences, etc.

  27. RESEARCH DESIGN 0t = Observation in time t of experimental group X = Treatment 0c = Control group

  28. Research design with limited power POST-TEST ONLY X O 1 POST-TEST WITH CONTROL GROUP X O1 Oc

  29. PRE-TEST POST-TEST O1 X O2 • PRE-TEST POST-TEST WITH CONTROL GROUP O1 X O2 Oc Oc

  30. Research designs with more causal power • CONTROL WITH MORE OBSERVATION IN THE PRETEST O1 O2 O3 X O4 Oc Oc Oc Oc

  31. PRE-TEST POST-TEST REMOVING THE TREATMENT O1 X O2 X O3 X 04

  32. CHANGES TO LOOK FOR • CONVERGENCE-DIVERGENCE • Positive change in the treatment group without change in the control group Treatment Control Treatment Pre-test Post-test

  33. Divergence • Positive increments at a different rate Treatment Control Post-test Pre-test

  34. Convergence • The treatment group catches up with the control group Treatment Control Pre-test Postest

  35. Cross pattern • The treatment group overpass the control group Treatment Control Pre-test Post-test

  36. Research design with more power (time series) • Pre-test post-test O1 O2 O3 X O4 O5 O6 • Pre-test post-test with control group O1 O2 O3 X O4 O5 O6 Oc Oc Oc Oc Oc Oc

  37. Changes to look for in time series No effect Change on the rate or slope Change on the intercept X POST-TEST TIME PRE-TEST

More Related