Danilo Yanich School of Public Policy & Administration Center for Community Research & Service

UAPP 702: Research Design for Urban & Public PolicyClass NotesBabbie, The Practice of Social Research, Chaps.4&5 Danilo Yanich School of Public Policy & Administration Center for Community Research & Service University of Delaware

Ch. 4: Research DesignPurposes of Research • Exploration:typically done for three purposes: • to satisfy the researcher’s curiosity and desire for better understanding • to test the feasibility of undertaking a more extensive study • to develop the methods to be employed in a subsequent study • Description:describe situations and events • Census is good example of descriptive research • Explanation:the “why?” of events, situations, behavior, attitudes, etc.

Logic of Nomothetic Explanation • Nomothetic explanation refers to the accounting of many variations in a given phenomenon • In contrast to… • Idiographic explanation that seeks an in-depth understanding of a single case

Criteria for Nomothetic Causality • Correlation: the variables must be correlated • Time order: the cause takes place before the effect • Non-spurious: the variables are non-spurious • Spurious relationship: a coincidental statistical correlation between two variables, shown to be caused by some third variable

Correlation • Some relationship---or correlation—between the variables must exist before we can consider causality • Correlation: empirical relationship between two variables such that… • Changes in one are associated with changes in the other • Particular attributes of one variable are associated with particular attributes of the other

False Criteria for Nomothetic Causality • Complete causation • Causation is incomplete and probabalistic • Exceptional cases • Exceptional cases do not disprove general overall pattern of causation • Majority of cases • Causal relationship may be true even if they don’t apply to the majority of cases • Example: lack of supervision & delinquency… • as long as unsupervised juveniles are more likely to be become delinquent, social science can say there is a causal relationship

Necessary and Sufficient Causes • Necessary cause represents a condition that must be present for the effect to follow • Ex: must be female to become pregnant • Ex: must take college courses to get a degree…but… • Simply taking courses is not a sufficient cause • Must take the right ones

Necessary and Sufficient Causes • Sufficient cause represents a condition that, if it is present, guarantees the effect in question • Not saying that sufficient cause is only possible cause for effect • Ex: skipping exam in course would be sufficient cause for failing, but students could fail in other ways, too • So, cause can be sufficient but not necessary

Units of Analysis • No limit to what or whom can be studied • Common social science units of analysis: • Individuals • Groups • Organizations • Social artifacts. • Important:what you “call” a given unit of analysis is almost irrelevant—but you must be clear what that unit “is” • Are you studying marriages or marriage partners? • Crimes or criminals? • Historic buildings or the process for selecting them? • Efficiency of the hotel or the satisfaction of customers?

Ecological Fallacy • Ecological in this context refers to groups or sets or systems, something larger than individuals. • Fallacy is to assume that something learned about such a unit says something about the individuals comprising that unit. • Babbie uses example of data that shows which precincts supported a female candidate… • Some census data for each precinct that shows that precincts with relatively young voters gave her more support • Could not assume that young voters were most likely to support a female candidate... • That is…we cannot assume that age affects support • The unit of analysis was the precinct, NOT the individuals in the precinct

Reductionism • Tendency to explain everything in terms of a particular, narrow set of concepts • Remember paradigms that predispose researcher to a particular explanation • Definition of order by coercion, shared values, exchange

Ch. 5: Conceptualization, Operationalization & Measurement • Conceptualization • The refinement and specification of abstract concepts • A specific agreed-upon meaning of the concept under study • Ex. “compassion” does not exist in any sense that we can measure in an objective sense • Operationalization • The development of specific research procedures (operations) that will result in empirical observations representing those concepts in the real world

What social scientists measureTable 5.1, p. 129

Indicators and Dimensions • Indicator • An observation that we consider as a reflection of the variable under study • Ex: attending church as an indicator or religiosity • Dimension • A specific aspect of a concept • Ex:action aspects of religiosity (attending church, giving money) and contemplative aspects (prayer, etc)

Operational definition • Specifies precisely how a concept will be measured • Operationalization • The development of specific research procedures (operations) that will result in empirical observations representing those concepts in the real world

Progression of measurement steps • Conceptualization ↓ • Nominal definition ↓ • Operational definition ↓ • Measurements in the real world “conceptual funnel”

Progression of measurementTable 5.2, p. 136

Operationalization Choices • Range of variation: Must be clear about the range of variation in any concept that interests you. • Babbie uses as an example studying certain ranges of income, i.e., using $100,000 as the floor for the highest income group rather than a higher amount • Attitudes toward nuclear power...might use a range of “favor it very much” to don’t favor it at all”... • But, that would leave out the people who are opposed to it. • Variations between extremes: Get as much detail in the measurement as possible. • Can always aggregate data (that is, combine precise attributes) into more general categories... • But can never separate out any variations that were lumped together during observation and measurement.

Two important qualities of variables:Exhaustive & Mutually Exclusive • Exhaustive:For the variable to have any utility in research, must be able to classify every observation in terms of one of the attributes composing the variable • Babbie uses example of political party affiliation that specifies just Democrat or Republican… • When that would leave out others who do not identify with either • Use “other” or “no affiliation” to make it exhaustive. • Mutually exclusive:Must be able to classify every observation in terms of one and only one attribute. • Babbie uses defining employed and unemployed in such a way that nobody can be both at the same time • Refer to Graber “social type” variable...farmer, n’er-do-well, etc. & Family Court gender variable.

Levels of measurement (NOIR) • Nominal:variables whose attribute have only the characteristics of exhaustiveness and mutual exclusivity • Examples: gender, religious affiliation, birthplace, etc • Ordinal: variables with attributes that can logically rank-order; the different attributes represent relatively more or less of a variable. • Examples: social class, conservatism, alienation, prejudice, “coolness” • Interval: variables in which the actual distance separating them can be expressed in meaningful standard variables • Examples: temperature, intelligence tests • Ratio: variables that have all of the characteristics of the previous levels of measurement AND are based on a true zero point • Examples: age, length of residence in a home, duration of news story, etc.

Implications of levels of measurement • Requirements of analytical techniques: • Certain analytical techniques require variables that meet certain minimum levels of measurement • Must plan analytical techniques according to the level of measurement at which you will gather your data. • Should anticipate drawing research conclusions appropriate to the levels of measurement used in your variables. • Caution: Seek highest level of measurement possible because... • Although you can reduce a ratio measure to ordinal... • You cannot convert an ordinal measure into a ratio measure... • It is a one-way street

Criteria of measurement quality • Precision and accuracy • Precision=fineness of the distinction made between the attributes that compose a variable • Saying that a woman is “43 years old” is more precise than saying that she is “in her forties” • Degree of precision is dictated by your research requirements • If your research question does not require her precise age, then additional effort to gather it precisely is wasted • However, if your needs are unclear, be more precise rather than less • Do not confuse precision with accuracy • Saying that someone was born in “Stowe, VT” is more precise than born in “New England” • But…suppose the person in question was born in Boston • The more general description of “New England” is less precise, but accurate

Criteria of measurement quality, p.2 • Reliability • Whether a particular technique, applied repeatedly to the same object, yields the same result every time • Example: Measuring weight using two different persons’ estimates versus a scale • Reliability does NOT ensure accuracy • Suppose the scale is set five pounds too light • Measurement would be reliable each time, but it would also be wrong each time • Ways to cross-check the reliability of measures • Test-retest method • Split-half method • Using established measures (Miller book is useful here) • Reliability of research workers

Criteria of measurement quality, p.3 • Validity • Refers to the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration • Social research does operate on agreements about the terms we use and the concepts they represent

Criteria of measurement quality, p.4 • Testing validity • Face validity — empirical measures that jibe with our common understanding of a concept • Ex. Grievances & worker morale • Criterion-based validity — based on external criterion • Ex. College board scores & student success in college

Criteria of measurement quality, p.5 • Testing validity • Construct validity — based on logical relationships among variables • Ex. Marital fidelity & marital satisfaction • Content validity — refers to how much a measure covers the range of meanings in a concept • Ex: test of math ability can’t be limited to addition alone

Criteria of measurement quality, p.6 • Tension between reliability & validity • Often a trade-off between the two because resources limit the research • Ex. Measuring morale by spending days on assembly line talking w/ workers seems a more valid measure of morale than counting grievances • If there is no clear agreement on how to measure a concept…measure it several ways • Ex. Recidivism, court success, hotel efficiency, etc. • Concept does not have any meaning other than what we give it. • Only justification to give concept a particular meaning is utility

Basic Research Outline* • The Social Problem • Present a clear, brief statement of the problem, with concepts defined where necessary • Show that the problem is limited to bounds amenable to treatment or test • Describe the significance of the problem with reference to specific criteria Source:Miller, Delbert C. 1991. Handbook of Research Design and Social Measurement, 5th Edition. Newbury Park: Sage Publications, pp. 15-16.

Basic Research Outline, p.2 • The Theoretical Framework • Describe the relationship of the problem to a theoretical framework • Demonstrate the relationship of the problem to previous research • Present alternate hypotheses considered feasible within the framework of the theory.

Basic Research Outline, p.3 • The Research Question/Hypotheses • Clearly statethe research questions or the hypotheses selected for test. (Null and alternate) • Indicate the significance of test hypotheses to the advancement of research and theory. • For policy research state how research might inform policy. • Define concepts or variables (preferably in operational terms). • Describe possible mistakes and their consequences. • Note seriousness of possible mistakes.

Basic Research Outline, p.4 • Design of the Experiment or Inquiry • Describe ideal design or designs with particular attention to the control of interfering variables • Describe selected operational design • Specifystatistical tests including dummy variables

Basic Research Outline, p.5 • Sampling Procedures • Describe experimental and control samples • Specify method of drawing or selecting sample

Basic Research Outline, p.6 • Methods of Gathering Data • Describemeasures of quantitative variables showing reliability and validity when these are known. Describe means of identifying qualitative variables • Includedescriptions of questionnaires or schedules • Describe interview procedure • Describe use made of pilot study, pretest, trial run.

Basic Research Outline, p.7 • Working Guide • Prepare working guide with time and budget estimates • Estimate total person-hours and cost

Basic Research Outline, p.8 • Analysis of Results • Specify methods of analysis

Basic Research Outline, p.9 • Interpretation of Results • Discuss how conclusions will be fed back into theory…OR… • Inform policy/practice.

Basic Research Outline, p.10 • Publication or Reporting Plans...Communication Plans • Monograph, Executive summary • Testimony to policy makers. • Presentations to institutions, non-governmental agencies, media, public. • Journal publication

The Policy Research Process* *D. Yanich example using model in: Miller, Delbert C. (1991). Handbook of Research Design and Social Measurement. Fifth Edition. Newbury Park, CA: Sage Publications, pp15-16

The Policy Research Process, p.2

Danilo Yanich School of Public Policy & Administration Center for Community Research & Service