Introduction to Research: Measurement

1. Introduction to Research: Measurement Part 2 of 3 By: Danielle Davidov, PhD & Steve Davis, MSW, MPA

2. OUTLINE • 1) Threats to research studies • 2) Steps in the research design process • 3) Identifying and defining variables • 4) Validity and reliability of measurement

3. Research Design • Starts Afterthe research question has been developed and refined • The who, what, when, where, and how of research • It comprises the Materials and Methods and Limitations sections of publications

4. Why is design important? • The goal of research design is to provide the most valid and correct answer to the question • i.e., we want to make sure we are “doing it right” • This is done by minimizing the threats to the soundness of your study’s conclusion(s): • CHANCE • BIAS • CONFOUNDING

5. Study Threats: Chance • The threat that the study’s findings are merely the result of random processes (chance) • i.e., the findings are a “fluke” • We can’t do much to control random error • Also referred to as: • Type 1 Error • Random Error • Unsystematic Error

6. Study Threats: Bias • The threat that the study’s results are due to an unfair preference given to one group or a set of outcomes in a study • We can try to control bias in our study design and subject recruitment • Also referred to as: • Systematic Error

7. Study Threats: Confounding • The threat that the association or relationship observed in the study is influenced by or related to another variable • We can control for this in our study design, subject recruitment, and data analysis techniques

8. Steps in Research Design • We try to minimize the three main threats during all stages of design process, which are: • 1) Identifying and Defining Variables* • 2) Selecting Measurement Methods* • 3) Selecting (Sample) Subjects • 4) Selecting a Research Design • 5) Establishing an Analysis Plan *We will be talking about steps 1 and 2 in this presentation

9. Identify & Define variables • What do you want to measure? (Identify) • Ex) Patient satisfaction levels with ultrasound vs. history and physical exam only • How do you want to operationalize “patient satisfaction?” (Define) • Ex) Answers of “Good”, “Very Good”, or “Excellent” on a survey given to patients about the care they received in the emergency department

10. Identification & Definition of Variables Classifying Variables: • Independent Variable • The variable that has an effect on or influences the dependent variable. This is the FACTOR/INTERVENTION • i.e.) History and Physical Exam or Ultrasound + H & P • Dependent Variable • The variable that is affected by, or dependent upon, the independent variable. This is the OUTCOME • i.e.) Patient Satisfaction

11. Identification & Definition of Variables Classifying Variables (continued) • Confounding Variables –a variable that is related to both the independent and the dependent variable • CONFOUNDER or CONTROL variable • Common confounders/controls in medical research: • Age • Gender • Race • Severity of Illness

12. Identification & Definition of Variables Controlling for Confounding Variables • Not adequately controlling for confounding variables can have disastrous consequences on your research • Identify and define as many as possible • From previous literature • From clinical observations • From theory What if we didn’t consider these important variables when examining the relationship between the Independent and Dependent variables???

13. Identification & Definition of Variables • Operationalizing variables • The process of defining variables in a measurable way.

14. Identification & Definition of Variables • Levels of Measurement (NOIR) • Nominal • Ordinal • Interval • Ratio Lower Higher

15. Nominal Level Data • Characteristic data that cannot be rank ordered • This data is “categorical” – made up of “categories”, not “levels” or “increments” • Ex) Ice cream flavors—vanilla is not “better” or “more” than chocolate • Examples: Gender, Race, Student, Marital Status, State or Country of Residence, Insurance Status, Discharge Status, etc. • Yes/No Responses are Nominal • This type of data is usually “descriptive” • Used to describe a population or sample

16. Ordinal Level Data • Data that can be rank ordered but that do not have measurable distances between each level of rank • Likert Scales - Strongly Disagree to Strongly Agree • Class rank - Freshman, Sophomore, Junior, Senior; PGY-I, PGY-II, PGY-III • Degree of illness: None, Mild, Moderate, Severe • Senior is a higher rank than Freshman, but there is no way to quantify how much higher Senior is vs. “Freshman” or how much “more” illness those with a severe illness have compared to those with a mild illness

17. Interval/Ratio Data • Data that can be ordered and that have a measurable distance between each level • The Interval Scale - Distances between positions are equal, but "0" is an arbitrary assignment. For example, with temperature, each degree is equally distant from another, but "0" does not mean that there is no temperature. It is simply a reference point on the scale. • The Ratio Scale- All positions are equally distant and "0" means that the value is truly "0". If you have "0" money, you have none. But if you have $200, you have twice as much as a friend who has $100. • Examples of Interval/Ratio Data: • Age • Height • Weight • Many Clinical Serum Levels • Blood Pressure

18. Levels of Measurement and Power • Defining variables at higher levels of measurement allows the use of statistical tests that have more Power • Power = the probability of finding a true relationship of difference if it genuinely exists • It is usually better to collect data at higher levels of measurement and then collapse into categories later • Ex) Age • What is your age? ____ (best) vs. • What is your age? vs. • 18 – 25 • 26 – 35 • 36 – 45 etc. vs. • Under 40 & over 40

19. Selecting a Measurement Method • Once you have defined and operationalized your research question’s variables, you must decide how to measure them and/or what measurement tool you will use. • There are two forms of error that we must minimize when selecting measurement methods and/or tools: • Random error (CHANCE) • Nonrandom error (BIAS AND CONFOUNDING)

20. RELIABILITY • To minimize random error we choose a tool or method that is RELIABLE • Reliability – The extent to which a measurement method or tool produces the same results over several measurements • AKA precision • Threats to Reliability • Observer error: different measurements from the same or different observers (i.e., blood pressure readings) • Instrument error: different measurements from the instrument itself due to extraneous environmental factors • Subject error: different measurements from the natural biological variability among humans

21. Assessing & Maximizing reliability • How to assess Reliability: • Repeat measurements on the same subject. • Give a survey at two different time points • Take blood pressure at two different time points • Use more than one observer. • Assess inter-rater agreement • Have two different people take blood pressure • How to maximize Reliability: • Standardize the measurement methods • Choose surveys and instruments that have been proven to be reliable • Train observers • Refine & update instruments • Repetition • Averaging the measures can cancel out error.

22. validity • To minimize nonrandom error we choose a measurement method and/or tool that is VALID • Validity – The extent to which a measurement method and/or tool measures what is sets out to measure • AKA Accuracy • Threats to Validity • Observer bias: conscious or unconscious distortion in the perception and/or reporting of the measurement • Subject bias: bias-distortion of self-reported measurements due to subjects beliefs and biases • Hawthorne Effects and Social Desirability • Instrument bias: consistently biased or inaccurate measurements due to such things as worn parts or mechanical malfunction • Lack of a clear gold standard: No “best” instrument out there • Abstract/behavioral variables: These things are difficult to measure • Patient satisfaction, pain, quality of life, intelligence

23. Maximizing Validity • Strategies for maximizing Validity • Blinding • Ex) Do not allow physician who is taking blood pressure readings to know which subjects are receiving blood pressure medication • Deception • Ex) Do not allow subjects to know which “group” they are in • Give placebos • Instrument Calibration • Make sure instruments are working properly • Use standardized/validated surveys and assessment tools • Find these from literature searches • Usually better to use “pre-made” surveys or instruments than creating one from scratch

24. In Summary • Identify and define your variables at the VERY beginning of your study • Don’t forget your control or confounding variables! • Using higher levels of measurement is better! • Choose instruments and data collection tools that are: • RELIABLE – produce the same results over time (precise) • VALID – produce results that represent “the truth” (accurate)

25. Next steps • Go through “Introduction to Research Part 3: Sampling and Design”

26. references HulleySB, Cummings SR, Browner WS, Grady D, Hearst N, Newman TB. Designing Clinical Research. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2001:37-49 Spector PE. Research Designs. Newbury Park, CA: SAGE Publications, Inc.; 1981. ISBN: 0-8039-1709-0 http://www.research-assessment-adviser.com/levels-of-measurement.html