Research Design Part 2

1. Research Design Part 2 Variability, Validity, Reliability

2. Objectives • Refine research purpose & questions • Variability • Validity • External, internal, criterion, content, construct • Reliability • Test-retest, inter-rater, internal consistency, instrument

3. Variability • Different values of the independent variable • 3 sources… systematic, error, extraneous

4. Variability • 1. Systematic • Variability within the Ind. variables • Design study to maximize systematic variability • Rewards vs. management styles • Select the right sample & methods

5. Variability • 2. Error • Sampling & measurement error • Eliminate as many conditions as possible • Similar leagues, ages, abilities • Increase reliability of the instrument

6. Variability • 3. Extraneous • Control as much as possible • Not a planned part of the research • Influence outcome that we don’t want • Examples…

7. Variability • 3. Extraneous • Examples • Measure teaching techniques of V & R between 2 sections of 497 to see level of comprehension. • Measure differences between a week of TRX & Crossfit using the same fitness assessment at the end

8. Main Function of Research… • Maximize systematic variability, control extraneous variability, & minimize error variability

9. Validity & Reliability • Validity • Degree to which something measures what it is supposed to measure • Reliability • Consistency or repeatability of results

10. Validity & Reliability Hits are spread across the target but consistently missing the center. You are hitting the target consistently, but missing the center. Consistently and systematically measuring the wrong value for all respondents consistently hit the center of the target Random hits spread across the targetbut seldom hit the center Get a good group average, but not a consistent one

11. Validity & Reliability • Can a measurement/instrument be reliable, but not valid? • Weighing on a broken scale • Can a measurement/instrument be valid, but not reliable? • To be useful, a test/measurement must be both valid and reliable

12. Validity • External validity • Internal validity • Test/criterion validity • Content validity • Construct validity

13. External Validity • Generalizability of the results • Population external validity • Characteristics & results can only be generalized to those with similar characteristics • Does sample represent the entire population • Demographics • Psych experiments with college students • Use multiple PE classes, intramural leagues, sports, teams, conferences • Control through sampling

14. External Validity • Ecological external validity • Conditions of the research are generalizable to similar characteristics • Physical surroundings • Time of day • AM vs. PM • More common in testing … GRE

15. Internal Validity • Confidencein the cause and effect relationship in a study. • Strongest when the study’s design (subjects, instruments/measurements, and procedures) effectively controls possible sources of error so that those sources are not reasonably related to the study’s results. • The key question that you should ask in any experiment is: • “Could there be an alternative cause, or causes, that explain my observations and results?”

16. Internal Validity • History • Extraneous incidents/events that occur during the research to effect results • Only impacts studies across time • Attendance at football games/coaching change • Survey at IHSA @ parent behavior & parent fight breaks out in middle of survey across the gym

17. Internal Validity • Selection • If there are systematic differences in groups of subjects • Gender – boys more active than girls • Higher motivation level • More positive attitude toward study • Compare GRE scores & grad school performance between sequences • Occurs when random sampling isn’t used

18. Internal Validity • Statistical regression • If doing pre-test/post-test those scoring extremely high or low on first test will often “regress to the mean” on the second test • Scoring based more on luck than actual performance • The regression effect causes the change & not the treatment • Don’t group the high/low scores for the post-test Note: The less reliable the instrument the greater the regression.

19. Internal Validity • Pre-testing • Pre-test can increase/decrease motivation • Gives subjects opportunities to practice • Practice can be a positive so they get a true score • Pedometers (A. McGee thesis) • Instrument can make people think after the pre-test • Motivation instruments

20. Internal Validity • Instrumentation • Changes in calibration of the exam, instrument – Experimental research • Changes in observer scoring • Fatigue/ boredom • Reality judging shows • Maturation • Experimental research

21. Internal Validity • Diffusion of intervention • Experimental research • Attrition/Mortality • Subjects drop out/lost • Low scorers on GRE drop out of grad school • Coaching techniques & loss of players

22. Internal Validity • Experimenter effect • Presence, demeanor of researcher impacts +/- • Course instructor is PI • Course evals • Coach or teacher conducting the study • Teacher staying in the room when they complete PAQ-C • Subject effect • Subjects’ behaviors change because they are subjects • Subjects want to present themselves in the best light • Hawthorn effect

23. Test/Criterion Validity • Degree to which a measure/test is related to some recognized standard or criterion • Increase test validity • Create an intelligence test and then compare subjects scores on our test with their scores on the IQ test • Use 2 motivation instruments • Giving subjects our intelligence test and the IQ test at the same time • Use abbreviated Myers Briggs – 126 vs. 72 items at the same time

24. Content Validity • Also called face validity • Degree to which a test adequately samples what is covered in a course • Usually used in education • Does a measurement appear to measure what it purports to measure? • No statistical measure/systematic procedure to test this

25. Content Validity • Often, experts (panel) are used to verify the content validity of measurements in research studies • Content validity is useful, but not the strongest/most credible way of evaluating a measurement • Examples • Rewards listing • Competency categories

26. Construct Validity • Degree to which a test/ measurement measures a hypothetical construct • Overall quality of measurement • Construct • Variables… recruitment, motivation, mental preparation • Examples • Do the selected variables completely measure recruitment? • How well does the instrument measure mental preparation? • Do the questions adequately test motivation?

27. Construct Validity • Threats to construct validity • Using one method to measure the construct • Inadequate explanation of a construct • Ex. Depression = lethargy, loss of appetite, difficulty in concentration, etc… • Measuring just one construct & making inferences • Using 1 item to measure personality • Ex. Myers Briggs = 4 dichotomies

28. Validity Overview • Content • Test • Construct • Test content • Standards • How well constructs describe relationship

29. Reliability • Degree to which a test/measurement yields consistent and repeatable results • Often reported as a correlation coefficient… Cronbach Alpha Cronbach's alpha Internal consistency α ≥ 0.9 Excellent 0.8 ≤ α < 0.9 Good 0.7 ≤ α < 0.8 Acceptable 0.6 ≤ α < 0.7 Questionable 0.5 ≤ α < 0.6 Poor α < 0.5 Unacceptable Look at articles.

30. 4 Sources of Measurement Error • 1. The participants • Health • Motivation • Mood • Fatigue • Anxiety

31. 4 Sources of Measurement Error • 2. The testing • Changes in time limits • Changes in directions • How rigidly the instructions were followed • Atmosphere of the test/conditions

32. Sources of Measurement Error • 3. The instrumentation • Sampling of items • Calibration of (mechanical) instruments • Poor questions • 4. The scoring • Different scoring procedures • Competence, experience, dedication of scorers • GRE…

33. Methods to Establish Reliability • Test-Retest Reliability (stability) • Alternate forms • Internal consistency • Agreement/ Inter-rater Reliability

34. Test-Retest Reliability (1) • Repeat test on same subjects at a later time • Usually retest on a different day • Use correlation coefficients between subjects’ two scores • Used extensively in fitness & motor skills tests • Used less for pencil and paper tests

35. Alternate Forms Reliability (2) • Alternate forms reliability • Construct 2 tests that measure the same thing • Give the 2 tests to the same individuals at about the same time. • Highly used on standardized tests • CPRP/CPRE exam (125 questions) • Rarely used on physical tests because of the difficulty to develop 2 equivalent tests

36. Internal Consistency Reliability (3) • Split half reliability • Similar to alternate form except 1 form is used • Divide form into comparable halves (even ?’s/odd ?’s) • Do not use first half vs. second half because of testing fatigue • Correlate # of odds & evens correct

37. Internal Consistency Reliability (3) • Average inter-item correlation • Identify question numbers that measure a construct. • Correlate the responses to these questions • Psychological tests

38. Inter-Rater Reliability (4) • Two or more persons rate or observe • Common in observational research & performance based assessments involving judgments • GRE writing exam scoring • Will be expressed as a correlation coefficient or a percentage of agreement • Does not indicate anything about consistency of performances

39. In Summary • Pick variables that have a chance of varying (systematic variability) • Pick a reliable instrument (error variability, statistical regression, reliability) • Use random sampling whenever possible (extraneous variability, internal validity) • Control external validity thru sampling process at multiple sites (population external validity)

40. In Summary • Control external validity thru similar environmental processes (ecological external validity) • Make sure survey measures what it is supposed to (content & construct validity) • Fully plan data collection process (reliability)