Reliability: Introduction

Reliability: Introduction

Reliability Session • Definitions & Basic Concepts of Reliability • Theoretical Approaches • Empirical Assessments of Reliability • Interpreting Coefficients • Test Refinement and Reliability

Conceptions of Reliability “My car won’t start!” “He’s often late!”   S.E.M.

Components of Measurement Measured Value = True Value + Systematic Error (Bias) + Random Error The usefulness of a measure depends on the ratio of the true value to any error variance that it produces

Classical Test Theory • Random error vs. systematic, or bias • Classical theory assumptions: • Error independent of score • Mean of errors = 0 • Observed score = true score + error • Random errors tend to cancel out if sufficient observations made

Internal Consistency • Logic: problem of change over time • Alternate forms • Split-half • Kuder-Richardson & alpha • Internal consistency • Item-total correlations • Number of items & reduction • in error term • Spearman-Brown formula reliability # items

Sources of Variance:Which to Include in Estimating Reliability? • Error • Patients • Observers • Time • Measurement instrument

Reliability Subject Variability Subject variability + Measurement Error Reliability = or, Subject Variability Subject Var. + Observer Variability + Meas’t Error

Generalizability Theory • Separates sources of variability: • Observer inconsistency • Between observers • Subject change over time • Quantifies these • Helps to show how to optimize design (and administration) of test given these performance characteristics.

Reliability versus Sensitivity of a Measurement Metaphor of the combs

Statistics to use: ICC vs. Pearson r ICC = 1.0; r = 1.0 r = 1.0; ICC < 1.0

What is the Reliability when: • Every student is rated “above average” • Physician A rates every BP as 5 mm Hg higher than physician B • The measure is applied to a different population • The observers change • The patients do, in reality, improve over time

Reliability: Introduction