Research Design in Clinical Psychology

1. Research Design in Clinical Psychology Lecture 3 Assessment (Chapters 13 & 14 in Kazdin)

2. Key Considerations in Selecting Measures Construct validity Psychometric qualities Sensitivity of the measure

3. Construct Validity • How well does the measure selected reflect the construct of interest • Extent the measure assesses the domain, traits, and characteristics of interest

4. Psychometric Characteristics • Reliability • The consistency of the measure • Within the measure (relationship among items) • Consistency across different parts or alt forms • Over time (test-retest) • Validity • Related to content • Does measure assess construct of interest

5. Reliability • Test-retest • Stability of scores over time • Alternate form • Correlation of different forms/parts of same measure • Internal Consistency • Consistency of items in a scale • Split ½, KR 20, and alpha • Inter-rater reliability • Agreement across observers on scores on a measure • % agreement, correlation, Kappa

6. Validity • Construct1 and Content2 • 1Extent overall measure reflects latent construct • 2Similar to construct, but focused on items • Concurrent1 & Predictive2 • 1Correlation between measures at same time points • 2Correlation b/n measure1 now & measure2 in future • Together criterion, but also used to reflect relation b/n measure & some criterion such as disorder status • Face • Convergent1 and discriminant2 • 1Correlation between measures at same time points • Form of concurrent validity • 2Correlation b/n measure1 now & measure2 in future

7. Sensitivity • Measurement sensitivity • Reflection of systematic variation • Affected by variable itself and way its manipulated • Critical features • Broad range of scores • Change in scores not constrained by pretest score

8. Actually Selecting a Measure Use of existing measures Vs Development of new measures

9. Picking a new measure • All things considered, its better to use existing measure because of standardization and available information about norms, etc…but • Can vary the measure if for some reason (time, sample characteristics) its not appropriate • New psychometrics MUST be reported • Can develop new measure

10. Modality of Assessment • Global vs. Specific • How are you doing in general vs. specific domain • GAF vs anxiety sensitivity • Publicly observable vs. private • Stable vs. Transient • Direct vs. Indirect • Is the purpose obvious to subject • Breadth of domains • Are there subscales & how related are subscales • Format

11. Three Modes of Assessment • Self-report • Easy to administer • Can assess diverse aspects of a construct • Wording, item order can have big effect • Problems of bias and other demand characteristics • Behavioral / Direct observation • Time and resource costs • Base rates • Bias and often in contrived situations • Physiological • Time and resource costs • Artifacts • Interpretation of results can be difficult

12. Obtrusive and non-obtrusive measurement

13. Limitations of Obtrusive Measures • May inflate correlation across measures • Limit generalizability • How much is situation like reality • Demand characteristics • Including influence of experimenter characteristics

14. Solutions for Obtrusive Measures • Instructions • Filler / buffer items • Deception • Use of computers or other identity obscuring strategies

15. Unobtrusive measures • Simple • Contrived situation in naturalistic setting • Archival records • Physical traces

16. Use of multiple measures • Allows for assessment of several distinct, yet related aspects of behavior • Allows for assessment of situation specificity • Trait and method variance • Use of a multi-trait, multi-method matrix • What if low relationships exist among measures?

17. Lack of Inter-relation among measures • Method variance • Multi-faceted nature of personality and behavior – example of impulsivity • Magnitude of characteristic • Course of behavior change • Synchrony vs desynchrony • E.g., do cognitive indices show change at the same rate as behavioral indices?

18. Assessment and Evaluation of Interventions

19. Clinical Significance • Practical value or importance of the effect • Did the intervention make a difference in the clients life functioning • Methods for establishing clinical significance • Comparison method • Subjective evaluation • Social impact

20. Comparison Method • Subjects behavior at end of tx can be compared to others • Clinical samples • No longer meets diagnostic criteria or performs considerably better than other patients w/o tx • Normals • Looks similar to individuals in community

21. Subjective Evaluation • Patient rates how they are functioning • Although necessary to get patient’s opinion on how they are doing, several sources of bias can occur

22. Social Impact • Change in pre-existing measure of everyday life functioning • Not developed specifically for testing • High external validity, but all the problems of unstandardized measures and limited by lack of precision in the real world

23. Treatment Characteristics

24. Treatment Characteristics • Disseminability • Complexity of procedures for patients and staff • Necessary training for staff • Likelihood of patient and staff adherence • Flexibility • Cost • Acceptability of Treatment • Greatly affects disseminability

25. Timing of Assessment • Pre-post assessment, typical but… • Often no clear endpoint • Provides little information of effects during tx • On-going, more work but… • Prevents complete loss of data for drop-outs • Information can be used to modify tx as needed

26. Follow-up Assessment • What measures will be used • Where will assessments be conducted • Who will provide data • Will it always be the subject? • How can we increase likelihood of all subjects completing follow-up • Trade-offs must be considered • All issues should be considered with attrition and generalizability in mind