A Process Approach to Outcome Measurement

1. This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation • In Slide Show, click on the right mouse button • Select “Meeting Minder” • Select the “Action Items” tab • Type in action items as they come up • Click OK to dismiss this box • This will automatically create an Action Item slide at the end of your presentation with your points entered. A Process Approach to Outcome Measurement

2. A Process Approach to Outcome Measurement • Topics covered in this presentation • Design of the evaluation • Description of the participants • Knowledge findings • Participant learning styles • Instructor teaching styles • Intention to change practice findings • Motivation to change findings • Lessons learned OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

3. The Study • These data were developed at the 2006 Optimal Management of HIV Conference; • My collaborators, Dr. Harold Kessler and Michael Saag, and I are in the process of analyzing these data for publication; • The analyses I report here are preliminary are completed only for the purpose of these discussions; OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

4. Two: Elements : Participant Descriptors and Process Measurement OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

5. Data gathered • Learning style • Preference indicated by degree of endorsement of one or the other end of a dichotomy Example I prefer Hands-on learning experience ………………………………...Learning through thinking and reasoning • Learning through simulation …………………………………Learning through lectures OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

6. Data gathered (continued) • Demographics • Questions on age specialty and Board status delivered on the Learning Style questionnaire • Motivation to change • Motivation indicated by a modified standard scale of motivation to change Example It’s important to use the new approaches I’ve learned Strongly Agree - Strongly Disagree OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

7. Data gathered (continued) • Pre-Post Tests • Knowledge • Simple multiple choice knowledge based tests, one question per presenter per test, rotated to prevent order bias Example: Which of the following antiretrovirals is the least likely to cause DSPN • indinavir • zalcitabine • stavudine • didanosine • Laminvudine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

8. Data gathered (continued) • Pre-Post Tests (continued) • Intent to change • Procedural intent questions provided prior and post the session Example Pre When treating HIV patients with PCP I use Strongly Agree - Strongly Disagree TMP-SMX or TMP-dapsone Post When treating HIV patients with PCP I Strongly Agree-Strongly Disagree intend to use…. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

9. Data gathered (continued) • Teaching style • 3-5 observers rated each session using simple descriptors of the presentation using agreement or disagreement with the description Example The Presentation focused on clinical application Strongly Agree - Strongly Disagree The presenter focused on the underlying science of medicine Strongly Agree - Strongly Disagree OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

10. Description of the participants • Of the total participants in the program, analysis is focused on the 62 complete datasets, these participants provided data for all three days both in the morning and afternoon; • While it is possible that they differed in some systematic way from the rest of the participants, complete data will be necessary for the ultimate analysis OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

11. Results: Participants were generally middle aged • The participants had an average age of 49; • The youngest was 29 and the oldest 73; • The median age was 47 indicating a slight skew to the younger side of 49. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

12. The average participant graduated from medical school in 1984 • The youngest participant graduated in 2003; • The oldest participant graduated in 1958 OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

13. Most participants were in infectious disease or general internal medicine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

14. Most of the participants were ABMS Board Certificants • 62% of participants reported Certificant status for a Board; • 37% were not members of a Board or did not respond. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

15. The most frequent Boards of participants were Infectious Disease or Internal medicine OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

16. The participants improved their performance • Participants improved their performance from pre to post measures • The improvement was approximately 10%. • The improvement was significant p < .001. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

17. Participants also showed an expectation of changing practice patterns • Intention or expectation of changing practice patterns was measured by asking the likelihood of a particular practice being adopted; • The likelihood was measured prior to the session (Morning) and after the session (Afternoon). • The stated likelihood increased from 3.4 to 4 on a 5 point scale. • The effect was assessed and the increase in likelihood is significant p<.001. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

18. Learning style was assessed a modification of a scale developed by Kolb • The approach to learning style that was employed in this study was a modified version of Kolb’s learning style inventory. • We employed his core scale, adapting it from a dichotomy to a numeric format with a Kolb descriptor on each end of the scale. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

19. Teaching style was assessed by an observer assessment scale • The items were selected to reflect the teaching characteristics for the Kolb learning styles; • During each presentation, 3-5 observers assessed the style of each presenter. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

20. Motivation to change • Participants were asked to endorse 30 statements of change commitment; • The statements were modified from the original focus to reflect motivation and readiness to learn new information and make changes in clinical practice patterns. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

21. A Process Approach to Outcome Measurement • Outcome measurement • Direct assessment/measurement of actual practice change or improvement in patient health status is difficult; • Focus is on valid secondary measures such as: • Self-report; or, • Vignettes. • The core question for CME measurement professionals is the validity of such secondary measures. • Assessment validity is typically based on a combination of: • Face validity (is it reasonable); • Construct validity (does it render a measurement correlated with some other measure deemed valid); and, • Predictive validity. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

22. First we looked at the acquisition of knowledge • We used the teaching style judgment questions as indicators of an underlying set of predominate teaching styles • We then used the averaged teaching style scores for each instructor as predictors of knowledge acquisition of the participants. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

23. Teaching and Knowledge Acquisition • We looked at the relationship between Teaching Style and Knowledge: • We found a significant relationship between the two measures; • The unobserved variable Teaching Style explains approximately 25% of the variance in Knowledge. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

24. We looked at the formation of an intent to change • First, we used the learning style questions as indicators of an underlying set of learning preferences • We then used those preferences as predictors of intent to change practice patterns OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

25. Learning Style, Knowledge and Intent to Change Practice • We looked at the relationship between Learning Style, Knowledge and Intent to change practice patterns: • We found a significant relationship between the two causal measures and Intent; • The unobserved variable Learning Style explains approximately 13% of the variance in Intent; • The variable Knowledge explains approximately 55% of the variance in Intent. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

26. Self report of practice change • First we looked at the relationship between Motivation to change and reported practice change: • Motivation to change was assessed using a series of readiness for change items; • Change items were taken as indicators of an underlying state of change readiness. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

27. Motivation and Intent to Change Practice and Self-reported Practice • We looked at the relationship between Motivation to change, Intent to change practice patterns and self-report of practice patterns at a 12 month follow-up: • We found a significant relationship between the two causal measures and self-report practice change; • We also found a relationship between motivation and intent to change. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

28. A Path Model of Cognitive Elements of CME Efficacy • We looked at the relationships among a number of variables as they relate to practice change. We found: • Teaching Style has a path to Knowledge; • Knowledge and Learning Style have paths to Intent to Change; • Motivation to Change and Intent to Change have paths to Self-reported Practice Change. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME

29. Discussion • All of these variables are related in a causal manner with self-reported practice change; • Each variable has a theoretical reason to be thought of as causally related to practice change; • These data suggest that there are multiple variables that could provide a meaningful estimate of the degree of efficacy of the CME program. OPMAN 2006 Data: Interim Analysis for a Discussion of Outcome Assessment in CME