An 8-Week Clinical Research Education Course for Neurology Residents

1. Educating Future Clinical Research Investigators: An 8-Week Clinical Research Education Course for Neurology Residents Jeanie McGee, MSHS, CCRC

2. Acknowledgements • Dissertation Chair • Warren McDonald, PhD • Dissertation Committee • Alireza Minagar, MD, FAAN • Rebecca Nolan, PhD, MP • Special Thanks • Robert Schwendimann, MD • Debra E. Davis, MD

3. Background • The physician-investigator has been labeled as an endangered species (Armstrong, DeCherney, Leppert, Rebar, & Maddox, 2009; Goldhammer et al., 2009; Guelich, Singer, Castro, & Rosenberg, 2002; Johnson, Subak, Brown, Lee, & Feldman, 2010; Lowe et al., 2007; Raghavan & Sandham, 2007; Siemens, Punnen, Wong, & Kanji, 2010; Teo, 2009; Wyngaarden,1979) • Lack of research funding • Lack of time and interest • Regulatory requirements • Lack of adequate exposure, education, and training in research during residency (Armstrong et al., 2009; Clancy, 2008; Krousel-Wood, 2006; Teo, 2009)

4. Background (cont.) • Consequences: • Delays in translational research • Industry hesitance to invest time and money • Decrease in competent and effective mentors • Devastating effects on patient outcomes

5. Review of the Literature • Medline (EBSCOHost) (Exploded with MeSH options) • Pubmed • Google Scholar • 2002 to 2012 (some historical literature) • Terms: • research education, residency research education, clinical research education, self-perceived competency, residency education, and neurology residency education • Research during residency is a critical topic. • Limited evaluation of research education during residency, especially neurology

6. Review of the Literature (cont.) • Institutional neurology residency program accredited in 2005 • 3 residents have participated in research elective (out of 27 graduated) • Mostly used as study time, so revised to require research plan with specific topic and protocol – 0 residents participated since (Dr. Davis, personal communication, August 28, 2012)

7. Problem Statement This researcher will develop, implement, and evaluate an 8-week clinical research educational course for residents in a neurology department at a Louisiana university medical center to prepare neurology residents for participation as clinical research investigators by increasing knowledge of fundamental clinical research methodology, design, and analysis, as well as by increasing the residents' self-perceived competency in these areas.

8. First Major Aspect of Problem Statement • Improving the quality of patient care by filling the research education gap • Physicians with research interests and skills needed for successful translation of new treatments and therapies Solution: Require research education during residency training

9. Second Major Aspect of Problem Statement • National recognition of the problem • National Institutes of Health (NIH) • Clinical Research Training Program – 1997 • Medical Research Scholars Program – 2012 • Resident Research Career Day - 2012 • Office of Clinical Research Training and Medical Education (OCRTME) • Summer internships, research electives for medical students, residents, fellows, and high school students • United States Food & Drug Administration (FDA) • Critical Path Initiative - 2009 • Clinical Investigator Training Course – 2009 • Accreditation Council for Graduate Medical Education (ACGME) • Require curriculum that advances “"residents' knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care" (ACGME, 2010, p.14). Solution: Incorporate research education at the local level

10. Third Major Aspect of Problem Statement • Research education in neurology residency • Only 10% of ACGME- accredited neurology residency programs have a research requirement • Over 50% of neurology residency programs require research, but offer no specific rotation or educational curriculum, nor protected research time • If available, research rotation has shortest average rotation time (mean = 0.3; range = 0 – 2 months) (Schuh et al., 2009) Solution: Implement a mandatory, structured clinical research curriculum for neurology residents

11. Fourth Major Aspect of Problem Statement • Extinction of the clinical research investigator • Downward trend since late 1970s (Wyngaarden, 1979) • Currently, only 4% of physicians engage in research activities (CISCRP, 2012) • Need to preserve the integrity of clinical research • CNS therapeutics are the most prevalent area addressed by pharmaceutical research (second only to oncology and immunomodulators, which are also common neurological therapies) (CISCRP, 2012) Solution: Promote interest and encourage clinical research activity involvement during neurology residency

12. Fifth Major Aspect of Problem Statement • Effects of research education for residents in order to build research competency • 87% of physicians believe statistics knowledge is important in patient care delivery, but only 17% report their statistics knowledge to be sufficient (West & Ficalora, 2007) • 88% report biostatistics and a strong research design foundation is necessary to effectively and accurately appraise published literature (West & Ficalora, 2007) • 75% of medicine residents reported they understand all statistics reported in the literature, while their mean score was 41.4% correct responses when tested on statistical concepts found in the literature (Windish, Huot, & Green, 2007) Solution: Implement curriculum that includes evaluation that effectively assessing discrepancies in self-assessed competency and actual level of knowledge

13. Sub-ProblemsSub-Problem 1. Do neurology residents who have taken a prior elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education?

14. Sub-ProblemsSub-Problem 2. Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

15. Sub-ProblemsSub-Problem 3. Does the resident’s age influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

16. Sub-ProblemsSub-Problem 4. Does the resident’s gender influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

17. Sub-ProblemsSub-Problem 5. Does the resident’s post-graduate year level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas?

18. Purpose of the Study The purpose of this dissertation research was to evaluate the effectiveness of an 8-week clinical research education course designed for neurology residents. The research curriculum was designed to include topics significant to research methodology, design, and analysis.

19. Study Goals • to provide neurology residents in a single department at a university medical center in Louisiana information to increase the knowledge level of residents in the topic areas of research methodology, design, and analysis

20. Study Objectives • (1) measure baseline levels of research knowledge and residents' self-perceived level of competency in research, • (2) measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention, • (3) evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research, and • (4) determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

21. Theoretical Framework Cognitivist Approach and Competency-Based Model • Jerome Bruner’s cognitivist approach • Self-discovery • Leveled learning • Prior knowledge • Information acquisition (derived from existing knowledge) • Information transformation (processing and analyzing new information) • Information evaluation (self-assessment of information accuracy)

22. Theoretical Framework Cognitivist Approach and Competency-Based Model • G. E. Miller’s framework for clinical assessment

23. Research Design • Quantitative one-group pre-test post-test • Dependent variables: • Resident level of knowledge in clinical research methods, design, and analysis • Resident self-perceived competency in clinical research methods, design, and analysis • Sample: • Neurology residents in PGY II, III, and IV training

24. Curriculum / Intervention • Weekly sessions for 8 consecutive weeks • 1 hour • Session topics: • Introduction to Research: Basic Research Concepts and Terminology • The Research Question and Hypothesis • Research Study Design and Sampling • Introducing Variables and Scales of Measurement • Introduction to Biostatistics • Research Ethics, Privacy, and Responsible Study Conduct • Basic Regulatory Requirements for Clinical Research • Review of Research Methodology, Design, and Analysis (Overview)

25. Methodology • Measure baseline levels of research knowledge and residents' self-perceived level of competency in research • Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention • Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research • Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

26. Methodology (cont.) • Measure baseline levels of research knowledge and residents' self-perceived level of competency in research • Measure levels of research knowledge and residents' self-perceived level of competency in research following the educational intervention • Evaluate potential changes from baseline to post-intervention in residents' research knowledge, as well as their self-perceived competency in research • Determine if demographic or descriptive variables, including resident gender, age, postgraduate (PGY) year level, future career plans, and history of prior research education, had an impact on either measure

27. Instruments • Developed by the researcher: • Evaluation Tool to Capture Neurology Resident Descriptive Data demographic assessment tool • Self-Assessed Level of Knowledge and Competency in Clinical Research (pre-assessment) (38 items – score 1 to 4 from least competent to most competent) • Level of Knowledge in Clinical Research Methodology, Design, and Analysis (pre-assessment) (50 multiple choice items with a possible score = 100%) • Self-Assessed Level of Knowledge and Competency in Clinical Research (post-assessment) (38 items – score 1 to 4 from least competent to most competent) • Level of Knowledge in Clinical Research Methodology, Design, and Analysis (post-assessment) (50 multiple choice items with a possible score = 100%)

28. Instruments • Considerations in the design of instruments: • American Academy of Neurology (AAN) evidence-based medicine • Association of Clinical Research Professionals (ACRP) Certified Clinical Research Investigator mock examination • Miller’s pyramid of competence

29. Data Collection • Collect demographic data • Collect baseline self-competency data • Collect baseline level of knowledge data • Implement curriculum intervention • Collect post self-competency data • Collect post level of knowledge data

30. Data Analysis • Data coded and entered into Microsoft Excel • Cleaned, verified • Data moved to SPSS v. 21 for statistical analysis

31. Predictions • Residents with prior research education will have higher mean scores on both knowledge and self-perceived competency at baseline • Residents with future career plans involving research and/or academia will have higher mean scores on both knowledge and self-perceived competency at baseline • Higher PGY levels will have higher mean scores on both knowledge and self-perceived competency at baseline • Male residents will have higher mean scores on both knowledge and self-perceived competency at baseline • Older will have higher mean scores on both knowledge and self-perceived competency at baseline

32. Data Analysis • Descriptive statistics • Average age = 35.5 years • PGY Levels: • PGY2: 30% (n=3) • PGY3: 40% (n=4) • PGY4: 30% (n=3) • 70% reported prior research education (only 1 during medical school or residency) • Length of time of prior education: range = less than 1 week (n=3) to three years or more (n=1) (others were less than one month (n=1), 3m to 6m (n=1), 6m to 1y (n=1) • Most common topics: • Developing and writing research question and hypothesis (n=5) • Research study design and sampling (n=5)

33. Data Analysis (cont.) • Descriptive statistics • 20% (n=2) reported career plans involving clinical research • 20% (n=2) reported career plans involving basic science research • 20% (n=2) reported career plans involving writing and publication • 30% (n=3) reported career plans involving work at an academic medical center • 20% (n=2) reported career plans NOT involving any type of research • 30% (n=3) have not yet decided whether or not they will engage in research after residency

34. Data Analysis (cont.) • Self-assessed Competency at Baseline • Mode = 1 “knows about the skill or concept or has heard about it, but is unable to do it” • Mode range 1 -2: • “knows about the skill or concept or has heard about it, but is unable to do it” • "knows how to do a skill/concept, but has never done it and is unable to show or explain how to do it“ • No item had a mode of 3 or 4 at baseline • 3: “knows how, can show or explain how to do it, but has never done it” • 4: “knows how, has done, and can do at any time”

35. Data Analysis (cont.) • Self-assessed Competency at Baseline • Possible total score = 152 (all 4s, representing highest level of competency) • Percentage scores calculated, mean = 63% at baseline (range = 42 – 125)

36. Data Analysis (cont.) • Level of Knowledge at Baseline • Mean score = 53 (range = 36 – 70)

37. Sub-Problem Results Sub-Problem 1. Do neurology residents who have taken a prior elective or required clinical research course(s) have a higher level of self-perceived competency in fundamental clinical research methodology, design, and analysis than neurology residents who have not taken a prior elective or required clinical research course(s) as determined by correlating results of a pre-test measure of self-perceived competency in fundamental clinical research with resident prior exposure to research education? Had an effect, but did not reach statistical significance.

38. Sub-Problem Results Sub-Problem 2. Does the neurology resident’s future career plans have an impact on their level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas? • Self-perceived competency: • Career interest involving clinical research (F=9.142; p=.016)* • Career interest involving basic science research (F=8.652; p=.019)* • Career interest involving writing and publishing scientific manuscripts (F = 8.652; p = .019)* • Level of knowledge: • No statistically significant effect

39. Sub-Problem Results Sub-Problem 3. Does the resident’s age influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas? • Self-perceived competency: • No statistically significant effect • Level of knowledge: • No statistically significant effect

40. Sub-Problem Results Sub-Problem 4. Does the resident’s gender influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas? • Self-perceived competency: • No statistically significant effect, but males had a higher mean (67, compared to 63 for females) • Level of knowledge: • No statistically significant effect, but males had a higher mean (54.57, compared to 49.33 for females)

41. Sub-Problem Results Sub-Problem 5. Does the resident’s post-graduate year (PGY) level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas? • Self-perceived competency: • No statistically significant effect • Level of knowledge: • No statistically significant effect on overall score, but statistics domain differences statistically significant (F=5.067; p=.044)*

42. Sub-Problem Results Sub-Problem 5. Does the resident’s post-graduate year (PGY) level influence level of knowledge in fundamental clinical research methodology, design, and analysis, and self-perceived competency in these areas? • Level of knowledge: • No statistically significant effect on overall score

43. Final Results • Self-perceived competency baseline to follow-up • (t = -2.558; p =.034)* • Baseline percentage mean = 48.4 • Follow-up percentage mean = 68.4 • Baseline modes = 1 – 2 • Follow-up modes = 3 – 4 • Four items: mode increased from 1: Does Not Know How to 4 Knows How, Has Done, and Can Do at Any Time: • Item 11 “Identify the difference between descriptive and analytic research study designs” • Item 21 “Identify the measures of central tendency” • Item 27 “Identify what information a confidence interval (CI) says about a statistical result” • Item 37 “Determine when a research study should be closed through the local Institutional Review Board (IRB)”.

44. Final Results • Self-perceived competency baseline to follow-up

45. Final Results • Level of knowledge baseline to follow-up • (t = -3.210; p=.012)* • Baseline mean = 52.67 • Follow-up mean = 65.78 • Knowledge level scores increased from 12.5% to 94.44% per resident

46. Final Results Level of knowledge baseline to follow-up by domains

47. 3.88% Difference 8.37% Difference Final Results Self-perceived competency and level of knowledge

48. Final Results Sub-problem 1. Differences in change in level of knowledge based on prior research education (F = 13.44; p = .008)* No prior education: 56.71% improvement Prior education: 12.55% improvement

49. Final Results Sub-problem 4. Differences in change in level of knowledge based on gender (F=15.028; p=.006)* Male: 24.08% improvement Female: 19.60% improvement (no statistical difference)

50. Attendance • Per resident • Range = 0 sessions attended to 8 attended (100%) • Mean total attendance = 66.7% (x2(df = 2) = 10.33; p = .005)*