Educational Research: Getting Started

1. Educational Research: Getting Started Steven B. Goldin Chief of Hepatobiliary and Pancreatic SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education Chief of Hepatobiliary and Pancreatic Surgery Making Life Better through Research, Education & Healthcare TM

2. NO DISCLOSURES EXCEPT: Yes, they really are my photos

4. Outline 1. Research: Why do it? 2. What is educational research 3. Research: How to get started 4. Features of rigorous studies 5. Pitfalls – validity and reliability • Surveys • Ethics, funding, resources

5. Research – why do it? ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)

6. Research- why do it? Faculty advancement Teaching and research separate academic surgeons from those in community practice Develop a national prominence/reputation May be a funding source

7. Introduction Importance of clinical productivity – how do you find the time to do a research project? Do research with your daily activities Data collection is common for all courses and rotations. medical students residents faculty Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?

8. Introduction Innovation and evaluation form the backbone of research ideas and projects Many research projects are done with students, residents, and faculty Steady supply of potential participants

9. Introduction Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research” Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method) Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor

10. Introduction Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question

11. Evidence Based Medicine Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)

12. Evidence Based Medicine Many EBM concepts are applicable to educational research and should be familiar to all educational researchers Medical educators, likewise, should teach based on evidence

13. Introduction Educational research is driven by the process of documenting the steps of a project This process is the same for all studies including bench research, studying teaching methods, or student attitudes Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice

14. Making it Scholarly • All research involves – asking a question and obtaining an answer in a systematic, thoughtful way • Scholarly teaching involves routinely asking questions about daily educational activities • Research is answering these questions and scholarship is the dissemination of the research results

16. Types of Studies Descriptive: study describes what is going on or what exists Relational: study looks at the relationships between two or more variables Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable

17. Research Steps • Ask the question – state clear goals (hypothesis and alternate hypothesis) • Prepare (literature review, discussions, conferences) • Specify the techniques or interventions • Test methods and ways of measuring impact • Pilot test the process and refine process • Select the best method to answer the question

18. Research Steps • Do the study • Capture and analyze data • Interpret findings and determine significance of the outcome • Negative results can be meaningful depending on the question • Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps) • Disseminate the findings (article, abstract, presentation, workshop)

19. Features of Rigorous Studies Large numbers of student participants Multi-institutional enrollment of participants Having a control or appropriate comparison group Measuring an objective outcome Measuring some outcome at least 1 month after the intervention Conduct the intervention more than once Estimate statistical power

20. Features of Rigorous Studies Measure objective outcomes including any evaluation other than self-assessment by students including: Tests of knowledge Course grades Objective structured clinical examinations or observations of standardized patient Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback Objective clinical outcomes such as blood pressure control Performance with mannequins, computer simulations, or laboratory animals Psychological inventories

21. Foundation Language Types of questions (descriptive, relational, causal) Time in research (cross-sectional vs. longitudinal with repeated measures or time series) Types of relationships (correlational vs. causal) Patterns of relationships (none, positive, negative) Variables and attributes (independent and dependent, exhaustive, mutually exclusive) Hypothesis and null hypothesis Types of data (quantitative vs. qualitative)

22. The Hypothesis Theoretically significant research questions are different from other types of questions. A research question is not a question: of fact. (How many residents passed the in-service or their qualifying examinations?) about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?) about things that can not be measured. (Do lucky students have more success than unlucky students?)

23. A good research question should: 1. seek new knowledge 2. use the scientific method 3. build upon existing scientific knowledge 4. pick up where existing theories and data end 5. lead to specific hypotheses which can be tested 6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings

24. Reliability and Validity Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement Reliability involves the quality of measurement and the consistency or repeatability of the measures Theories of validity involve each step which are interdependent

25. Reliability and Validity Studies may be reliable but not valid valid but not reliable neither reliable nor valid both reliable and valid Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity

26. General Surgeon Shortage

27. Surveys - References Sage Publications, Inc; 2nd edition (November 12, 2002), 10 volumes, 1434 pages Gall, Borg, and Gall. Educational Research: An Introduction (8th edition). Allyn and Bacon. 2006

28. Surveys One of the most important areas of measurement in applied social research. Two types Questionnaires Interviews Construction of the Survey Types of questions Decisions about question content Decisions about question wording Decisions about response format Question placement and sequence

29. Surveys Often done because users think they are simple and quick ways to publications Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been: poorly designed poorly developed poorly administered poorly analyzed Reinforces the reconceived notion that surveys are not worth doing Like experimental research, good survey research is very systematic and rigorous

30. Surveys The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey What information do you want from respondents? attitudes/beliefs? knowledge? behavior? attributes (demographic characteristics)?

31. Surveys Structure the questions carefully Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher

32. Surveys A questionnaire should: 1. look attractive/appealing to complete 2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible 3. be organized with items appearing in some logical sequence 4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening 5. place important items that are key to the study near the beginning Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate. Pilot the instrument

34. Ethics Principle of voluntary participation Informed consent Risk of harm – physical or psychological Confidentiality Principle of anonymity Person’s right to service (no treatment control group) Institutional Review Board (IRB) Students and residents are considered a vulnerable population Social and Behavioral Institutional IRB vs. Biomedical IRB

35. Funding Most surgical education research is conducted without the benefit of research grants ASE Foundation – CESERT program (www.surgicaleducation.com) National Board of Medical Examiners Emergency Medicine Foundation Agency for Health Care Research and Quality Sages American College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research

36. Resources ASE Educational Research Committee Institutional Department of Medical Education Departments of psychology, education, sociology, and public health Biostatistician ASE – Surgical Education Research Fellowship Program (SERF)

37. Conclusions Clinical demands are taking on larger levels of importance Protected time is a luxury Research is required to advance academically Educational research is a viable option even when faced with high clinical demands

38. Conclusions Many educational research projects have significant limitations Insufficient sample sizes Lack of generalizability when interventions are assessed at only 1 institution or are conducted only once Lack of appropriate control groups Use of subjective and non-validated instruments to assess outcomes Assessing only short-term outcomes

39. Conclusions Educational research challenges often relate to measurement issues, bias and individual differences An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key! Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting

40. References Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do

42. Educational Research: Getting Started Steven B. Goldin Chief of Hepatobiliary and Pancreatic SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education Chief of Hepatobiliary and Pancreatic Surgery Making Life Better through Research, Education & Healthcare TM