Educational Research: Study Methodology

1. Educational Research: Study Methodology Pamela M. Williams MD Dept of Family Medicine Uniformed Services University

2. Educational Research Broadly Defined Any investigation related to the education of medical professionals Undergraduate (medical school) Graduate (residency) Continuing medical education

3. Basic (Educational)Research Steps Identifying a problem Examining relevant variables through a literature review Constructing a hypothesis Creating a research design to investigate a problem Collecting and analyzing data Drawing conclusions Writing and publishing the results

4. Session Topics Basics of research paradigms & design Measurement & data collection Sampling Threats to validity IRB approval�is it needed?

5. Session Ground Rules Interactive discussion Minimal lecture Small group activities Please Ask questions Share knowledge and experiences Divide into small groups here.Divide into small groups here.

6. Quality research starts by asking good questionsQuality research starts by asking good questions

7. What is your research question? Break into your small group Consider the problem you identified last evening Take 2-3 min to write a research question Articulate your question to peers & refine as needed To review: Optimally, your research question is: Based on literature/theory Includes sample description (e.g. 3rd year medical students) Includes study design (e.g. relationship, difference between groups, etc) Includes the independent & dependent variables Measurable Stated as a question or hypothesis Or FINER Feasible Interesing Novel Ethical ReleventTo review: Optimally, your research question is: Based on literature/theory Includes sample description (e.g. 3rd year medical students) Includes study design (e.g. relationship, difference between groups, etc) Includes the independent & dependent variables Measurable Stated as a question or hypothesis Or FINER Feasible Interesing Novel Ethical Relevent

8. Optimally, your question is� Based on literature/theory Includes sample description (e.g. 3rd year medical students) Includes study design (e.g. relationship, difference between groups, etc) Includes the independent & dependent variables Measurable Stated as a question or hypothesis

9. Selecting a Research Design

10. Research Paradigms Inductive: Investigator begins with observations and attempts to explain what has been observed by generalizing. Deductive: Investigator begins with a theory and collects data to test it. Rather than thinking in terms of qualitative vs. quantitative vs mixed methods (one way to consider research paradigms), we are going consider research paradigms today in terms of deductive vs. inductive (and largely focus on the former).Rather than thinking in terms of qualitative vs. quantitative vs mixed methods (one way to consider research paradigms), we are going consider research paradigms today in terms of deductive vs. inductive (and largely focus on the former).

11. Research Design Decision Tree Exploratory: Categorizing or mapping new conceptual areas; looking at new relationships Descriptive/Correlational: Describing something as a whole when it is new or needs to be described. Explanatory /confirmatory: Existing research seeking to explain or confirm finding to your particular situationExploratory: Categorizing or mapping new conceptual areas; looking at new relationships Descriptive/Correlational: Describing something as a whole when it is new or needs to be described. Explanatory /confirmatory: Existing research seeking to explain or confirm finding to your particular situation

13. Small Group Exercise Review your set of three papers As a group, answer the following for each study: What is the research question? Is it a deductive or inductive research process? What is the study design? Be prepared to Briefly tell us about the study Describe characteristics of the methodology used

14. Inductive Research What is the phenomenon? Methods Interview Observation Think aloud, stimulated recall Chart review Surveys

15. Uses in Medical Education Needs assessment Program development Curriculum evaluation Performance evaluation And more!

16. Challenges of Qualitative Research Methods Data overload Time demands of processing and coding Adequacy of sampling Generalizability of findings Researcher bias Credibility and quality of conclusions

17. Descriptive/Correlational Research How are variables associated? How do we begin to make sense of what we observe? Methods Surveys Chart review Archived data

18. Experimental & Quasi-experimental Does the predictor cause the relationship? Methods Control over treatment & measure Randomization Control groups

19. Non-experimental Designs Case control Cross-sectional Time series (serial surveys) Cohort (Panel) Prospective Retrospective

20. Research Design Decision Tree Exploratory: Categorizing or mapping new conceptual areas; looking at new relationships Descriptive: Describing something as a whole when it is new or needs to be described. Explanatory /confirmatory: Existing research seeking to explain or confirm finding to your particular situationExploratory: Categorizing or mapping new conceptual areas; looking at new relationships Descriptive: Describing something as a whole when it is new or needs to be described. Explanatory /confirmatory: Existing research seeking to explain or confirm finding to your particular situation

22. Small Group Work As a group identify 2 possible study designs for your project

24. Educational Research:Measures & Outcomes Brian Smoley, MD, MPH Sandy Kimmer, MD, MPH

25. Review: Variables Independent (IV): Variable that is manipulated, measured, or selected to observe the relationship to some other observed variable (i.e. it is expected to influence some other variable) Dependent (DV): Variable that is observed and measured in response to an independent variable (i.e. it is expected to increase, decrease, or vary systematically as the IV changes Control: Anything held constant

26. An Example Do first-year interns who complete a resident run in-service review course score higher on the in-service exam than those who do not complete the course? What is the independent variable? What is the dependent variable? What are possible control variables?

28. Small Group Work Take 10 minutes to consider the variables for your various projects.

29. What potential threats to validity for your projects?

30. Threats to Validity Internal History Maturation Repeated measurement Statistical regression Selection Loss of subjects Investigator bias External Is the sample representative of the population? Can the study be generalized? Are the conditions the same as other environments? Hawthorne Effect Hawthorne Effect: Did the subjects act differently because they were subjects in the study?Hawthorne Effect: Did the subjects act differently because they were subjects in the study?

31. Sampling Who are your �people�? To whom would you like the results to be generalized? How do you select your sample in away that enables you to generalize the results of this sample? What do you wish to generalize to this sample? What are some ways of selecting an appropriate population?

32. How and why are you sampling? Random sampling Stratified sampling Cluster sampling Systematic sampling Challenges to sampling Sampling bias Size Population Self-selection Snowballing Available group use

33. Do I need IRB Approval? Research is a systematic investigation including research, development, testing, and evaluation to develop or contribute to �generalizable knowledge.� -Belmont Report: �if there is any element of research in an activity, that activity should undergo review of the protection of human subjects�. Current federal regulations state the same. -Difference between research vs. evaluation studies. -If you plan on publishing your results, the study is �research� and requires IRB approval. -Study subjects cannot be required to be subjects or to complete an research instruments: RESEARCH is voluntary. -This contrasts with regular evaluation done for courses or programs in which you may indeed require a student to complete an exam or evaluation as part of their student assessment or continuous quality improvement. -May be able to do research on pre-existing data sets, but 1. it shouldn�t be your �plan� and 2. talk with IRB before proceeding *Much educational research is exempt.-Belmont Report: �if there is any element of research in an activity, that activity should undergo review of the protection of human subjects�. Current federal regulations state the same. -Difference between research vs. evaluation studies. -If you plan on publishing your results, the study is �research� and requires IRB approval. -Study subjects cannot be required to be subjects or to complete an research instruments: RESEARCH is voluntary. -This contrasts with regular evaluation done for courses or programs in which you may indeed require a student to complete an exam or evaluation as part of their student assessment or continuous quality improvement. -May be able to do research on pre-existing data sets, but 1. it shouldn�t be your �plan� and 2. talk with IRB before proceeding *Much educational research is exempt.

34. Questions to Consider� Is the data normally collected in your course? Is the data being specifically collected for your study? Will the data be anonymous? Will you use any identifier? Is it linked? Do you want to publish or �make public� your results? Do you think you need informed consent?

35. Potential Categories Exempt Expedited Full review Once deemed research your IRB/human subjects committee will place your research in 1 of 3 categories. Most, but not all, will fall into one of the first two categories.Once deemed research your IRB/human subjects committee will place your research in 1 of 3 categories. Most, but not all, will fall into one of the first two categories.

36. Take home? Remember the Basic (Educational) Research Steps !!! Identify a problem Review the literature Construct a hypothesis Identify a research design to investigate the problem Collect and analyze your data Draw conclusions

38. Deductive Research Process -What situations or problems tend to puzzle, fascinate, challenge or interest you? Which are most interesting? Feasible? Fundable? Best overall?-What situations or problems tend to puzzle, fascinate, challenge or interest you? Which are most interesting? Feasible? Fundable? Best overall?