Rigorous Research

1. Rigorous Research Engineering Education Larry D. Gruppen, Ph.D. University of Michigan

2. Objectives • Describe scientific research standards • Compare and contrast engineering and educational research • Identify a few of the peculiar challenges of educational research

3. Agenda • Standards • Engineering Research (er) • Educational Research (er) • Educational Research in the Professions • Science and Application • Theoretical Frameworks • Research Areas • Comparing the er & er

4. Engineering Research • What are the guiding principles for rigorous research in your “regular” (STEM) discipline? • Take a few moments individually to list (in your journal) the qualities and characteristics of rigorous research in your discipline. • As a group, develop a list of research standards in STEM.

5. Guiding Principles forScientific Research in Education • Question: pose significant question that can be investigated empirically • Theory: link research to relevant theory • Methods: use methods that permit direct investigation of the question • Reasoning: provide coherent, explicit chain of reasoning • Replicate and generalize across studies • Disclose research to encourage professional scrutiny and critique National Research Council, 2002

6. 1. Significant questions that can be investigated empirically Curriculum DesignTeachingAssessment Outcomes Problem-based Clinical Competence Expertise learning teaching Knowledge Cognition Independent Educational Skills Competence learning technologies Attitudes Ethical reasoning Professional socialization Examples of research areas inprofessions education

7. 2. Link research to relevant theory • Learning theories • Psychometrics • Adult Development • Moral Development • Socialization • Cognition • Expertise • Social context of education Examples of conceptual frameworks in education

8. 3. Methods for Direct Investigation (examples) Quantitative methods • Tests • Surveys & questionnaires (defined response) • Faculty or peer ratings Qualitative methods • Focus groups • Interviews • Observations Gathering the empirical data

9. 4. Reasoning What makes a convincing argument • Theoretical foundation - results fit with expectations • Strength of observed relationships • Elegance - conciseness, simplicity, explanatory power • Elimination of alternative explanations • Study design • Confounding variables Interpreting the results

10. 4a. Study Design in Educational Research Common constraints and possible responses • Control group - logistical and ethical limitations • Delayed intervention cross-over design • Non-equivalent control group design • Randomization - ethical constraints • Self-selection • Historical group assignment • Sample size - often small, limited by class size • Multiple iterations of intervention • Collaborative studies - multiple institutions, collaborators

11. 4b. Confounding Variables • Time on task • Maturation • Prior experience • Motivation • Self-selection • Individual capabilities - dexterity, visualization, abstraction, communications, etc. • Affective - ‘math aversion,’ computer-phobia

12. 5. Replicate and Generalize Setting the results in a larger context • MUST know the literature • Strict replication is rare in educational research • Replication with extension - to new topic, setting, learners, etc. Going beyond your individual study

13. 6. Disclose • Scholarly journals • Conference presentations • Peer-review is the core issue • One of the few quality controls we have

14. Theory and Practice

15. Comparing er and er ? Compare er and er • As a group, list the similarities and differences between research in engineering and research in education. Use the following matrix to guide your discussion.

17. Journaling Questions How comfortable are you with the “messiness” of educational research? What is appealing about it to you? What do you find bothersome? Why?