Using Mixed Method Approaches in Institutional Research

1. Using Mixed Method Approaches in Institutional Research Rick Kroc University of Arizona Richard Howard University of Minnesota Arizona Association for Institutional Research Annual Meeting Tucson, Arizona March 2007

2. Purpose of Decision Support is to reduce the risk to the decision maker CERTAINTY

3. Structure and Subjectivity SUBJECTIVITY QUESTION CERTAINTY STRUCTURE

4. Research Process

5. Ways of Knowing Two Classic Paradigms Positivist Constructivist there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables there are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher)

6. Conceptual Model of the Constructivist Research Process After a literature review and pilot study, define the Study: Problem Purpose Question Data Collection & Management COMMUNICATE New Theory/Model Data Verification with Respondents: FACTUAL ERROR Data Restructuring/Coding & Analyzing/Read-Theme Analysis Verification with Reviewers: Interpretation Error TEST Theory/Model Iterative Process Delimited by PP&Q

7. Conceptual Model of the Positivist Research Process After a literature review and pilot study define the project Problem Purpose Question Operationalize theory or model constructs For internal decision support- test meaning of the findings Data Collection and Management Communication of new knowledge Data restructuring and analysis (Interpretation of Results) For external contexts- test/replicate for generalization

8. RESEARCH PARADIGMS APPROACHES Constructivist Positivist Quantitative Qualitative Researcher Generalizes Consumer Generalizes

9. Mixed Methods Combines the collection, analysis, and interpretation of qualitative and quantitative data. Often used in studies that require an understanding of not only “what” is observed but also “why” the observed behavior occurred. Such as in assessment, evaluation, action research, and other decision support studies.

10. Use of Mixed Method Approaches in Institutional Research Provides answers to two questions: What - What happened and what was the outcome? Answers tend to be arrived at using quantitative methods Why - Why did people behave the way they did? Answers tend to be arrived at using qualitative methods

11. So What? What - goals are - how measured – Supports Accountability (External) Why – goals were met or not – people issues – Often the focus of Assessment/Evaluation (Internal)

12. Ways of Knowing A Third Paradigm Pragmatism Use what ever approach and methodology that will best answer the question. Constructivist Positivist there are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher) there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables

13. Charge from President Likins “As Chair of the NCAA Division I Presidential Task Force on the Future of Intercollegiate Athletics, I have both the opportunity and the responsibility to examine the culture of higher education athletics programs in America. My responsibilities begin at home. My task force experience as well as events at colleges and universities around the country led me to conclude that it is appropriate to examine the environment defined by the culture of high-level athletic competition here at The University of Arizona. At the national level a serious movement to strengthen the academic experience of student-athletes is well underway. We must examine the presence of any cultural influences here that have the potential of distorting the shared values of academics and athletics in higher education.”

14. Focus of the Study The study was divided into two phases. Phase I, mostly quantitative in nature, was designed to look somewhat broadly at the two fundamental study areas: • Student athletes’ academic experience; and • Student athletes’ culture and environment. Using qualitative methods, Phase II was designed to dig more deeply into the critical issues identified from Phase I. Detailed, targeted recommendations would be developed from these two phases.

15. Athlete’s academic experience Academic preparation Progress to degree Student engagement Academic support Post-eligibility academic experience Other academic issues Athlete’s culture and environment Alcohol and drugs Dishonesty and cheating Gambling Racism Sexism Sexual behavior Violence Weapons Domain of the Study

16. Phase 1—Quantitative • Board of Regents and NCAA reports on athletes’ grades and graduation rates (ten year history). • Institutional student records data. • Health and Wellness Survey, administered annually to assess student alcohol, drug, and sexual behaviors (seven year history). • Dean of Students Code of Conduct and Code of Academic Integrity violations (five year history). • Coalition on Intercollegiate Athletics (COIA) report. • Annual University Survey of Graduating Seniors, assessing student engagement, satisfaction, and other academic issues (five year history). • NCAA GOALS survey (student-athletes and student non-athletes). • Additional UA-specific survey questions appended to GOALS survey (student-athletes and student non-athletes).

17. Phase II—Qualitative • Focus groups • Interviews • Follow-up surveys • Populations: • Students (athletes and non-athletes) • Coaches • Faculty • Staff • Advisors

18. Athletic culture spans a spectrum of areas, some of which (alcohol use, for example) have become fairly well understood over the years from quantitative methods, others of which (physical violence, weapons use) need the deeper exploration and understanding provided by qualititative methods. The Value of a Mixed Methods Approach

19. The formative and summative aspects of this study, which is more like program evaluation in many ways than like a traditional research study, lend themselves to a mixed methods approach. The Value of a Mixed Methods Approach

20. Audiences and stakeholders consist of some people who respond best to systematic, inferentially rigorous, quantitative data, but also others who want to hear the complex, richly detailed stories derived from the individuals who are immersed in the culture. Different audiences resonate to different approaches. The Value of a Mixed Methods Approach

21. Mixed methods also means that multiple methods have been employed, which is essential to using triangulation as a means to establish greater validity of the findings. Any single approach used in the messy world of social science research will be flawed. If a wide array of methods leads to consistent conclusions, then analysts and stakeholders feel more confident about conclusions and recommendations. The Value of a Mixed Methods Approach

22. Questions??