Mixed Methods Dissertation Design Challenges and Opportunities: A Sequential, Explanatory Approach to Studying Students’

1. Mixed Methods Dissertation Design Challenges and Opportunities: A Sequential, Explanatory Approach to Studying Students’ 2/4 Transfer Robin LaSota, PhD, University of Washington Post Doctoral Research Associate, University of Illinois Urbana-Champaign (UIUC) Office of Community College Research and Leadership (OCCRL) Higher Education Collaborative Presentation February 25, 2014 College of Education at the University of Illinois Urbana-Champaign

2. Presentation Overview • Evaluation Criteria – Mixed Methods Research • Dissertation Design Summary • Quantitative and Qualitative Strands • Design Choices • Paradigm • Findings • Policy Inferences • Analytical Reflections • Mixed Methods Analysis Revisited

3. Some Criteria for Evaluating Mixed Methods Research • Clearly stated research purpose(s) and reasons for design choices about methods and mixing • Alignment between research methods and purpose • Conclusions offer the “value added” from mixed methods approach • Conclusions include an analysis of study legitimacy, limitations, and paradigms used for mixed method analyses

4. Reasons for Mixing Methods Triangulation Complementarity Development (of sampling, instrumentation, measurement decisions, etc.) Initiation (of theoretical perspective, new questions, contradictions) Expansion of breadth and range of inquiry From: Greene, Caracelli, and Graham (1989) in Creswell and Clark (2011), p. 62.

5. My Research Purpose • To offer “complementary findings regarding strategies, policies, and factors including improvements to students’ upward transfer, particularly for students most at risk of not transferring and attaining a baccalaureate degree” (p. 251) • Each strand of inquiry (quantitative and qualitative) has separate research questions and purposes

6. A First Area of Inquiry Q1.1 How do student behaviors, community college characteristics, and state policies and conditions influence students’ upward transfer probability? Q 1.2 How do these factors, policies, and conditions influence upward transfer probability, particularly for low-income and first-generation community college students?

7. Student, College, and State Factors Influencing 2/4 Transfer

8. A Second Area of Inquiry Q2.1 What are promising practices in colleges and states aimed at improving students’ upward transfer, and how may they constitute a system of support? Q2.2 How do leaders engage in ongoing innovation around these practices?

9. Explanatory Sequential Mixed Methods Design

10. Research Paradigm – Quantitative Strand • To “quantitatively measure theoretically and empirically-derived factors (from prior research) found to produce a greater likelihood of students’ transfer from public two-year to four-year institutions in a recent, nationally representative sample, particularly for students who are low-income and first-generation to earn a bachelor’s degree.” (pp. 7-8)

11. Sampling Design Choices from BPS • N=5010 community college students; weighted N=1,528,900 • Students not co-enrolled in two or more colleges • BPS nationally representative longitudinal population survey of all postsecondary entrants • BPS not representative of states, colleges, or CC entrants • State and community college factors investigated build upon existing literature

12. Which State and Community College Factors May Influence Transfer? • State articulation and transfer policies? Very little, if any influence(Roksa, Kienzl, Goldhaber & Gross) • State cooperative agreements? Maybe. (Kienzl) • Community college practices? Depends. Perhaps not much. (MDRC) • Community college expenditures? Slight/student services expenditures(Gross and Goldhaber). None(Stange). • Community college smaller size, higher faculty-to-student ratio? Yes. (Bailey et al., Gross & Goldhaber) • Degree of college mission stratification (emphasis on transfer-oriented programs vs. non-transfer oriented, e.g. health/vocational/technical). Influential. (Dougherty) • Proximity and selectivity of nearest public four-year institution? Maybe.(Rouse)

13. Rationale for Multi-Level Modeling Methodology Results of Unconditional Model or Intra-Class Correlation– • State location – explains 2% of variance in 2/4 transfer probability • Primary college attended – explains 6% of variance • Student characteristics– explains most of the variance Therefore, used multi-level logistic regression – • Randomly varying intercepts and slopes between colleges and states for • Low-income, first generation • First generation, not low income • Planned to transfer at time of entry • Declared major in health/vocational/technical field

14. Positive Predictors Associated with Upward Transfer Probability

15. Negative Predictors Associated with Upward Transfer Probability

16. College Characteristics: Association with Upward Transfer • Proportion of associates’ degree completions in health/vocational fields (neg., p<.10) • College transfer-out rate (2% increased odds of transfer) in regression without analysis of random effects by slope • County-level unemployment (neg., p<.10) Not sig. = i.e. per-student expenditures for instruction or student services, distance to nearest public four-year institution, distance to nearest non or less-selective four-year institution, faculty-to-student ratio, community college enrollment size, percent of full-time faculty, percent of full-time students

17. State Policies’ Association with Students’ Upward Transfer • Main Effects Model, Random intercepts only, no varying slopes • + 35% higher 2/4 transfer odds: State with one standard deviation higher Gross State Product Per Capita in 2003 • None of the State Articulation and Transfer Policy Components explained variance in 2/4 transfer probability. Policy Components • Transfer data reporting • State transfer incentives • State transfer guide • Transferable general education curriculum • Statewide cooperative agreements • Common course numbering • Statewide articulation/transfer policy

18. Regression Results: Slopes for Sub-Populations that Vary by College and/or State Random and Fixed Effects Model showed that these factors moderate 2/4 transfer probability for these populations. Low-Income, First Generation: • Higher gross state product • Common course numbering • College transfer-out rate First Generation, Not Low Income: • Higher Gross State Product • Common Course Numbering Planned to Transfer (vs. Not Transfer Intending): • College transfer-out rate Health/Vocational Major (vs. business/undeclared): • State articulation/transfer policies not sig. • Transfer-out rate not sig. Green = Sig. at p<.05; Gray=Not Sig, p>.05

19. Findings:Quantitative Inquiry Strand • Affirmed prior research about ambiguous or unknown effects of state transfer and articulation policies • Offered new evidence about the role of state common course numbering in increasing first-generation students’ transfer • Influential college-level factor – College mission focus; college’s transfer-out rate • Full-time attendance and transfer intention are particularly influential student factors

20. Some Implications: Quantitative Strand • Promising areas for policy intervention, esp. in high schools • Help students create specific plans for obtaining a bachelor’s degree aligned in a specific field and outline a transfer pathway • Promote continuous full-time attendance and advising with incentives and accountability • Widely promote available state resources and policies for improved transfer

21. Rationale for Case Study Design • Goal: To explore and identify possible state policy actions and college policies or practices that enhance student 2/4 transfer probability • Structure analysis for meaningful contrasts relative to the goal • States and Colleges with Higher Transfer vs. Average Transfer Rates (within their state) • Policy Innovative States in Articulation and Transfer • Colleges Engaged in Data-Use and Innovation • States with significant CC sector and states & colleges with student populations of interest

22. Research Paradigm – Qualitative Strand • To offer “a more nuanced and detailed picture” of college and state practices supporting students’ upward transfer, that “built on and complemented my quantitative findings” (p. 101) • Use framing literature to “interpret institutional cultures, data use, and innovation in practices at the six colleges”…and “to help pinpoint the context for innovation…[as well as] transfer-related practices.”

23. State Case Selection: Florida, Georgia, and Washington • Used OLS regression to find states performing above average in transfer, controlling for state and student population characteristics • Considered prior research on policy innovative states in transfer and articulation • Chose states with a considerable proportion of postsecondary students enrolled in two-year colleges and with racial/income diversity

24. College Selection: Above-Average and Average Performer • Used OLS regression to find colleges performing above average in transfer, controlling for college and student population characteristics • Consulted State Higher Education Executive Officers (SHEEO) from each state and Aspen Prize Top 120 data • Used SHEEO advice and college’s participation in Achieving the Dream indicators of data-use and innovation

25. Qualitative Methods • Interviews with state policy officials in articulation and transfer (N=20) • Interviews with college administrators, faculty, and student affairs staff (N=110) • Individual interviews and focus groups with students (N=49) • N=179 overall Semi-structured, one-hour interviews Analytic Strategy: Analytic Memo Writing and Data Synthesis

26. Findings: Qualitative Inquiry Strand

27. Findings – Advising in Above-Average Performers Transfer not a universal outcome or push for all students… College-level systems of support for transfer generally constrained… Above-average colleges generally have: • Academic leaders who champion students’ transfer and successfully engage others in this work • Mandatory student advising models • Student affairs staff dedicated to coaching students’ on transfer

28. Findings – Advising in Above-Average Performers Above-average colleges also tend to have: • Faculty contracts which include student advising hrs. • Faculty and staff engaged in planning out-of-class supports and enrichment experiences for students that aid transfer • Campus supports for TRIO and similar STEM programs for low-income, minority, and first generation students • Key Support for Stronger Advising: Active communication/coordination with public and private four-year institutions within major fields by administrators and faculty

29. Findings – State Policy as a Context for Colleges’ Innovation

30. Common Course Numbering: Lessons Learned from Florida Moderating positive influence of common course numbering (CCN) for first-generation college students from quantitative inquiry… • CCN Proxy for a more robust transfer policy context? • CCN built from communication across lower and upper division faculty and programs • Florida: CCN in place for 30 yrs; created when 2 yrs and 4 yrs were governed in one system

31. Some Implications:Qualitative Inquiry States: • Incentives and support for college-level innovation • Support for measuring innovation effectiveness • Build transfer into performance accountability Colleges: • Collaborative problem-solving re: transfer • Broad implementation of personalized learning & transfer advising • Incentives to be transfer champions States and colleges: • More efficient, accessible processes to using data for decision support about students’ transfer

32. Analytical Reflections - Quantitative • Data Limitations • BPS measures of academic and social integration • State policy measures binary coding • No adequate measure of policy strength for the period • Available college-level data mostly not predictive of transfer • Not a causal inference multi-level model • Does not examine reasons for stopping out or mixed attendance – e.g. role of financial aid

33. Analytical Reflections - Qualitative • Examined broad scope of practices affecting transfer probability (from pre-college to graduation check/final term advising) rather than one or two specific innovations • Used analytical memo writing not software-based coding methodology • Inductive approach to claim formulation rather than deductive hypothesis-testing • Different framing literatures inform each strand, complementary analyses, not necessarily integrated

34. Mixed-Methods Analysis Revisited • “Complementary strengths, overlapping weaknesses” of the two strands • Strands answer different questions, however, from the QUAN findings, QUAL cases consider: • Role of common course numbering • Students’ experience with academic advising and colleges’ work to reform it • Additional policy supports for improved upward transfer (not represented in original coding)

35. With Appreciation • To all the participants in my study • To Debra Bragg and OCCRL for post-doc support • To NISTS for the honor of the award and presentation with you • To my chair, Bill Zumeta • To my co-advisor, Marge Plecki • To my committee members: • Mike Knapp • Bob Abbott • Jennie Romich • To my fellow doctoral students • And to IES and AIR for funding Sponsored by the US Department of Education, Institute of Education Sciences (#R305B090012) and the Association of Institutional Research Dissertation Grant

36. Questions/Discussion Follow up contact information: rlasota@illinois.edu Office of Community College Research and Leadership College of Education at Illinois occrl@illinois.edu occrl.illinois.edu