Qualitative Methods in Institutional Research

1. Qualitative Methods in Institutional Research Workshop CAIR Annual Meetings November 12, 2003 Judith Richlin-Klonsky, Ph.D. UCLA Student Affairs Information & Research Office jrichlin@saonet.ucla.edu http://www.sairo.ucal.edu

2. The goals of the workshop • To outline the issues that need to be considered when undertaking a qualitative research project. • To describe experiences with specific qualitative research techniques. • To respond to questions about using qualitative methods in institutional research.

3. Our agenda • Introductions • Overview of qualitative methods in institutional research (JR-K) • Quick & Easy Focus Groups (WW) • Blitzkrieg Ethnography (BB) • “Fishbowl” Discussions (JR-K) • Q&A/Discussion

4. What do we mean by “qualitative research?” Examples of qualitative data-collection techniques: • Individual interviews • Group interviews • Focus groups • Ethnographic observation • Participant-observation • Archival and/or documentary research • Journaling (see handout for descriptions)

5. What do we mean by “qualitative research?”(2) • Not just a set of techniques. • Not just “non-quantitative” research methods.

6. What do we mean by “qualitative research?” (3) Research design and methods that help us fulfill specific goals as researchers: • By gathering data through participants’ own words and/or (inter-)actions, we seek to learn about the phenomenon being studied from their pointof view. • By spending time with experts (those directly involved), we seek to learn in depth about the phenomenon being studied.

7. What do we mean by “qualitative research?”(4) Unlike most quantitative institutional research, qualitative research typically doesn’t: • Measure. (Ex: “What is the rate of graduation?”) • Produce a quantified product (Ex: “86% of our students are satisfied with their learning experience.”) • Seek to establish a causal relationship. (Ex: “Summer attendance shortens time to degree.”)

8. “But it’s not scientific…” “Whether qualitative techniques are considered scientific description or explanation, or whether they offer merely a prelude to scientific inquiry, depends on the philosophical stance taken with respect to the nature of social science. Indeed…some have argued that qualitative data provide the only empirical foundation on which social science can be built” (Walker, p. 3).

9. Research design: When is qualitative research most useful? When the research topic is: • Sensitive. • Complicated. • Unmeasurable. • Concerned with interaction and/or process.

10. Research design: When is qualitative research most useful? (2) When the research objective is: • To learn about research subjects who are few in number. • To brainstorm (generate as many ideas as possible, not reduce to a single number). • To identify important issues to be explored more broadly through quantitative methods. • To interpret, illuminate, or illustrate.

11. Examples of data-collection issues

12. Data collection issues All: How to maintain subject privacy and demonstrate to IRB that you are prepared to maintain anonymity or confidentiality?

13. Data analysis Analysis is aimed at • Answering the research question. • Identifying themes, categories, or types.

14. Data analysis (2) • “Analysis of qualitative material is more explicitly interpretive, creative, and personal than in quantitative research, which is not to say that it should not be equally systematic and careful” (Walker, p. 4).

15. How do we know when we have “valid results?” Saturation When not hearing new types of experiences, observations, roles, etc. Triangulation within qualitative methods Data gathered at different times or places, at different levels Between multiple observers of the same phenomenon. Triangulation between methods Quantitative and qualitative data

16. Reporting issues “The report on a qualitative study will be completely different than that on a quantitative study. The structure for the report will have emerged from that used in the analysis: the findings will be presented in terms of impressions gained, as hypotheses rather than as firm conclusions. [continued]

17. Reporting issues (2) …There will be no tables… …and no references to numbers or percentages; at most the researcher will use such phrases as ‘most respondents’ or ‘around half’ or…‘a minority view was….’ The researcher will be mainly concerned to identify and describe the range of behavior and options rather than to indicate…how many hold each view. In all cases the description of beliefs, attitudes, and motivations should be supported by evidence in the form of verbatim quotations form the interviews and discussions” (Morton-Williams, in Walker, p. 41).

18. Reporting issues (3) How to encourage the audience to give appropriate attention and weight to the data? • Not too little: avoid being dismissed because of “small N.” • Not too much: avoid emotional effects of direct participant voices.

19. Selected resources: qualitative methods • Sharon B. Merriam, Qualitative Research and Case Study Applications in Education, John Wiley & Sons, 1998. • Sharan B. Merriam & Associates, Qualitative Research in Practice: Examples for Discussion and Analysis, Jossey-Bass, 2002. • Jean J. Schensul and Margaret D. LeCompte, Editors, Ethnographer’s Toolkit, Altamira Press, 1999.

20. Selected resources: qualitative methods (2) • Robert Walker, Editor, Applied Qualitative Research, Gower Publishing Company, 1985. • Elizabeth Whitt, “’Making the Familiar Strange’: Discovering Culture,” in Cultural Perspectives in Student Affairs Work, George D. Kuh, Editor, American College Personnel Association, 1993.

21. Transition • The following slides describe use of student “fishbowl” discussions.

22. UCLA Student “Fishbowl” Discussions Judith Richlin-Klonsky, Ph.D. UCLA Student Affairs Information & Research Office jrichlin@saonet.ucla.edu http://www.sairo.ucla.edu

23. The “Fishbowl” Discussions: 1999-2003 • Format • Background • Permutations

24. Five “Fishbowl” permutations

25. Recruitment and planning(see appendixes of reports at http://www.sairo.ucla.edu, “Recent reports”) • Campus leaders only as general participants. • Invitations to be considered for participation: • “ Cold calls” based on lists generated from student records, using parameters related to gender, race/ethnicity, class standing, source direct/transfer. • Referrals from Student Affairs directors. • Ad in Daily Bruin • Offered incentive as “thanks” for participation • Made clear it would be videotaped • Topic areas identified in dialogue with Student Affairs leadership.

26. Selection • Participation not automatic, not promised. • Each candidate contacted personally for a short conversation. • Selection based on perceived contribution to group “mix,” ability and interest to share experience, energy. • Frequent follow-up contact prior to event.

27. Room setup • Varied by year, depending on anticipated size of audience and location: classroom; stage; in law school classroom “well”; tiered lecture hall; meeting room. • Goal is to arrange students so that they can easily conduct conversation, but that audience can see and hear them • Facilitator and assistant sit with students.

28. Types of subjects addressed • Academic experience • Co-curricular activities • Personal and academic development • Perceptions of campus climate • Access to/use of student services • Residential life (especially crowding) • Use of technology

29. Afterwards • Refreshments • Mingling

30. Reviewing and summarizing the data • Review notes taken during discussions. • View duplicated copy of videotape. • Make notes from videotape, transcribing comments that concisely capture main points being made. • Use notes to summarize student comments in major areas covered in discussion.

31. Summaries • Full reports and executive summaries posted on the SAIRO web site at http://www.sairo.ucla.edu (“Recent Reports”).

32. Examples • Campus Diversity “I came from a very heterogeneous background, mostly minorities… My thinking was when I left high school, oh cool, I’m going to UCLA and there’s going to be all these other people, but this time it won’t be cliqued, meaning that all the pilipinos will be together and all the [shrug] will be together. But when I went to college, it was more of the same… and I was surprised especially with all of the diversity issues coming up and especially with the abolition of affirmative action… I kinda find it weird that each of their ethnic group clubs have outreach for their ethnic group and it is cliqued even more.” (ud, 21:24:32)

33. Examples • Campus diversity “It’s such a diverse campus, but people [of the same background] still congregate together.” [ld] “When I walk around campus, the only ones who are smiling at you are the ones that look like you.” [ld]

34. Examples • Transition to UCLA “I’ve been told for the last two years [in hs], ‘College is going to be such a different experience for you.’ And then I got here and it was more different than I ever thought. You can’t describe it. You can’t explain it. It’s just a transition. And it’s just a transition. And it’s a different part of life that you have to go through…. Just be prepared to change your outlook on academics and everything.” [ld]

35. Examples • Learning to persevere A member of the senior group used the metaphor of UCLA as “an emotional roller coaster” and the image resonated with other participants as well. Part of the bumpy ride they described was the discouragement they felt after arriving as someone who had been very “impressive” in high school and then feeling “thrown in a pile” with other high-achieving students. Another student agreed that at UCLA you have to “cope with ups and downs, have to deal with discouragement regularly.”

36. Examples • Learning to persevere Other seniors described in various ways their personal developments in light of the challenges they faced in a large, complex, and competitive setting: UCLA does grind you into a tough person. In applying for jobs, you have that toughness going for you – created by adversity. UCLA has taught me to persevere.

37. Examples • Student culture: time to degree “Never once during this fifth year have I regretted that decision. It was the best thing. It helped me mature and I think I would really encourage first year students…to consider a fifth year…” (21:14:57)

38. Reports • Full report and executive summary distributed to all those invited to the event: Student Affairs AVCs and directors, Chancellor and his executive team, deans. • Posted on web site.

39. Uses • Some in audience moved and/or energized by hearing from students, especially in early years and in 2003 “good-bye” discussion. • Concrete changes made, e.g., changes in hours of operation for some Student Affairs offices • Contribution to planning for various areas (e.g., technology development, external affairs). • Format also used in session for student loan counselors and planned as a forum for staff experiences.