Methods and Tools in Qualitative Assessment for Libraries

1. Methods and Tools in Qualitative Assessment for Libraries Susan M. Weaver Kent State University, East Liverpool Campus Director of Library Services Professor of Library and Media Services

2. Research is an inquiry process that has clearly defined parameters and has as its goal the discovery or creation of knowledge …and/or investigation of a problem for local decision making.

3. Qualitative Research • One of 2 major approaches to methodology in social sciences • In-depth study of human behavior and the reasons that govern human behavior • Narrative oriented and uses content analysis methods

4. Qualitative vs Quantitative QUANTITATIVE: • focuses on numerical or statistical data • is closer to a “scientific” approach to data collection and analysis • sees the world as observable events and facts that can be measured • often called the “positivist paradigm”

5. “Not everything that can be counted matters and not everything that matters can be counted.” From a sign hanging in the office of Albert Einstein

6. Qualitative vs Quantitative QUALITATIVE: • focuses on the context (environment) in which events occur • focuses on social interactions that are complex and evolving, making them more difficult to measure or interpret numerically • is the “interpretivist paradigm”

7. Four Common Methods of Qualitative Research Observation Interviewing Group Discussion HistoricalStudy

8. Interviewing • Collects information about personal perceptions of events, processes and environments. • Can be formal with set questions… • Or…informal and anecdotal • Cornerstone of Qualitative Research QualitativeMethods

9. Observation • Systematic recording of observable events • Insight into unconscious behavior • Reveals hidden attitudes or views • Determine patterns of behavior, contextual factors affecting behavior and interactions between subjects. Qualitative Methods

10. Group Discussion • Focus Groups: 6 – 12 participants • Similar interests or backgrounds (not a representative sample) • Interactions among the group members as well as the moderator/facilitator (researcher). • Nominal Group Technique Qualitative Methods

11. Historical Study • Study of the historical record (budgets, policies, administrative documents) • Relies on documentary evidence as well as first-hand observations

12. ASSESSMENT • Assessment “is” research but not all research is assessment! • Fits into the definition of research • …and/or investigation of a problem for local decision making • Assessment is the process of determining the value of services and resources offered by the library • Only users can determine the value of library offerings.

13. EXERCISE 1

14. Observation • “Present” oriented, recording what occurs as it occurs • “Reality Verifier,” can check to see if people do what they say they do • Behavior is observed in a natural setting • Can study people who cannot be interviewed (children, the elderly) • Can study people unwilling to be studied

15. Interviewing • Immediacy • Mutual exploration: both parties can explore meanings (of ? and answer) • Investigation of causation…”why” they behave in the way they do • Personal Contact: friendlier more personal approach to data collection • Speed

16. Historical Investigation • Historical perspectives are useful when pressure is to “change.” • Understanding of library economic trends useful in understanding and defending budgets

17. Combining Methods Interviews + Focus groups + Surveys + Document Analysis “Triangulation” is collecting information using multiple research strategies “Mixed methods” is currently the preferred terminology for using multiple research strategies

18. Group Discussion: Focus Group A Focus Group session is a small group discussion, usually consisting of 6 to 12 participants, guided by a facilitator and used to gain an understanding of participants’ attitudes and perceptions on a particular topic.

19. Why “Groups?” Groups dominate organizational life! In Academic Libraries Users students, faculty, non-users, post-graduate students Staff administration, reference, cataloging Outside university administration, Groups granting bodies

20. Advantages • Speed and economy:a lot of data (perspectives) can be collected in a single, brief session. Low cost. • Transparency:participants know what is going on. • Interaction:between participants • Flexibility:“real-time” clarification and feedback • Open-endedness:exploration of unforeseen issues, more detail than surveys • Non-verbal communication data: anger, discomfort, excitement • Good will

21. Disadvantages • No (or little) quantitative data • Lack of privacy • Difficulty in recording and analyzing open-ended responses • Moderator bias (inhibiting, monopolizing or misdirecting discussion) • Unwillingness of participants to reveal unpopular views • Participants who feel they are not equal to other participants

22. The Process 1. Planning 2. Recruiting the Participants 3. Conducting the Discussion sessions 4. Analyzing & reporting

23. Planning • Groups • How much data is needed…How many groups? • reliability • Scheduling • The location • The time • The Questions • Choose 6 – 10 questions, grouping like questions • Ask controversial questions near the middle

24. Planning: Writing Questions • Open-ended Questions: Ask questions that encourage description and depth Do like the new web page? What is your opinion of the new web page? • Use simple clear language How would you rate the federated search engine? How would you rate the new search engine that searches multiple databases all at one time?

25. Planning: Writing Questions • Avoid biased questions: This year we put new state-of-the-art computers in the Library. What is your opinion of the workstations? What are your thoughts about the library’s computer workstations • One Concept per question: How would you rate the electronic journal collections such as the EJC and JStor? How would you rate JStor? The EJC?

26. Planning: Writing Questions • List areas to probe How do you learn about library programs and services? Probe: What suggestions do you have for improving communication? • Choose relevant subjects • Consider the experience of the subjects

27. Exercise

28. The Process 1. Planning 2. Recruiting the Participants 3. Conducting the Discussion sessions 4. Analyzing & reporting

29. Recruiting the Participants • Who should be included • Homogenous groups • Size of the group • Recruitment • Must be willing, dependable • Enticements • Prepare name tags

30. Conducting the Session • Select a Moderator • Moderator duties: • welcomes group • provides an overview of topic • outlines discussion rules • asks the questions • initiates and guides discussion • Maintains focus and fairness • Asks follow-up questions • summarizes proceedings and closes the meeting

31. Conducting the session • Video or audio record session • Transcribe into text • Take Notes (Memoing) • Observation notes • Methodological notes • Theoretical notes • Personal notes • Write follow-up comments

32. Analyzing & Reporting • Because we are already experts in the topic we are studying • Because the process immerses us into the culture being investigated • Because this tutorial focuses on assessment rather that “knowledge creation”… It may be a simple as spreading out the notes and writing about what was seen and heard!

33. “…it is not important to know everything in order to understand something.” C. Geertz, The Interpretation of Cultures, p. 20.

34. Analyzing DATA REDUCTION DATA DISPLAY CONCLUSION DRAWING AND VERIFDICATION

35. 1. Read a unit of text Assign a category 2. Read next unit of text Assign a new category 3. Is it the same category?

36. EXERCISE

37. Detailed Data Analysis • Coding • Open Coding • Selective Coding • Content Analysis • Computer Analysis

38. CAQDASComputer Assisted Qualitative Data Analysis Software COMMERCIAL Qualrus ($399) Atlas.ti ($498) MAXqda ($415) Ethnograph ($295) HyperRESEARCH ($370) NVivo 7 ($495) OPEN SOURCE (free software) AnSWR WEFT QDA Transana

41. According to G. E Gorman & Peter Clayton inQualitative Research for the Information Professional

42. Advantages Assists calculation & quantification Eases process of writing & rewriting Encourages systematic work Forces organization of data Disadvantages Depends on quality of data entered Risks data loss Removes researcher intellectually from data CAQDAS

43. Advantages Focuses Analysis Support referencing sorting and coding Facilitates creation of statistical and graphic display Disadvantages Risks decontextualization (sense of context) Uses technical processing methods on data more suited to other methods CAQDAS

44. What Computers can do… • Enter and edit notes • Coding notes • Storing and retrieving data • Memo-writing and theory building • Displaying and mapping data

45. ReportingWriting Qualitative Research Reports • Introduction to the problem • Literature review • Discussion of research methodology and procedures • Presentation of the findings • Statement of the implications and conclusion

46. Think of it this way… write the Tell them what you are going BEGGINNING to tell them (intro, literature review, methodology) write the MIDDLE Tell them (findings) write the Tell them what you’ve told END them (implications and conclusions)

47. Exercise Quickly look over the qualitative research article and, with a pen, indicate these parts: • Introduction(to the problem) • Literature review • Discussion ofresearch methodologyand procedures • Presentation of thefindings • Statement of the implications(Discussion) and conclusion

48. Your Focus Group Plan