Incorporating an Evaluation Plan into Program Design: Using Qualitative Data

1. Incorporating an Evaluation Plan into Program Design: • Using Qualitative Data • Connie Baird Thomas, PhD • Linda H. Southward, PhD • Colleen McKee, MS • Social Science Research Center • Mississippi State University • Positioned for Progress Conference • April 30, 2014 • Jackson, MS

2. Workshop Goals • Understand the difference between qualitative and quantitative data • Learn how to capture qualitative data in a rigorous objective way • Explore best practices related to qualitative data collection

3. PROJECT ABC – Literacy Intervention Program • Conceptualize—“If we do X, then Y will result” • ACTIVITIES: (X) • Pilot program in 10 schools • Group tutoring • One-on-one tutoring • Curriculum • Take home materials • Parent participation • Trained facilitators • OUTCOME (Y): Proficient reading by fourth grade

4. What is Qualitative Research? Data collection method in which the researcher: • Seeks an in-depth, “rich” view of topic • Explores an idea or topic to describe or give it meaning • Asks participants broad, general questions • Collects detailed views of participants or subject • Analyzes and codes data for description or themes

5. Characteristics of Qualitative and Quantitative Research Qualitative Quantitative Goal: predict, confirm Design: structured Sample: random, large representative Data collection: surveys, questionnaires Data interpretation: objective • Goal: describe, discover • Design: flexible, emergent • Sample: small, non-random • Data collection: interviews, observation • Data interpretation: subjective

6. Qualitative Research … • Does • Explain “how” and “what” • Describe “what happened” • Explore processes over time • Explore meaning of event to those involved • Does not • Prove/disprove hypothesis • Measure variables • Test theories • Show relationship between variables • Make generalizable statements

7. Methods of Qualitative Research Methods Data “Words don’t fly individually- they fly in flocks.” Geoffrey Nunberg- University of CA, Berkley School of Information • In-depth Interview • Focus group • Case Study • Direct Observation

8. Methodology In-depth Interview • Advantages • Open structure allows for individual to express detailed opinions and experiences • Interviewer can easily follow-up and/or clarify statements • Appropriate for sensitive or controversial topics or special populations • Limitations • Requires skilled interviewer • Responses may be biased • Not generalizable • Time/labor intensive One-on-one conversation between interviewer and respondent regarding a specific topic Sometimes audio/video recorded

9. Methodology Focus Group Advantages • Group setting often stimulates additional thoughts and ideas • Gives more insight into why certain opinions are held • Can better understand processes (changes that have occurred over time) • Observe nonverbal responses Limitations • Group/time management • Peer pressure among respondents • May stress resources Focused small group (usually 6-10 participants) discussion guided by a leader or moderator Useful for populations with shared experiences

10. Methodology Case Study Advantages • Provides rich, detailed information • Lends to understanding of processes, interactions and context • Uses different methods of data collection (interviews, observations, document analysis, etc.) Limitations • Applicable only to site or event under study (limited generalizability) • Requires extended time and effort In-depth study of a single unit- can be a person, organization, program, etc. Sometimes referred to as “telling a story” When to Use Case Study* Is the program is unique? Is it premature to establish impact measures? Is there unexplained variation in program impacts? Will understanding the program implementation in detail help design future programs? *E. Balbach. Using Case Studies to do Program Evaluation. (1999) California Department of Health.

11. Methodology Direct Observation Advantages • Helps to understand the social, physical, political, and economic context in which participants live • Can provide important information that was previously unknown • Observe participants in their natural settings • Complements other forms of data collection (triangulation) Limitations • Time consuming/resource intensive • Difficult to document everything- may have to rely on memory • Objectivity is a challenge (observer bias) • Observer effect Observing the participants in their natural environment and setting- either as becoming a part of the setting or being strictly an observer

12. Issues to Consider • Purpose of research (re: evaluation) • Subjective method (reliability and validity) • Researcher bias • Labor intensive • Cost

13. Words fly in flocks…

14. Example: • Assessing the Impact of the Mississippi Healthy Student Act: Year Four Report

15. Questions

16. Thank You • Contact us at: • mskidscount@ssrc.msstate.edu • www.kidscount.ssrc.msstate.edu • Tel: 662-325-7127 • Connie Baird Thomas, PhD. • Tel: 601-407-2742 • Email: connie.baird@ssrc.msstate.edu • Linda H. Southward, PhD • Tel: 662-325-0851 • Email: linda.southward@ssrc.msstate.edu • Colleen McKee, MS • Tel: 662-325-0851 • Email: colleen.mckee@ssrc.msstate.edu