510 likes | 941 Views
Data Collection Methodologies. Module 7. Overview. Definition of data collection Data sources Characteristics of data collection strategy Questionnaires, focus groups, interviews: pros and cons. Data Collection. Observable, measurable units of analysis
E N D
Data Collection Methodologies Module 7
Overview • Definition of data collection • Data sources • Characteristics of data collection strategy • Questionnaires, focus groups, interviews: pros and cons
Data Collection • Observable, measurable units of analysis • Relevant to the purpose of the evaluation • Specific to the unit of analysis • Appropriate procedures and tools • Collected, recorded, analyzed, to draw conclusions
Data Sources • People: program managers ,administrators, beneficiaries, donors, program staff, government officials, etc. • Documents: program strategy, annual plans, progress reports, financial reports, • Infrastructure observation: building and grounds, laboratories, program sites, • Observation of group dynamics: nature of meetings, Board meetings, etc.
Tools for Data Collection • Questionnaires • Interviews • Focus groups • Observation • Performance tests
Experimentation • To ensure an optimal validity of data collected, am ideal model would the experimentation approach. • Given resources constraints in many evaluations experimentation is not always feasible nor accepted • More sophisticated techniques exist to ensure validity and reliability of data
Data Collection Strategy • Consider the following: • Size, geographical dispersion, literacy level of target population • Resources available for data collection (time, people, money, technology) • Sensitivity of issues to evaluate • Reliability and validity sought • Needs of the evaluation • Developing an adequate strategy is key!
Questionnaires and Surveys • Printed or electronic list of questions • Distributed to a predetermined group • Completed and returned to evaluator
When to use a questionnaire • Target population is literate, and large (over 200) or dispersed • Need for categorical data • Need for quantitative data and statistical analysis • You have access to people who can process and analyze this type of data • Need to examine responses by sub-groups
6 Steps To Effective Questionnaires 1. Develop the purpose 2. Draft the items 3. Sequence the items 4. Design the questionnaire 5. Pilot the questionnaire 6. Develop a strategy for data collection and analysis
1. Develop the Purpose • Specific to the information you need • Add value • Relate to the evaluation • Refined to relevant questions only
2. Draft the Items Formulate into the following item types: • Fill in the blank • Question requiring one word answer • Multiple Choice • Question with 4-8 defined answers • Comment-On • Open ended question, written response
2. Draft the Items (cont`d) • List • Requesting list of 3 to 5 short responses • Can be in order of importance • Likert Scales • Statement to be ranked for agreement • Rank • List of items to be ranked by importance
3. Sequence the Items • Group into themes, or by question type • Rewrite questions, eliminate redundancies • Where necessary, use filter questions • guide respondents to different branches • Sequence with logical progression
4. Design the Questionnaire • Group similar themes together • Organize into titled sections • Format into booklet • Use space effectively • Decide on the need for pre-coding • Include a title and introductory explanation
5. Pilot the Questionnaire • Test questionnaire with small group • Clarify wording • Identify ambiguity • Revise • Retest if necessary
6. Develop the Strategy • Select the sample • Determine distribution plan • Develop a cover letter • Prepare distribution package, and send • Monitor responses, follow-up • Enter data • Analyze data
Effective Questionnaires • Gather valid information • Provide logical and organized data • Facilitate process of data entry and analysis • Simple, timely and cost efficient
Interviews • Dynamic process of gathering information • Personal interaction between two individuals • Follow a guide or protocol • Require recording of responses
When to use interviews • Need to have the views of key informants • Target population is small • Need for depth of information rather than breadth • You have reason to believe that people will not respond to a questionnaire • Informants are willing and available to meet
7 Steps to an Effective Interview 1. Define interview purpose 2. Draft interview questions 3. Sequence the questions 4. Consider process needs 5. Prepare introduction and closing 6. Prepare to record responses 7. Pilot test questionnaire
1. Define the Interview Purpose • Define objectives for the interview • Link objectives to the evaluation • Identify the data needed • State the purpose for each part of interview • Focus on content rather than process
2. Draft Interview Questions • Carefully drafted and worded • Minimize ambiguity • Combine open-ended with close-ended
3. Sequence the Questions • Vary question types • Open-ended (What do you think about…) • Close-ended (Did the program begin in 1998?) • Organize questions into themes or sections
4. Consider Process Needs • To manage the interview process, use: • transitions to move to each theme • reminders to paraphrase or summarize • lists of standard probes
5. Prepare Introduction and Closing • Introduction to set tone and rapport • state purpose of interview • explain who the interviewer is • confirm confidentiality and use of data • Closing to signify the end of interview • reinforce usefulness of interview • thank respondent • confirm arrangements for follow-up
6. Prepare to Record Responses • For key informant interviews: • tape record (with permission) • take brief notes on notebook • detail notes immediately after interview • For normative interviews: • check boxes • space for efficient data recording
7. Pilot-test Interview • Validate the interview: • protocol • content • flow • arrangement to record responses • If more than one interviewer, validate: • ability to use protocol
Sample Protocol - Program Evaluation Interview • Introduction: “We have been asked by.. To conduct a .. We would like to discuss the ...” • Theme 1 - Program relevance • Theme 2 - Program success • Theme 3 - Program cost effectiveness • Conclusion: “In conclusion, what are the main strengths and weaknesses of this program?”
Effective Interviews Are: • Planned sufficiently • clear objectives • Focused on the process • control over context and process • two-way communication • awareness of non-verbal communication • Conducive to establishing trust • Dependant on a cooperative interviewee
Effective Interviewing Skills Include: • Active Listening • relaxed, eye contact, verbal feedback • Openness and Empathy • accepting what is said without judgement • Paraphrasing/Summarizing • rewording to clarify response • Controlling the Process • setting context, shaping responses
Focus Groups • Group discussions of a predetermined issue • Members share common characteristics • Moderated by facilitator • Responses are recorded
When to Conduct Focus Groups • You need rich description to understand client needs • Group synergy is necessary to uncover underlying feelings • You have access to a skilled facilitator • Informants are willing to speak in groups
5 Steps to Effective Focus Groups 1. Define the purpose 2. Select an appropriate sample 3. Determine date and location 4. Develop Focus Group Guide 5. Plan to record responses
1. Define the Purpose • Based on the objectives of the evaluation • Identify information needed and why • Develop the purpose of the focus group
2. Select an Appropriate Sample • Individuals who are: • informed about the topic of discussion • in the best position to share information • represent all target populations • Pre-screen participants to ensure they meet the required characteristics • Select 6 - 12 participants
3. Determine Date and Location • Make clear arrangements regarding • purpose • starting time • how long it will last • neutral convenient location • participants feeling at ease
4. Develop Focus Group Guide • Construct effective questions • content related to evaluation purpose • open-ended to facilitate discussion • limit to five or six • Sequence questions • ensure natural flow • facilitate transition from one to the next • lead logically through variety of topics
5. Plan to Record Responses • Taping gives full record of discussion • inform and attain consent in advance • Detailed notes are indispensable • taken by assistant to moderator • organize data in predefined categories • document relevant observations • avoid cues about value of responses • underline or highlight reference points
Effective Moderation Includes • Active listening • Facilitation of discussion • Awareness of group dynamics • Suspension of personal bias • Alertness to time schedule • Probing for elaboration • Affirmation of responses • Respect for different perspectives
Observation: when to use • When the setting is important for the evaluation • When the interaction amongst people is a key factor of the evaluation • Before developing a questionnaire, in order to get a feeling for some of the issues • To understand cross-cultural issues
Steps in Observation • Define the population or setting to observe • Select representative places to visit/observe • Develop an observation schedule with the dimensions to observe • If there are many observers, train observers for consistency
During Observation • Prepare for an entry phase so the observer does not impact the setting too much • Record data according to the observation dimensions selected: interaction, environment, light, smell, infrastructure, etc. • Record qualitative (how good, how bad) as well as quantitative (how many) data • Exit from the setting and record data immediately
Performance Tests: When to use • When there is a need to define aptitudes or abilities • When the study requires to select good performers (as opposed to people to train for good performance) • When there is a need to diagnose performance problems
Categories of performance tests • Aptitude tests • Attitude scales • Psychological tests • Performance tests
Choosing a performance test • First criteria is to match the test to the purpose • Second is to match the resources (time, money, technology), context and culture to the purpose • If the purpose is to screen out the best individuals, aptitude tests make sense • If the purpose is to assess progress, pre- and post-activity tests make sense
Scoring performance tests • Criterion referenced (Can a person perform according to a given standard?) • Normative (Who is the best in a group?)