Instruments and Data Collection

1. Instruments and Data Collection New Mexico AmeriCorps April 20, 2006 Sue Hyatt, Project STAR Coach

2. Agenda • What are instruments? • Measuring outputs and outcomes • Linking indicators and targets to instruments • Data Collection Issues • PM Plan

3. 1. Develop Logic Model & Project Plan 2. DevelopPM Plan Performance Measurement Six Steps 6. ReportResults ContinuousProgramImprovement 5. AnalyzeData 3. Developand Test Instruments 4. CollectData

4. Types of Performance Measures • Outputs • Intermediate Outcomes • End Outcomes

5. What is an Instrument? • Document or form used to collectinformation from data source • Data source: Person, place or group that provides you with information to measure indicators • Examples: Attendance rosters, tally sheets, questionnaires, checklists • Adapt existing instruments or develop your own

6. Advice about Instruments • Make sure instrument will measure indicators • Consider who will administer instrument • Consider who will complete instrument • Consider when to administer instrument • Consider data accessibility

7. Instrument Considerations • Connect to results • Measures the identified PM • Aim for appropriate method • Measures the correct population (data source) • Lower burden of use • e.g. Captive audience, (part of project) • Balance between “best” method and “manageable” method • Assure that sites have instruments or summary tool

8. Exercise What Does the Instrument Measure?

9. Indicators and Targets For each result (output, intermediate outcome and end outcome), you need to specify: • Indicator • Target • Instrument COMMON ISSUE: Indicators and targets are NOT tied to the specific instrument being used!

10. What is an Indicator? • Information you collect to determineprogress toward achieving results • Concrete and measurable • Answers the question, “What is myevidencethat something has occurred?” • Output indicator usually begins with “Number of...” • Outcome indicator usually begins with “Percent of...”

11. What is a Target? • Describes amount of change youanticipate achieving each year • Based on indicator – specific, measurable amount of change that will signify success • Outcome Target: • Who will experience change? • How many people will experience change? • How much change will occur (compared to baseline)? • Over what time period will change occur (one year)?

12. Advice about Targets • Consider available programresources. • Consider community context. • First year targets may be “educated guesses”. • Be prepared to adjust targets based on experience. • Targets should be ambitious but also realistic. • Targets are based on indicators and the content of the instrument.

13. Exercise How Targets are Built from Instruments: Tutoring Data Aggregation Exercise

14. Data Collection Issues • Can you access the data source (beneficiaries, organizations)? • Does an instrument (site or project) exist? • Are they willing to give you data? (burden, capacity) • Can you get the data when you need it? • Are your helpers (e.g. members) trained? • Do you have a mechanism to collect summarized data from sites?

15. Methods • Logs, Tally Sheets, Attendance Rosters • Survey, Questionnaire (Check list, Index, Scale) • Interview (Interview Guide) • Service Recipients Interview • Expert/Provider/Key Informant • Observation (Observation Guide) • Case Study (Measurable change of a case over time) • Focus Group (Focus Group Guide) • Secondary Data (Existing records and information) • Test / Experimental Designs

16. Exercise Program Instrument and Data Collection Review and Discussion

17. Additional Resources: • Contact Project STAR • http://nationalserviceresources.org//sites/star/ • 1-800-548-3656 • star@aiweb.com