How to Assess Student Learning in Arts Partnerships Part II: Survey Research

1. How to Assess Student Learning in Arts Partnerships Part II: Survey Research Revised April 2, 2012 Mary Campbell-Zopf, Ohio Arts Council Mary.Campbell-Zopf@oac.state.oh.us Craig Dreeszen Ph.D., Dreeszen & Associates craig@dreeszen.com

2. Your Instructors

3. Our intended learning outcomes • Understand the use and limitations of written assessment instruments • Write effective survey questions • Understand survey distribution options • Interpret survey data • Qualitative data analysis • Quantitative data analysis • Report survey findings

4. Written Assessment Instruments • Evaluation forms • Tests • Pre- and post-tests • Standardized tests • Surveys

5. Using Written Assessments Good reasons to use: • Efficient • May reach large samples • Test qualitative conclusions from small samples • Provide quantitative data • Enables reticent people to participate May not be appropriate if: • Uncertain what to ask • Questions are ambiguous • Questions are complex • Respondents not literate

6. Two Sequencing Options Start with interviews or focus groups: • Test questions • Learn likely responses • Explore conclusions • Create fixed-response questions • Distribute to larger sample Start with survey: • Explore ambiguous survey findings in interviews or focus groups • Investigate reasons for survey findings

7. Survey Steps • Decide what data is needed • Design questionnaire • Select sample • Acquire lists • Pretest • Distribute and remind • Analyze and report

8. Determine Needed Data • Learning outcomes • To what extent were did participants learn what was intended • Unintended outcomes • Program satisfaction • What liked best • What needs improvement • Recommendations • Marketing advice • Advocacy evidence

9. Write Questions Open-ended questions • Short answer • Long answer • Prompted Fixed-response questions • Yes/No • Rating Scale • Choice (one or multiple) • Matrix rated • Ranked Ask basic demographic questions too

10. Review Sampling • Select representative sample of larger population • Or survey entire population • Acquire mailing or email lists

11. Open-ended Questions

12. Fixed-Response Questions [Replace sample with OAC AP grantee examples]

13. Practice Writing Questions • Try writing an open-ended question • Then, write a fixed-response questionand response choices

14. Paper and Online Surveys • Mailed surveys • Envelopes • Address labels • Postage • Cover letter • Survey • Return envelope and postage • Handout surveys • Online surveys • SurveyMonkey.com • Zoomerang.com • Email invitation • Survey • May provide links to survey results and organization website • Survey closed notice

15. Online Survey Features • Screening questions • Skip logic • Quick response • Inexpensive • Track responses • Send reminders to unresponsive email addresses

16. Test Survey • Send early draft survey to partners • Revise • Send next draft to small sample of your population • Ask for feedback • Look for ambiguity or misunderstanding • Revise final survey

17. Distribute Surveys Paper surveys • Address and mail • Remind (newsletter, announcements…) Handout surveys • Distribute and collect Online surveys • Time right • Email invitation with link to survey URL • Web site links • Social media links • Track responses • Send reminders

18. Statistical Significance Response Rate Number of responses divided by number of surveys distributed 90 responses 200 surveys sent = 45% response rate Deduct undeliverable surveys Margin of error When sampling populations, larger samples have smaller margin of error Use online calculator If you survey 200 students and 90 respond, margin of error = 7.7 If 120 respond, margin of error is 5.7 Some apparent results may be due to chance – not statistically significant No margin of error if entire populationsampled, e.g., all participants in your program

19. Reduce large amount of data Display it in notes, tables, or charts Make sense of it Statistical analysis of numbers Content analysis of words Analysis of data

20. For each question, read all answers Look for patterns and clusters Code similar statements Count clusters of similar statements Select typical quotes Write summary Do the same for next question Making sense of words

21. Count Average Note patterns and clusters Note most frequent responses Cross tabulate to compare variables Check statistical significance Write summary Making sense of numbers

22. Report Results with Text “Findings: The Arts Partnership Program works as intended. The Arts Partnership program furthers the OAC strategic plan, especially Goal 3, to “Help citizens of all ages learn and thrive through the arts.” Quantitative evidence cited Just over half of AP grantees (55%) cited quantitative evidence to support their evaluation results in the final report.” Executive summary Interpret results Key-point summary Headings Explanation Data Quotes Appendix with detailed data and methods

23. Display Results with Graphics • Data tables • Bar charts • Histograms • Pie charts • Line charts • Scatter plots

24. Report Results Report intended learning objectives and actual learning outcomes Summarize activities Suggest best practices and needed improvements Acknowledge funders and partners Provide back-up data Sample OLGA report form

25. Use Results • Draw conclusions • Report to stakeholders • Use data to improve programs and account to funders authorizers