Unit 7: Using your evaluation – Communicating, reporting, improving

1. Unit 7: Using your evaluation – Communicating, reporting, improving Who What (Content) How (Format) When

2. “Think like a wise man, but communicate in the language of the people.” − William Butler Yeats

3. Why communicate? The proper function of evaluation is to speed up the learning process by communicating what might otherwise be overlooked or wrongly perceived… Success is to be judged by… success in communication… Payoff comes from the insight that the evaluator’s work generates in others. - L. J. Cronbach Cronbach, L.J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass, p. 8.

4. Plan with the end in mind.

5. Types of communications - reports • Written report - long, short, summary • Success story • Impact statement; spotlight • Elevator story • Press release • Media appearance • Public meeting • Memo, email, fax, postcard • Newsletter • Personal discussion • Bulletin, brochure • Display/exhibit • Audio/video presentation

6. Communication plan WHO? WHAT? HOW? WHEN?

7. WHO - Potential Users • County board/City council (elected and appointed officials) • Funders • Program Committee • Participants • Team / Staff • Volunteers • Collaborating agencies; partners • Businesses; business groups • Police – law enforcement • Schools boards, parent-teacher organizations • Community • Church organizations • State legislators • Professional organizations • You

8. WHO is the Audience? INTERNAL Primary audience or secondary audience?? 1. 2. 3. 4. EXTERNAL 1. 2. 3. 4.

9. WHAT – will you say, report, communicate?? • What does the audience care about? • What do you want the audience to have? • What type of information resonates with the audience: numbers, quotes, stories?

10. HOW will you communicate – what formats will you use? • Report • Impact statement • Executive summary • Personal discussion • Oral presentation • Photo display • Press release • Newsletter, bulletin • Poster • ??????

11. HOW – what format, style will you use?? • Format depends on purpose and audience • Written, oral • Short, long • How does the audience prefer receiving information? How does the audience best learn? • What do you have the resources to do?

12. WHEN will you communicate? • Quarterly? • Opportunistically? • When requested? • ???

13. Reporting outline follows your logic model

14. Formal evaluation reports typically include… • Abstract/executive summary • Introduction • Purpose of the evaluation; key questions • Program background, description • Methods/procedures • Data sources • Data collection procedures • Sampling • Limitations • Results • Discussion • Conclusions/recommendations • References • Appendices

15. Effective communications • Tailor message to issue and audience • Avoid jargon and technical terms • Be clear, concise • Use active voice • Eliminate wordiness • Check writing, grammar • Be accurate, balanced, impartial • Be timely • Use graphics, quotes, photos, real stories • Consult a communications specialist • Write-rewrite-rewrite

16. Myths • One report is enough. • People read written reports. • Complex analysis and big words impress people. • Oral reports have the same effect as written reports. • Describing limitations weakens report. • Everything should be reported. • The audience knows why they are getting the report.

17. Ingredients of a Good Success Story • Involves human interest: catches your attention • Demonstrates behavior change; action taken • change in practice that results in benefits to individual, community, environment... • Uses words of/meaning for people involved • Indicates profit to the individual, if appropriate • Based on reliable, credible information • Includes numbers and narrative • Presents balanced, fair assessment

18. Make sure the evaluation isn’t ignored • Engage stakeholders in doing the evaluation: in the design, data collection, in analysis, interpretation of results… • Get the information to the right people • Address issues that people think are important • Keep it in front of people • Present it in time to be useful

19. Discuss limitations Written reports: • Be explicit about your limitations Oral reports: • Be prepared to discuss limitations • Be honest about limitations • Know the claims you cannot make • Do not claim causation without a true experimental design • Do not generalize to the population without random sample and quality administration (e.g., <60% response rate on a survey)

20. Reporting results to the media All Media: • Avoid using too many statistics. • Focus on the key points. • For quotes, speak more globally about the issue. • Always give the source and timeliness of your stats. It’s the “news peg.” Steve Busalacchi Director, News & Information Wisconsin Medical Society

21. Reporting results to the media Radio and TV: • Do not offer exact statistics – ear cannot track. “73.6% of respondents” vs. “Nearly three quarters of those surveyed” • Don’t go into great detail. Have backup info ready. Steve Busalacchi Director, News & Information Wisconsin Medical Society

22. Using graphics What do I want my audience to learn – remember – from my graph or table?

23. Using graphics • Title • Clear units of measure • Date(s) data collected • Simple, straightforward design without “clutter” • Font size 10 point or larger • Explicit data source(s) • Sample size, if applicable for the audience

24. Graphs • Bar graphs: show comparisons • Stacked bar • Multiple bars • Vertical, horizontal bars • Pie charts: show parts of a whole • Line graphs: show progress over time

25. Bar graph • Compare groups of client outcomes • Compare points in time Each group you compare gets a “bar”

27. Horizontal Graph

28. Vertical Graph

29. Column Chart

30. Pie charts: parts of a whole • Useful for displaying proportions • Percentage of clients achieving the outcome • Percentage of participants who are boys • Percentage of sites that are rural • Ensure individual parts add to 100%

31. Pie Chart

32. Line graphs: time lines • X axis conveys the time • Y axis is the variable of interest • Multiple lines or multiple line graphs show performance of different groups

33. Line Graph

34. Area Graph

35. Pictogram End of November Total!

36. Graphing using Microsoft Excel • Click on graph icon on toolbar, or select “insert” and “chart” • Proceed through chart wizard • Select the type of graph you want • Highlight the range of data to include • Set up titles, axis labels, scales, legend • Finish • Edit the chart using drop down menu for charts • Click on chart to highlight it • Use the top toolbar selection “chart” to edit parts of the chart