Welcome 2013 HIV/AIDS Community Information Outreach Program Evaluation Webinar

1. Welcome 2013 HIV/AIDS Community Information Outreach Program Evaluation Webinar

2. Online HIV/AIDS Resources http://sis.nlm.nih.gov/outreach/training_outreach.html

3. Presenters LaFrancis McMurray, MPH Health Education Specialist Oak Ridge Associated Universities lafrancis.mcmurray@orau.org Arletha Williams-Livingston, Ph.D., MPH Evaluation and Organizational Capacity-building Consultant Transforming Communities, Inc. transformingcomm@gmail.com

4. Purpose The purpose of this webinar is to build organizational evaluation capacity, skills and knowledge, to discuss program evaluation requirements and to connect participants to evaluation technical assistance.

5. Objectives Understand the importance of program evaluation Effectively evaluate ACIO projects Access evaluation technical assistance as needed

6. Today’s Topics What is Evaluation? WhyEvaluate? Building OrganizationalEvaluation Capacity ACIOEvaluation Requirements,ResourcesandTechnical Assistance

7. What is Program Evaluation? Program – Your campaign, action, training, intervention Evaluation –Is the systemic investigation of the merit, worth, or significance of your work Program Evaluation – Is a way to evaluate the specific projects and activities of community groups and organizations

8. In other words… Program Evaluation is a combination of: – Good Reporting – Good Detective Work – Good Historical Documentation and Storytelling Program Evaluation helps organizations see how and in what ways they are really helping the community, OR how and if an organization is achieving its goals and objectives.

9. Three Main Types of Program Evaluation Process Evaluation Outcome Evaluation Impact Evaluation There are many more…

10. Process Evaluation Describing and documenting the course of a program For Example: – Whoare the Members or Participants? – Whatactivities were conducted? – Howmany people were in attendance? – WhenandWhereare events and meetings being held? – What staff was there? – Whowas involved in the planning? – Whowas involved in the implementation?

11. Outcome Evaluation What were the immediate effects of the program? Changes in skills, knowledge, attitudes, and behavior For example: – Did the participants learn anything? – Do the participants have a different attitude? – Do the participants do something differently or do something new? – Was there a campaign victory?

12. Impact Evaluation What were the ultimate effects on the community? For example: – How was the community affected? What were the effects? Three types of Impact evaluation – Immediate (more like an outcome) • Knowledge of housing shortage issue – Intermediate • Advocacy work and heightened policy making and community awareness of housing issue – Ultimate • Decrease in percentage of the target population who are in need of housing assistance

13. Why Evaluate?

14. Evaluation Is Used for: Program Development Mass Campaigns Policy Formations Issue Advocacy Organizing Communities Funding Applications

15. Program Evaluation is used to… Determine if goals were met Show effectiveness and impact of a program Provide evidence of the process and outcomes

16. Questions to Ask When Developing an Evaluation Plan Whois the Evaluation for? Howwill the results be used? Whatdo we need to find out? Where should we gather the information? Whydo we want to find that out? Whenwill the findings be needed?

17. Building Evaluation Capacity

18. Capacity Building Values Interdependence Cultural diversity Contextualized interpretation Empowerment

19. ACIO Information Access Categories Evaluation Information Measures Questions Tools Resources

20. Information Access Category I Information Retrieval

21. Information Retrieval To provide or improve access to electronic AIDS- and other health-related information resources by the organization or by the clients they serve and to implement access to state-of-the-art AIDS-related information tools via the Internet and/or mobile phone technologies. This may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.

22. Project Examples Online living library database Information retrieval skills development training for health outreach workers and local community based organizations Increase access to critical information for at risk youth using a multimedia hub-spoke model of online, social media and mobile platforms Mobile-Internet-based HIV/AIDS Education: Linking the Public to Science Purchase of computer equipment/software for cyber lab

23. Measures Awareness Marketing/awareness of information Community partnerships/community engagement Access Retrieval skills Access to computers/mobile/Internet Proof of access/number of visits/types of resources

24. Evaluation Questions Awareness In what ways is information advertised to potential consumers/clients? How are community partnerships engaged to help provide information access to residents? Access How often are clients retrieving information? How are clients accessing information via computers, Internet and mobile? How many times & which types of resources clients are accessing?

25. Sample Evaluation Tools Awareness Marketing plan objectives, outcomes Number of MOAs/MOUs Number & quality of community partnership/community advisory board meetings Observations of community partner engagement Access Social Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics

26. Resources Social Media/ Health communications planning with evaluation component https://www.healthcommworks.org/Account/LogOn?signInArea=SocialMediaWorks US Department of Education Evaluating Online Learning www.ed.gov/admins/lead/academic/evalonline/evalonline.pdf Evaluation strategy for mobile device learning http://www.salt.org/salt.asp?pn=procabs&ss=l&key=153320&pid=2406

27. Information Access Category IISkills Development

28. Skills Development Training to develop skills in accessing or using HIV/AIDS-related information including the use of NLM online databases and mobile resources as well as general AIDS information resources in order to improve services and benefit the community. This may include training for the staff of the agency, development of training programs for clients and patrons, training programs for other agencies, etc.

29. Project Examples Capacity-building training for youth serving professionals; and peer-to-peer education Information retrieval skills development training for health outreach workers and local community based organizations Training classes and workshops on NLMs online resources and social media sites Training for new HIV positive peer educators/counselors as online instructors and facilitate new training classes featuring health literacy

30. Measures Training Development of trainings/types of: access/utilization/ general computer skills/database training Training target populations: staff/clients/ patrons/agencies/special populations Number of trainings/trained users Training satisfaction Training outcomes: knowledge/skills

31. Measures (cont’d) Utilization Types/amount of resources/information utilized: Internet/social media/email/mobile/brochures/factsheets/face-to-face Utilization outcomes: Knowledge/skills/empowerment/efficacy/service improvement/community outcomes

32. Evaluation Questions Training What types of trainings are being developed? Who is the target training population? How many people are being trained? (Attendance/completion) Are consumers satisfied with trainings? What are the training outcomes/skills developed? Utilization What type of information consumers are utilizing? How is that information being utilized? What are the outcomes associated with consumer utilization of information resources?

33. Sample Evaluation Tools Training Training Development Template Sign-In sheets Pre/Post assessments Satisfaction surveys Utilization Social Media monitoring tools i.e. Tweetdeck, Website analytics i.e. Google Analytics Questionnaires, Surveys, Qualitative analysis of responses

34. Resources Training development template Sample pre/post skills assessments Writing measurable learning outcomes http://www.gavilan.edu/research/spd/Writing-Measurable-Learning-Outcomes.pdf Evaluating Training Programs-Kirkpatrick’s 4 levels http://www.wa.gov/esd/training/toolbox/tg_kirkpatrick.htm

35. Information Access Categories III and IVResource DevelopmentEquipment Acquisition

36. Resource Development To support an organization to meet the needs of their clients or communities by developing specific educational or informational materials such as culturally or language specific fact sheets, bibliographies, guides or other products.

37. Equipment Acquisition To improve access to electronic AIDS and other health-related information resources which may require the purchase of hardware (computer equipment), software, or contractual relationships necessary to obtain a connection to the Internet for the purpose of accessing AIDS-related information or sharing AIDS-related information.

38. Project Examples Produce and syndicate culturally and linguistically appropriate messages Ready Reference Guides to person with HIV/AIDS or at high risk for HIV Culturally competent and age appropriate information in English and Spanish emphasizing prevention, risk reduction and skills building Develop and facilitate access to factsheets on the treatment of HIV/AIDS and related information written in nontechnical language and translated into several languages

39. Measures Resource needs Target population needs assessment/types of resources needed Resource development Resource materials created: type such as cultural specific, bibliography Resource materials utilized/sustained/maintained/updated Purchases of hardware/software to develop resources Utilization of consultants/contractual relationships to develop resources Access Purchases of hardware/software to access and share information Utilization of consultants/contractual relationships to access and share information

40. Evaluation Questions Resource needs Has a resource needs assessment been conducted for the target population? If so, what were the outcomes of that needs assessment? Resource development What types of resource materials are being developed? How are the resource materials being utilized/sustained/maintained/ updated? Was hardware/soft purchased to develop resource material? If so, what kind and how was it used? Were consultants/contractual relationships used to develop resource material? If so, who and what work did they complete?

41. Evaluation Questions (cont’d) Access Was hardware/soft purchased to help agencies provide access/share materials? If so, what kind and how was it used? Were consultants/contractual relationships used to help agencies provide access/share materials? If so, who and what work did they complete?

42. Sample Evaluation Tools Resource needs Outcomes of Community/resource needs assessment Resource development Resource development template Resource material tracking (How utilized/sustained/maintained/updated) Comparison checklist for equipment/software purchases Consultant/contractual reports Access Equipment/software inventory Utilization log Consultant/contractual reports

43. Resources TeamLab-University of Southern California Designing materials for particular audiences/understanding culture http://teamlab.usc.edu/learn/roadmap/audience.html CenterLink-The Community of LBT Centers Conducting Needs assessment http://www.lgbtcenters.org/how-to-do-a-community-needs-assessment.aspx

44. Resources (cont’d) Harvard Family Research Project Evaluation Exchange, Understanding Cost-Benefit and Cost-Effectiveness Evaluation http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/methodology-15/at-what-price-benefit-cost-analysis-and-cost-effectiveness-analysis-in-program-evaluation TechSoup.org Website for Non-Profits and technology. Planning and how to articles http://www.techsoup.org/support/articles-and-how-tos/articles-by-subject?subjectid=14

45. ACIOP Evaluation Requirements Quarterly reports Final reports Individual agency evaluation conference calls Evaluation technical assistance

46. What’s next? Individual agency calls to discuss: – Agency Evaluation capacity – Project evaluation plans – Technical assistance Mid Year Webinar – Milestones – Evaluation needs of agencies – Final reports

47. Q & A

48. Contact Information Arletha D. Livingston, Ph.D., MPH, CHES Email: transformingcomm@gmail.com Phone: 404.368.9252