Program Evaluation

1. Program Evaluation Just enough knowledge to be dangerous Belinda De La Rosa and Megan Mustafoff Off. of the Dean of Students and Counseling Ctr.

2. What to expect • Definitions • Know your Audience & Population • Goals • Methodologies • Report Writing • If time—walk you through an evaluation thought process

3. What is program evaluation? • Program evaluation is the systematic assessment of the processes and/or outcomes of a program with the intent of furthering its development and improvement. As such, it is a collaborative process in which evaluators work closely with program staff to craft and implement an evaluation design that is responsive to the needs of the program. For example, during program implementation, evaluators can provide formative evaluation findings so that program staff can make immediate, data-based decisions about program implementation and delivery. In addition, evaluators can, towards the end of a program or upon its completion, provide cumulative and summative evaluation findings, often required by funding agencies and used to make decisions about program continuation or expansion. Source:http://www.washington.edu/oea/services/research/program_eval/faq.html

4. How is evaluation different than research? • Evaluators use many of the same qualitative and quantitative methodologies used by researchers in other fields. Indeed, program evaluations are as rigorous and systematic in collecting data as traditional social research. That being said, the primary purpose of evaluation is to provide timely and constructive information for decision-making about particular programs, not to advance more wide-ranging knowledge or theory. Accordingly, evaluation is typically more client-focused than traditional research, in that evaluators work closely with program staff to create and carry-out an evaluation plan that attend to the particular needs of their program. Source:http://www.washington.edu/oea/services/research/program_eval/faq.html

5. How is evaluation different than assessment? • The primary difference between evaluation and assessment lies in the focus of examination. Whereas evaluation serves to facilitate a program's development, implementation, and improvement by examining its processes and/or outcomes; the purpose of an assessment is to determine individuals or group's performances by measuring their skill level on a variable of interest (e.g., reading comprehension, math or social skills, to mention just a few). In line with this distinction—and quite common in evaluating educational programs where the intended outcome is often some specified level of academic achievement—assessment data may be used in determining program impact and success. Source:http://www.washington.edu/oea/services/research/program_eval/faq.html

6. Other Types of Evaluations • Needs Assessments • Return on Investment (ROI) • Environmental Scan • Learning Outcomes • Can you think of others?

7. Know the Program • Important to work closely with program staff • Educate yourself as much as possible (content analysis, document analysis, procedural audits, interview all stakeholders, peer reviewed journal articles)

8. Goals of Program Evaluation • Formative • Process • Summative

9. Know Your Audience • Audience • Who are you doing this evaluation for? • Are they stakeholders? • Do they benefit directly from the program? • Are there political considerations (can you mediate them)? • Internal or external evaluators • Formal or informal investigation • When is Institutional Review Board paperwork required (security)?

10. Know Your Target Population • Get to know as much as you can about your target population • Work with program staff closely • Observe, research, investigate, talk • Ask questions • Once you know as much as you can decide on a sampling technique based on the goals of the evaluation • Incentives

11. Office of the Dean of StudentsAssessment Inquiry • Do you routinely evaluate your unit? • If so, on what cycle? • Type of Evaluation* • If you evaluate routinely how far back do you have data? • Do you generate any routine reports? • If so, on what cycle, to whom, and how is it disseminated? • Are other types of data collected or are one-time studies available? • Does your unit have an individual who works primarily with data? If so, please provide their name and contact information.

12. Theory and planning • Know or establish program goals • Logic model (inputs, outputs, 1-3-5 goals) • Learning outcomes (specific and measurable) • Measure program goals, gaps and need • What information do you need to know • Is this information already collected somewhere • How will you collect the information you don’t already have—spend more time using it than collecting • What will you do with the information collected

13. Methods & Tools • Quantitative • Frequencies (head counts) • Surveys (phone, mail, online, in person) • Qualitative • Focus groups • Individual interviews (phone, in person) • Case Studies • Mixed methods research • Uses both qualitative and quantitative methods

14. Project designs • Pre-test/post-test • Case-control • Cost-benefit analysis • Procedural audit • Quasi experimental

15. Quantitative methods • Surveys • Likert scales (strongly agree—strongly disagree) • Dichotomous response (yes or no) • Continuous response (income) • Categorical response (partner status) • Frequencies • Survey data • Head counts • Time count • Space size and quantity (2 20x20 rooms) • Can you think of other frequencies relevant to your project?

16. Qualitative methods • Case studies • One on one interviews • Focus groups • Content analysis (websites, policies, internal documents)

17. Sampling • Probability sample • Random sample • Random stratified sample • Non probability sample • Convenience sample • Snowball sample • Quota sample • Judgment/expert samples • Methods to recruit • Flyers • List serves • Newspapers • Classrooms • Can you think of others?

18. Bias and Power • Non response bias • Coverage bias • Selection bias • Sample size

19. Ethical Considerations • IRB must be consulted for exemption and reviews • Special protected populations • Children, elderly, minorities, students • Fact check with your participants and be sensitive to positional power. • Secure data and consent forms physically and electronically. • Destroy data according to institutional or IRB recommendations • Follow FERPA, HIPAA or other federal, state or local laws regarding information gathering, storage, and sharing for your population http://irb.illinois.edu/

20. Recruiting • Incentives paid in advance/completion • Individual incentive • Money, coupons, gift cards, swag • Lottery incentive • Support of community/preliminary notification • Get the support of respected individuals in that population to encourage participation • Emotional appeal what will this do for them or people they care about. • Follow up requests and deadline extensions • Brevity of instrument

21. Analysis of qualitative data • Coding • Data is reviewed and segmented. Each segment is assigned a code or theme. Reports are prepared by summarizing the frequency and distribution of the codes or themes. • Summative evaluation • Review of the objectives of the evaluation and then the use of content analysis, utilizing coded notes, to pull recurring regulator concepts judged on being either internally homogeneous or externally heterogeneous. The process is then repeated by two other independent researchers in order to determine and verify reliability of the themes pulled. Inter-coder reliability increases the reliability of this analytical method.

22. Analysis of Quantitative data • Frequencies • Contingency tables (categorical variables) • Odds ratios • Regression • Power Analysis

23. Report writing outline • Executive summary and abstract • Literature review • History of the program • Evaluation • Methodology • Results • Discussion • limitations • Outcomes • recommendations

24. Report writing • Present data in ways administrators can utilize it to transform operations, re-envision goals, expand services or collapse services. • Always include an executive summary and never just write an executive summary. • For frequently written reports that are just summaries of data be sure to document methods and procedures for institutional memory. Don’t forget to record outcomes and changes due to reports. • Use visual aids such as charts, graphs, flow charts, etc. as necessary and to a level that will be usable by your audience

25. Planning & Budgeting • Mission, Vision, Goals developed • Unit goals aligned with Division • Programs based on assessable goals, with performance indicators • Planning to available financial resources • Funding based on priorities and results Assessable Outcomes • Improvement • Judging The Data Against Desired Results • Comparing Data To Peers • Reporting Data to internal constituents • Demonstrating accountability to external stakeholders • Proposing improvement initiatives based on assessment findings • Determine and effect change based on the data • Separate out data-reporting from planning for improvement • Improving assessment methods Implementation (Everyone in Division implements goals) Application of Findings Instrumentation Tracking Data Collection Analysis • Evaluation • Needs/interest assessments • Web-based surveying • Assessment of learning outcomes • Surveys: NSSE, Student Satisfaction, IPEDS, CORE, etc. • Administrative program reviews • Benchmarking studies • HR Data • Program, service & facilities evaluations • Contributions to institutional accreditations • Student Record Data Remember that Data Flows Whitney, Karen. “Fear not the data…use the data to tell your story.” University of Illinois Student Affairs. Champaign. 25 Jan. 2010.

26. For More InformationSee these Websites • http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf • http://managementhelp.org/evaluatn/fnl_eval.htm • http://www.tgci.com/magazine/A%20Basic%20Guide%20to%20Program%20Evaluation.pdf • http://extension.psu.edu/evaluation/ • http://www.programevaluation.org/ • http://www.uwex.edu/ces/pdande/evaluation/ • http://www.portal.mohe.gov.my/portal/page/portal/ExtPortal/MOHE_MAIN_PAGE/Tender_Contract/BUDGET/files/PLANNING_PROGRAM_EVALUATION.pdf • http://ocw.jhsph.edu/courses/fundamentalsprogramevaluation/lectureNotes.cfm • http://www.musc.edu/vawprevention/research/programeval.shtml • http://teacherpathfinder.org/School/Assess/assess.html • Sudman, S & Bradburb, N (1986). Asking Questions: A Practical Guide to questionnaire Design. San Francisco: Jossey-Bass Publishers.