Evaluating “By Appointment” Programs

1. Evaluating “By Appointment” Programs Pamela J. Way, Ph.D.

2. “By appointment” programs • Are designed to provide individualized, one-on-one interactions • Are usually one hour in length • May be peer to peer

3. Before the evaluation…. • Write mission statements for your agency and your programs • These statements define anticipated, broadly-stated goals and outcomes • Inform the daily “doing business” process • Inform the evaluation path

4. Building your assessment • How do you decide what to evaluate? • Choose the program • Choose one of the program’s components • Choose your target population

5. What do you want to know? • Customer usage/satisfaction • This is baseline information; start here before designing a more complex evaluation • Learning outcomes – tutors? • Learning outcomes – tutees?

6. A word about definitions Part of deciding what you want to know is knowing what – exactly – that is! What’s your definition of “learning outcomes?” Work in your groups to come up with some possibilities….

7. Assessing resources • Time • Money • Space/equipment • Human resources

8. What kinds of data are there? • Quantitative • Qualitative • “Mixed” methods

9. Doing the legwork • Review current literature in the field • Find, evaluate, and select from existing instruments • Write your own questions

10. The IRB An “IRB” – Institutional Review Board – review is required of all projects involving human participants • Minimize the risk to human subjects • Ensure fully informed consent • Promote equity in human research

11. When you’re ready to roll… • Pilot the survey • Go “live” • Review incoming data

12. Data analysis • Descriptive • T-tests • ANOVA • Correlation/Regression • Categorical analyses

13. Going public Although an evaluation effort will serve you and your program well, publishing or otherwise disseminating your results (conferences, etc.) will lend credibility to your program and your institution.

14. References – quantitative statistics Statistics (Hays). A very dense but very complete textbook on designing studies as well as analyzing data. Statistics (Freedman, Pisani, & Purves). Not as dense or complete as Hayes, but somewhat easier to navigate. PDQ Statistics (Norman & Streiner). Down and dirty stats. Great for beginners. Mixed Methodology: Combining Qualitative and Quantitative Approaches (Tashakkori & Teddlie).

15. References – qualitative statistics Collecting and Interpreting Qualitative Materials (Denzin & Lincoln). Survey Research Methods (Babbie). This is often considered the qualitative research “Bible.”

16. References - instruments Developing and Using Tests Effectively (Jacobs & Chase). More geared to faculty, but many useful suggestions; easy to read and process Psychological Testing: Design, Analysis, and Use (Friedenberg). Includes sources of information about tests. Mental Measurements Yearbook. Full text info and reviews of many standardized tests in the educational field.

17. References - other Publication Manual of the American Psychological Association. Essential if you want to publish your results. On-line data bases:Academic Search Premier, PsycARTICLES, PsycINFO. Google Scholar is also quite good. SPSS – reasonably priced quantitative analysis software. Relatively easy to use.

18. Discussion Board • Please go to the Discussion board and respond to the prompt you will find for this module. • Thank you.