THE OBJECTIVES-ORIENTED EVALUATION APPROACH Case Study: When Hard Questions Are Asked: Evaluating Writing Centers. Jam

1. THE OBJECTIVES-ORIENTED EVALUATION APPROACHCase Study: When Hard Questions Are Asked: Evaluating Writing Centers. James H. Bell Gabriel M. Telleria Program Evaluation and Policy Analysis Spring 2008 January 23, 2008

2. THE OBJECTIVES-ORIENTED EVALUATION APPROACH • Why is evaluation important? • Why do we evaluate? • Evaluation is “the systematic collection and interpretation of evidence, leading, as part of the process, to a judgment of value with a view to action” (Beeby, qtd. In Wolf 3, qtd. in Bell 8, 2000).

3. THE OBJECTIVES-ORIENTED EVALUATION APPROACH • Establish broad goals or objectives • Classify the goals or objectives • Define objectives in behavioral terms • Find situations in which achievement of objectives can be shown • Develop or select measurement techniques (indicators) • Collect performance data • Compare performance data with behaviorally stated objectives

4. Advantages • focus on outcomes • ease of use, simplicity • forces objectives to be set • widespread acceptability

5. Disadvantages • oversimplification • outcomes-only orientation • may neglect the value of the objectives themselves • may overlook unintended outcomes

6. Case Study: When Hard Questions Are Asked: Evaluating Writing Centers. By James H. Bell • Should writing centers evaluate themselves on a regular basis? • What is the best-type of evaluation approach for a Writing Center? • What will an evaluation achieve?

7. What is the best-type of evaluation approach for a Writing Center? • Evaluation is the “systematic collection and interpretation of evidence, leading, as part of the process, to a judgment of value with a view to action (from Bell, 2000). • 4 important aspects of evaluations for writing centers: • Systematic • Interpretation • Judgment • Action

8. What is the purpose of the evaluation? • Improvement • Audience: program personnel • Formative evaluation • i.e. • Funding • Audience: Supervisors, consumers, funders • Summative evaluation

9. So how should writing centers be evaluated? • Consumer-oriented, • Adversary-oriented • Management-oriented • Naturalistic and participant-oriented • Expertise oriented • Objectives oriented

10. Objectives-Oriented • Writing centers are essentially trying to accomplish things. • Writing centers are aiming to alter behavior • Objectives-oriented evaluation specializes in documenting behavior change. • Speaks to administrators and funders.

11. The Writing Center at the University of Northern British Columbia • Formative-evaluation approach implemented during start-up. • Followed by a small-scale evaluation. • Focuses on one aspect of the program at a time. • An objective-oriented evaluation with a summative emphasis was chosen. • Determine whether the positive evaluations students gave to the center lasted. • Determine whether students had learned something and gained long-term value.

12. Results • 100% of clients were immediately satisfied with the objectives focused on in their conferences and they could immediately apply the newly gained knowledge to their school work. • 83.3% agreed or agreed strongly that they were able to apply the knowledge gained after two weeks. • 86.6% said that what they learned would be of benefit for the future. • ¾ of the clients agreed that they could still apply the knowledge gained through the sessions, even after two months. • The study was replicated, with new staff and tutors, and different student evaluator. • Results were slightly less enthusiastic but still followed the same pattern as the previous evaluation.

13. What did the evaluation achieve? • Confirmed that the Writing Center had a positive impact on students’ writing. • A small-scale, objectives-oriented evaluation method works well for evaluating writing centers. • There is value in evaluations • Identities need • Legitimizes the goals of the writing center • Improves a center’s chances long-term sustainability.

14. Questions or comments? • Thank you.