The CIPP Approach to Evaluation

1. The CIPP Approach to Evaluation Owen Perkins

2. Management-Oriented Evaluation Approach • Meant to serve decision makers at all levels • Good decision making is dependent on evaluative information • A systems approach to evaluation • Decisions are made about inputs, processes, and outputs • Approach was a shift from program objectives to the decision(s) of program managers • Evaluation of the decision(s) the administrator(s) must make

3. The CIPP Evaluation Model • Daniel Stufflebeam, supported a decision-oriented evaluation approach to assist administrators make good decisions • His approach was developed in the 1960’s • His Evaluation method proposed that administrators face four kinds of decisions • Context evaluation • Input Evaluation • Process evaluation • Product evaluation • Emphasis is on decision making by program administrators

4. Context Evaluation • Determine needs to define objectives of the program • Asks “what should we do?” • Collect data to determine goals of program • Example: A standardized test is given to every student in 6th grade across a state. What are the objectives of this program? Raise level of education, standard to evaluate teacher performance, etc.

5. Input Evaluation • Determine resources, alternative strategies, and determine plan that has best potential to meet the needs of the program • Asks “how should we do it?” • Involves determining the steps and resources needed to accomplish goal or objective

6. Process Evaluation • To evaluate how well the plan has been implemented • Asks “are we doing it as planned?” • Continuously monitor the program, allows decision maker to see conflicts, strengths and weaknesses of program and materials, how well it is following the established plan, etc.

7. Product Evaluation • Determine program attainments • “What results were obtained?” • “How well were needs reduced?” • “How did the program work?” • Compare actual outcomes to anticipated outcomes • Should the program being evaluated be continued, modified or dropped?

8. Stufflebeam suggested evaluators follow these general steps: • Focus the evaluation • Collection of Information • Organization of Information • Analysis of Information • Reporting of Information • Administration of the Evaluation

9. Focusing the Evaluation • Identify the level of decision making being addressed • For each level of decision making analyze potential decisions • Define criteria for each decision situation by specifying variables for measurement and standards for use in the judgment of alternatives • Define policies within which the evaluator must operate

10. Collecting Information • Specify the source of information to be collected • Specify the instruments and methods for collecting the needed information • Specify the sampling procedure to be used • Specify the conditions and schedule for information to be collected

11. Organization of Information • Provide a format for the information that is to be collected • Designate a means for performing the analysis

12. Analysis of Information • Select the analytical procedures to be used • Designate a means for performing the analysis

13. Reporting of Information • Define the audiences for the evaluation reports • Specify a way to present information to audiences • Develop format for evaluation reports • Schedule the reporting of information

14. Administration of the Evaluation • Summarize the evaluation schedule • Evaluate the potential of the evaluation design for providing information that is valid, reliable, credible, timely, and pervasive • Specify and schedule means for periodic updating of the evaluation design • Provide a budget for the total evaluation program

15. Strengths and Weaknesses of CIPP • Holistic approach: • Broad, vague, and summative • Good or bad? • Idealized notion of process and program: • What the process should be rather than its actual form • Too top-down and managerial in approach

16. Imprisonment and Career Development: An Evaluation of a Guidance Program for Job Finding By: Gemma Filella-Guiu and Angel Blanch-Plana

17. Article Abstract • This study describes the evaluation results of PORO; a three level occupational guidance program • Designed to improve offender’s job finding possibilities and to reduce reoffending after imprisonment • Used Stufflebeam’s CIPP model, considering the efficiency, effectiveness and impact of the program • Evaluation outcomes and practical suggestions for career development applications addressed to prison populations are reported

18. PORO Definition • The Guidance Program for Job Search (PORO) • Go figure? • Wikipedia: Poro is a fraternal secret society of Sierra Leone and Liberia

19. Program Description • All internees (prisoners) are categorized by their individual judicial situation • PORO comprises three levels of guidance to internees: • Internees given professional and occupational training in prison • Internees given ordinary parole permits • Has two distinct sections: • Internees go into community on daily basis • Probation and freed internees in community

20. CIPP’s Use in Study • Context Evaluation is the delineation and specification of the program’s environment, the population and sample of individuals to be used, and the program objectives • Input Evaluation identifies and assesses system capabilities, determines methods that meet objectives and needs, and studies the relations between cost and potential benefits of each proposed method • Process Evaluation helps to provide feedback to the teachers on progress of the program • Product Evaluation allows decisions to be taken regarding the continuity, termination or modification of the program

21. Context Evaluation • “What should we do?” • Collect data to determine goals of program

22. Context Evaluation: Collect Data • 145 people in the experimental group • 51 in the control group • (N = 196) • 11% females and 89% males • Between 25 and 35 year old • Serving sentences of less than 4 years. • Specialties taken in the occupational courses were car mechanics, graphic arts, industrial dressmaking, building, electricity, binding, carpentry and computing

23. Context Evaluation: Determine Goals • The main objectives of this intervention were: a) To improve individual competence in job searching b) To facilitate the acquisition of useful job search knowledge and skills c) To increase professional self-esteem “The accomplishment of these specific aims is as fundamental for enabling ex-convicts to adjust to the job market and the community on a long-term basis”

24. Input Evaluation • “How should we do it?” • Determine resources, alternative strategies, and determine plan that has best potential to meet the needs of the program

25. Input Evaluation: Determine plan that has best potential to meet the needs of the program • “The contribution of an expert group was requested to assess the design and methodological validity of PORO in order to achieve the best possible adaptation to a confinement context” • “The potential benefits that might be obtained were rewarding enough considering the assignment of operational, material and human resources” • = Cheap Program

26. Process Evaluation • “Are we doing it as planned?” • To evaluate how well the plan has been implemented

27. Process Evaluation: How well has the plan been implemented? • “Feedback from all course teachers and 52 randomly selected participants on the program were used to evaluate the process dimension” • “No serious difficulties arose during the teaching program on the majority of occupational courses” • “Internees were not very enthusiastic about the program in its initial stages. However, in the end, this lack of motivation was gradually overcome during the program development, final opinions being highly positive”

28. Product Evaluation • “What results were obtained?” • “How well were needs reduced?” • “How did the program work?” • Determine program attainments

29. Product Evaluation: Determine Program Attainments • Increase in job knowledge and skills scores • Increase in professional self-esteem scores • Based on the findings “it should be expected that other internees would reach these same employment rates when their sentences have been served • The program had a high level of efficiency: • “The material, human and operational resources used in the [program] development did not pose any additional expense” • The program did not have any adverse effects: • “Undesirable contingencies were not noted once the investigation finished, nor were unforeseen relevant consequences due to the occupational guidance [program] detected”

30. Conclusion The “results…give further support to the idea that career intervention addressed at imprisoned people in adverse social and economic circumstances…is of paramount importance for improving the employment possibilities of people whose [behavior] and lack of skills are troublesome to them and to society”

31. Conclusion, cont’d • This case study is an excellent example of the CIPP evaluation approach • Each step of the approach was carefully considered

32. The End GO HOKIES!