Joint Committee on Standards for Educational Evaluation

1. Joint Committee on Standards for Educational Evaluation Evaluation Standards Joan Kruger Spring 2008

2. http:/www.wmich.edu/ evalctr/jc SPONSORING ORGANIZATIONS: American Association of School Administrators American Counseling Association American Evaluation Association American Educational Research Association American Indian Higher Education Consortium American Psychological Association Canadian Evaluation Society Canadian Society for the Study of Education Consortium for Research on Educational Accountability and Teacher Evaluation Council of Chief State School Officer Council of the Great City Schools National Association of Elementary School Principals National Association of School Psychologists National Association of Secondary School Principals National Council on Measurement in Education National Education Association National Legislative Program Evaluation Society National Rural Education Association Joint Committee on Standards for Educational Evaluation

3. What is it? • Created in 1975, the Joint Committee (JC) is a coalition of major professional associations concerned with the quality of education • It is incorporated in the US as a private non-profit organization • It is accredited by the American National Standards Institute (ANSI) and all standards become certified by ANSI

4. What is its mission? • To promote concern for evaluations of high quality based on sound evaluation practices and procedures • To meet existing and emerging needs in the field of evaluation

5. Where is it located and what is the role of CES? • The Joint Committee is housed at the Evaluation Center, Western Michigan University • Canadian Evaluation Society has been a Sponsoring Organization since 1994 and participates as a voting member, participates in special projects, and promotes all Standards (Program, Personnel and Student) to CES members.

6. Direct Products of the JC Publications: 3 sets of evaluation standards • Personnel Evaluation Standards. 2nd ed. 2008. Corwin Press • Program Evaluation Standards. 2nd ed. 1994. 3rd in testing. Sage • Student Evaluation Standards. 2003. Corwin Press.

7. Associated Accomplishments • Adoption and adaptation by schools, states, and organizations nationally and internationally. • Numerous associated materials (e.g., Checklists for metaevaluation). http://www.wmich.edu/evalctr/checklists/ • Widespread translation and/or dissemination internationally, for use in Europe (especially Germany and Switzerland, Latin America, Africa and others)

8. Pervasive Nature of Program Evaluations • Primary tool for guiding program evaluation, crossing all academic disciplines and courses • Evaluations regularly clients and stakeholders in a wide variety of decision situations.

9. Impact of Program Evaluations • Evaluations done well can be of significant service. • Clients are victims of and harmed by poor evaluations in both high- and low-stake evaluations

10. Barriers to Sound Program Evaluations • Inadequate preparation • Inadequate administrative and technical support • Professional disagreements • Lack of shared language • Inadequate policy

11. What is available? Assessment & measurement design Performance management systems Standards for student and personnel evaluation What are the gaps? No standards for the programs evaluations across disciplines. No desktop reference to ensure quality Why do we need Program Evaluation Standards?

12. What is a Standard? • A principle commonly agreed upon by experts in the conduct and use of evaluation for the measurement of the value or quality of an evaluation. • A standard is seen as a principle governing good practice vs. a rule • Technical guide to conduct evaluations • Criteria of good practice for those who receive and are affected by an evaluation

13. What is a Standard? • A guideline may provide the evaluator with a recommendation to help ensure basic quality evaluation. Other evaluators, in the same situation in the same or similar institutions following the guidelines, may also ensure basic quality. • The CES felt standards used as guidelines would help evaluators make more consistent quality judgements over time.

14. What is a Standard is not. Rules are much more restrictive. The evaluator must or should conform to rules. This may lead to a narrower rating of a program or project by not allowing the evaluator to adapt the standard to the norms of the evaluation setting. Rules are usually policed by a professional body, and offenders may be reprimanded.

15. How are the Standards Organized? All three sets of Standards are organized into four main categories: • Utility • Propriety • Feasibility • Accuracy

16. Utility To ensure that an evaluation will serve the information needs of intended users.

17. Propriety To ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation as well as those affected by its results. Additional resource: Canadian Evaluation Society Ethical Guidelines http://www.evaluationcanada.ca/site.cgi?s=5&ss=4&_lang=EN

18. Feasibility To ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.

19. Accuracy To ensure that an evaluation will reveal and convey technically adequate information bout the features that determine worth and merit of the program being evaluated.

20. How are the Standards Presented in the Book? A uniform pattern is followed: • Standard Statement • Overview (Explanation, Rationale, Caveats) • Guidelines • Common Errors • Two illustrative case studies • Supporting documentation (references)

21. Number of Standards(Program: proposed 3rd ed)

22. National Public Hearings National Public Hearings: The Joint Committee will hold open forums at meetings of each of the sponsoring organizations to gain additional feedback from members of these organization (See 5.3.5 of the Operating Procedures-). 5.3.5 National Public Hearings National Public Hearings shall be held to provide an open forum for discussion and critique of the standards. Hearing schedules shall be publicized and all interested parties will be encouraged to participate. In addition, each Sponsoring Organization will be asked to select members of their organization and encourage them to respond to the draft through the vehicle of the hearings. (NOTE: If the standards under consideration are to be submitted to the American National Standards Institute for approval as American National Standards, the Joint Committee shall also arrange for announcement of the draft in ANSI's STANDARDS ACTION for comment, in accordance with Section 1.2.6 of the ANSI Procedures for the Development and Coordination of American National Standards.)

23. Process for Hearing • Overview of the Standards • Comments from CES members and others (criticisms, observations, recommendations) • Comments from Field test participants

24. Further Communication • Visit the JC web site at http://www.wmich.edu/evalctr/jc/ and enter ES review page. • Contact Arlen Gullickson arlen.gullickson@wmich.edu

25. Next Steps • Complete field tests • Complete National Hearings • Revise • Final Approval by the Joint Committee • Submission to American National Standards Institute for certification • Publish

26. Evaluation Web Sites Canadian Evaluation Society (CES) http://www.evaluationcanada.ca/ Canadian Evaluation Society Educational Fund (CESEF) – for students and those new to eval. http://www.evaluation-education.org/index.html Saskatchewan Chapter, CES http://sk.evaluationcanada.ca/ American Evaluation Association (AEA) http://www.eval.org/