Meta-evaluation of Information Systems: Ensuring Quality and Value

1. S519: Evaluation of Information Systems Meta-evaluation D-Ch11

2. Meta-evaluation • It is evaluation of an evaluation (Scriven, 1991) to determin the quality and/or value of an evaluation • http://en.wikipedia.org/wiki/Evaluation#cite_note-7

3. Five criteria • Validity • Utility • Conduct • Credibility • Costs

4. Validity • To what extent, the conclusions of the evaluation are justified • Check carefully the facts they use and values they assign • Question everything • Constraints (budget, time, etc.) • Using KEC checkpoints • Using standards (http://www.eval.org/Publications/GuidingPrinciples.asp)

5. Validity • Covers all relevant sources of value • Comprehensively covers process, outcome and cost • Include no irrelevant or illicit criteria • Data used to directly address the criteria • Include analyses that are appropriate for the data • Clearly states how data are interpreted • Is clear about where evaluative conclusions come from • Include valid recommendation

6. Utility • Are the findings • Relevant to the questions or decisions being faced by the audience • Timely • Clearly communicated • Cost effective

7. Conduct • Legal • Ethical • Professional standards • Cultrual appropriateness • Unobtrusiveness • Minimal disruptive to the evaluand

8. Credibility • Familiar with the context • Independence, impartiality, and/or lack of conflict of interest • Expertise in evaluation and in the evaluation subject field

9. Cost • How reasonable the costs of an evaluation

10. Meta-evaluation rating table • Table 11.1

11. Exercise • D-p217, exercise 2 • Form a pair, do the evaluation • Form a group to select the best result, • Design a poster • Present them in a poster • Using three sentenses to summarize your evaluation • What you learn • What you feel • How to improve