S519: Evaluation of Information Systems

1. S519: Evaluation of Information Systems Analyzing data: Synthesis D-Ch9

2. Synthesis methodology • It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand. • Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.

3. Synthesis methodology • Merit determination • To develop the rubrics • To use rubrics to summarize the multiple findings • Rubrics are one of the simplest methods to blend data. • But when data is a bit more complex, it is difficult to use a rubric as the only tool • Data are not equally important or reliable • Multi dimensions or multi components • Different nuances and combinations (such as Table8.3)

4. It is not • It is not meta-analysis • A special statistical technique to give a weighted average of effect sizes across multiple studies – for quantitative studies • It is not literature review or a summary • A judgment from a reviewer’s point of view.

5. Keep in mind • Doing poorly on some minimal important criteria • Doing poorly on some crucial criteria • Are very different!

6. Evaluation • Synthesis for „ranking“ • If it is „ranking“ (relative) evaluation: • Consider each alternative and make explicit comparisons • Synthesis for „grading“ • If it is „grading“ (absolute) evaluation: • Consider different context settings and provide better interpretation of merit

7. Qualitative or quantitative • Quantitative synthesis • Using numerical weights • Qualitative synthesis • Using qualitative labels

8. Synthesis for “grading” • The primary evaluation question is for absolute quality or value • How well did the evaluand perform on this dimension or component? • How effective, valuable, or meritorious is the evaluand overall? • Is this worth the resources put into it?

9. Quantitative weighting example with „bars“ • Case: Personnel evaluation in a small accounting firm • 13 defined tasks (e.g., telephone, reception, data entry, etc.) • Each employee has responsibility for 4-6 tasks • Evaluation: • Importance weighting (through the voting of the selected stakeholders) • In-depth discussion with business owners • Derive the importance metric and bars

10. Quantitative weighting example with „bars“ • Evaluation • Define the levels of importance: • 3 to 5 levels work well in most case • Do not go to too many levels (why? Is this useful?) • For example • task • 1. minor task (1) • 2. normal-priority task (2) • 3. high-priority task (3) • 4. extremely high-priority task (4)

11. Quantitative weighting example with „bars“ • Evaluation • Setting up rubrics for each 13 tasks • Normally 4-6 level is sufficient • Example: Performance Rubric • 1. Totally unacceptable performance (1) • 2. Mediocre (substandard) performance (2) • 3. Good performance (expected level) (3) • 4. Performance that exceeded expectations (4) • 5. All-around excellent performance (5) • Synthesis – draw the overall conclusion • See Exhibit 9.2 (p158)

12. Exercise Lab • Personal evaluation in a small accouting firm How about Alice according to Exhibit 9.2?

13. Exercise Lab • Personal evaluation in a small accouting firm How about John according to Exhibit 9.2?

14. Exercise Lab • Perosnal evaluation in a small accouting firm How about Chris according to Exhibit 9.2?

15. Exerice • How about Chris • Mean= 1*2+2*3+4*4+4*5+3*3+1*3/(1+2+4+4+3+1)=56/15=3.73 • What is Chris‘ performance?

16. Qualitative weighting example 1 (with no „bars“) • Case: a school-based health program evaluation • It contains 9 different components: nutrition education, mental health services, safer sex, legal service and others. • How to evaluate these systems in low-budget and short period of time whether they are meeting important needs of the students and their families • Evaluation: • Interview • Student surveys

17. School health system evaluation • Survey question design: • Two quantitative questions • How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful) • How satisfied were you with the program? • One qualitative question (open-end)? • What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?

18. School health system evaluation • Survey result about nutrition system shows in Table 9.1 • Look at table 9.1, think about: • How can you draw a conclusion from this result about the nutrition system? Is it good or bad?

19. School health system evaluation • Setting the importance for these three questions (1-strongest data, 3=weakest data) • 1. Ratings of usefulness (directly related to needs) • 2. Responses to the open-ended question • 3. Satisfaction ratings • Creating rubrics for each question • Table 9.2 for question 1 and question 2 • Table 9.3 for open-ended question

20. School health system evaluation • How to grade the nutrition system based on the first two quantitative questions: • Based on Table 9.1, come out with the rubric as Table 9.2 • Why 90% is select, 70%-90%.. • How to draw Table 9.2 from Table 9.1 and collected data?

21. School health system evaluation • Table 9.3 • Rubric for converting data from qualitative evaluation - open-ended responses into merit ratings • Is that a good way to do this? • Are you happy with this table? • If not, how do you want to improve it?

22. School health system evaluation • Synthesis to draw overall conclusion • Step-by-step • Start with the strongest data (question 1) • Blend with open-ended comments • Finally take the satisfaction ratings into account • See table 9.4 for the whole process

23. School health system evaluation • How to draw final conclusion? Usefulness ratings Final coclusion: Merit of the nutrition program Satisfaction ratings Open-ended comments Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results See table 9.4 Discuss how to apply this to your group project

24. Qualitative (nonnumerical) weighting example 2 • Bar • A minimum level of performance on a specific dimension • Performance below this cannot be compensated for by much better performance on other dimensions (see Exhibit 9.2) • Hard hurdle (also referred as global bars) • Overall passing requirement for an evaluand as a whole (see Exhibit 9.2) • Soft hurdle • Overall requirement for entry into a high rating category • Place a limit on the maximum rating (e.g., I want all As for my classes)

25. Qualitative (nonnumerical) weighting example 2 • Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“. • Evaluation • 27 subdimensions of organizational learning capacity (see table 9.5) • Data collection: survey and interview • Rubric: similar as Table 8.2 • Importance is built by using strategy 6 in Chapter 7 • Using program theory and evidence of causal linkages (p118-125)

26. Biosleep • Evaluation • Synthesis • Pack the ratings on the subdimensions into 8 main dimensions • Combine the ratings on these 8 main dimensions to draw an overall conclusion

27. Biosleep • Dimension by dimension • Layer by layer Sub-dimnention1 Dimnention1 Sub-dimnention2 Overall rating Sub-dimnention3 Dimnention2 Sub-dimnention4

28. Biosleep • Synthesis • Subdimensions  Dimensions • Using Table 9.6 to draw conclusions of dimentions based on subdimensions • Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4 • Dimensions  overall evaluation • Based on Table 9.7 (created based on literature review, • What is your conclusion for the evaluation of Biosleep? And why?

29. Exericse Lab • Form your group project • Discuss on how are you going to grade your evaluation? • Which example you would like to follow? • How to develop rubric for dimension and overall?