1 / 29

S519: Evaluation of Information Systems

S519: Evaluation of Information Systems. Analyzing data: Synthesis D-Ch9. Synthesis methodology. It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand.

murielm
Download Presentation

S519: Evaluation of Information Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. S519: Evaluation of Information Systems Analyzing data: Synthesis D-Ch9

  2. Synthesis methodology • It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand. • Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.

  3. Synthesis methodology • Merit determination • To develop the rubrics • To use rubrics to summarize the multiple findings • Rubrics are one of the simplest methods to blend data. • But when data is a bit more complex, it is difficult to use a rubric as the only tool • Data are not equally important or reliable • Multi dimensions or multi components • Different nuances and combinations (such as Table8.3)

  4. It is not • It is not meta-analysis • A special statistical technique to give a weighted average of effect sizes across multiple studies – for quantitative studies • It is not literature review or a summary • A judgment from a reviewer’s point of view.

  5. Keep in mind • Doing poorly on some minimal important criteria • Doing poorly on some crucial criteria • Are very different!

  6. Evaluation • Synthesis for „ranking“ • If it is „ranking“ (relative) evaluation: • Consider each alternative and make explicit comparisons • Synthesis for „grading“ • If it is „grading“ (absolute) evaluation: • Consider different context settings and provide better interpretation of merit

  7. Qualitative or quantitative • Quantitative synthesis • Using numerical weights • Qualitative synthesis • Using qualitative labels

  8. Synthesis for “grading” • The primary evaluation question is for absolute quality or value • How well did the evaluand perform on this dimension or component? • How effective, valuable, or meritorious is the evaluand overall? • Is this worth the resources put into it?

  9. Quantitative weighting example with „bars“ • Case: Personnel evaluation in a small accounting firm • 13 defined tasks (e.g., telephone, reception, data entry, etc.) • Each employee has responsibility for 4-6 tasks • Evaluation: • Importance weighting (through the voting of the selected stakeholders) • In-depth discussion with business owners • Derive the importance metric and bars

  10. Quantitative weighting example with „bars“ • Evaluation • Define the levels of importance: • 3 to 5 levels work well in most case • Do not go to too many levels (why? Is this useful?) • For example • task • 1. minor task (1) • 2. normal-priority task (2) • 3. high-priority task (3) • 4. extremely high-priority task (4)

  11. Quantitative weighting example with „bars“ • Evaluation • Setting up rubrics for each 13 tasks • Normally 4-6 level is sufficient • Example: Performance Rubric • 1. Totally unacceptable performance (1) • 2. Mediocre (substandard) performance (2) • 3. Good performance (expected level) (3) • 4. Performance that exceeded expectations (4) • 5. All-around excellent performance (5) • Synthesis – draw the overall conclusion • See Exhibit 9.2 (p158)

  12. Exercise Lab • Personal evaluation in a small accouting firm How about Alice according to Exhibit 9.2?

  13. Exercise Lab • Personal evaluation in a small accouting firm How about John according to Exhibit 9.2?

  14. Exercise Lab • Perosnal evaluation in a small accouting firm How about Chris according to Exhibit 9.2?

  15. Exerice • How about Chris • Mean= 1*2+2*3+4*4+4*5+3*3+1*3/(1+2+4+4+3+1)=56/15=3.73 • What is Chris‘ performance?

  16. Qualitative weighting example 1 (with no „bars“) • Case: a school-based health program evaluation • It contains 9 different components: nutrition education, mental health services, safer sex, legal service and others. • How to evaluate these systems in low-budget and short period of time whether they are meeting important needs of the students and their families • Evaluation: • Interview • Student surveys

  17. School health system evaluation • Survey question design: • Two quantitative questions • How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful) • How satisfied were you with the program? • One qualitative question (open-end)? • What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?

  18. School health system evaluation • Survey result about nutrition system shows in Table 9.1 • Look at table 9.1, think about: • How can you draw a conclusion from this result about the nutrition system? Is it good or bad?

  19. School health system evaluation • Setting the importance for these three questions (1-strongest data, 3=weakest data) • 1. Ratings of usefulness (directly related to needs) • 2. Responses to the open-ended question • 3. Satisfaction ratings • Creating rubrics for each question • Table 9.2 for question 1 and question 2 • Table 9.3 for open-ended question

  20. School health system evaluation • How to grade the nutrition system based on the first two quantitative questions: • Based on Table 9.1, come out with the rubric as Table 9.2 • Why 90% is select, 70%-90%.. • How to draw Table 9.2 from Table 9.1 and collected data?

  21. School health system evaluation • Table 9.3 • Rubric for converting data from qualitative evaluation - open-ended responses into merit ratings • Is that a good way to do this? • Are you happy with this table? • If not, how do you want to improve it?

  22. School health system evaluation • Synthesis to draw overall conclusion • Step-by-step • Start with the strongest data (question 1) • Blend with open-ended comments • Finally take the satisfaction ratings into account • See table 9.4 for the whole process

  23. School health system evaluation • How to draw final conclusion? Usefulness ratings Final coclusion: Merit of the nutrition program Satisfaction ratings Open-ended comments Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results See table 9.4 Discuss how to apply this to your group project

  24. Qualitative (nonnumerical) weighting example 2 • Bar • A minimum level of performance on a specific dimension • Performance below this cannot be compensated for by much better performance on other dimensions (see Exhibit 9.2) • Hard hurdle (also referred as global bars) • Overall passing requirement for an evaluand as a whole (see Exhibit 9.2) • Soft hurdle • Overall requirement for entry into a high rating category • Place a limit on the maximum rating (e.g., I want all As for my classes)

  25. Qualitative (nonnumerical) weighting example 2 • Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“. • Evaluation • 27 subdimensions of organizational learning capacity (see table 9.5) • Data collection: survey and interview • Rubric: similar as Table 8.2 • Importance is built by using strategy 6 in Chapter 7 • Using program theory and evidence of causal linkages (p118-125)

  26. Biosleep • Evaluation • Synthesis • Pack the ratings on the subdimensions into 8 main dimensions • Combine the ratings on these 8 main dimensions to draw an overall conclusion

  27. Biosleep • Dimension by dimension • Layer by layer Sub-dimnention1 Dimnention1 Sub-dimnention2 Overall rating Sub-dimnention3 Dimnention2 Sub-dimnention4

  28. Biosleep • Synthesis • Subdimensions  Dimensions • Using Table 9.6 to draw conclusions of dimentions based on subdimensions • Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4 • Dimensions  overall evaluation • Based on Table 9.7 (created based on literature review, • What is your conclusion for the evaluation of Biosleep? And why?

  29. Exericse Lab • Form your group project • Discuss on how are you going to grade your evaluation? • Which example you would like to follow? • How to develop rubric for dimension and overall?

More Related