S519: Evaluation of Information Systems

1. S519: Evaluation of Information Systems Analyzing data: Rank Ch9-p171

2. Last week • What is synthesis methodology?, why do we need that? • What is synthesis for grading? • What is the rubric for dimensions • What is the rubric for components • How to merge all the conclusions to get the final grade?

3. Synthesizing for „ranking“ • What are „ranking“ evaluations? • Examples? • Difference comparing with „grading“ evaluation?

4. Qualitative and quantitative • Qualitative • Qualitative weight and sum (QWS) • Quantitative • Numerical weight and sum (NWS)

5. Numerical Weight and Sum (NWS) • It is a quantitative synthesis method for summing evaluand performance across multiple criteria. • It includes • Assign numerical importance weight and a numerical performance score to each criteria (dimention) • Mutliply weights by performance scores • Sum these products • The summing result represents the overall merit of the evaluand

6. Numerical Weight and Sum (NWS) • It fits for • There are only a small number of criteria (why) • There is some other mechanism for taking bars into account (why) • There is defensible needs-based strategy for ascribing weights.

7. Training program evaluation • A comparative evaluation on three different interventions for training managers • A mountain retreat featuring interactive sessions with multiple world-class management gurus • An in-house training and mentoring program run by human resources, • A set of videos and latest book on management from managment guru Peter Drucker

8. Training program evaluation • Needs assessment for this evaluation • Bear in mind that this is a comparison evaluation • How do you want to compare these programs, what are the key features of the programs • Identify the dimension of merit (Process, Outcomes and Cost) • Decide the importance of the merit (giving weights to merits, based on needs?) • See Table 9.8

9. Training program evaluation • Next steps • Data collection (what are your experiences for your project data collection?) • Data analysis • Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example) • Convert weights into numbers (see Table 9.10) • Convert ratings into numbers (see Table 9.10) • Synthesis step (how? See Table 9.11) • How to interpret Table 9.11

10. Exercise • Do it by your own hand: • Converting Table9.9 to Table9.10 (defining your own numeric value for importance and grading scales) and try to find out which program is the best comparing with others. • If suddently, the cost criterias become extremely important, will this change the final result? • Work on your own • Form the pair and discussion • Pros and cons for NWS?

11. Qualitative Weight and Sum (QWS) • It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit. • It is a ranking method for determining the relative merit of two or more evaluands • It is not suitable for grading • It fits for • Personnel selection, products/service/proposal selection

12. QWS • Step1: Determine importance in terms of maximum possible value • How (see Chapter 7, six strategies) • Table 9.12 (compare with Table 9.8) • Step2: Set bars • Bar is the cut point between acceptable and unacceptable criteria. Such as: • Too expensive to afford • Too long away from their work

13. QWS • Step3: Create value determination rubrics • Rubrics are level-based (see Chapter 8) • Description on each level, how to deal with bar? • Unacceptableno noticeable valuemarginally valuablevaluableextremely valuable • Such as what performance would look like at each level • Each dimension can have its own rubrics or each group of dimensions can have their own rubrics • Each group of questions can have their own rubrics • Synthesis step can have its own rubrics • Example: Rubric for rating finanical cost of training (see table 9.14) • How to set up similar rubric for Table 9.12 on extremely valuable criteria (see Table 9.15)

14. QWS • Step4: Check equivalence of value levels across dimensions • The validity of the QWS method is highly dependent on ensuring the rough equivalence on the value levels defined for each dimension • For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels • How to do that? Put them into a matrix. • See table 9.16

15. QWS • Step5: rate value of actual performance on each dimension • Rating table 9.9 according to rubric (table9.16) • See Table 9.17 • Step6: tally the number of ratings at each level and look for a clear winner • For each program, how many symbols they got? • Throw out programs with unacceptable ratings, see whether there is a clear winner?

16. QWS • Step7: refocus • Delete the rows with similar score (see table9.18) • Count how many symbols each of them got? • Can we find the clear winner? • Yes or no? • Why? • How should we go further?

17. Exercise • Form a group to work on this • Library wants to subscribe to one of the three magazines and ask you to conduct an evaluation and propose the best solution: • Journal of the American Society for Information Science and Technology • Journal of Information Science • Scientometrics • (or choose some magezines you are familiar with) • List the criteria you think are important • Following the steps of NWS and QWS and tell me your findings and show me the justification of your findings.