S519: Evaluation of Information Systems

1. S519: Evaluation of Information Systems Analyzing data: Causation Ch5

2. Step5: Analyzing data • Dealing with the causation issue, basically be able to answers following questions: • How certain does the client need us to say that the evaluand „caused“ a certain change? • What are the basic principles for inferring causation? • What types of evidence do we have available to help us identify or rule out possible causal links? • How should we decide what blend of evidence will generate the level of certainty needed most cost-effectively?

3. Certainty about causation (D-ch5) • Each decision-making context requires a different level of certainty • Quantitative or qualitative analysis • All-quantitative or all-qualitative • Sample choosing • Sample size • Mix of them

4. Inferring causation: basic principles • Two basic principles: • Look for evidence for and against the suspected main cause (i.e., evaluand) • Look for evidence for and against any important alternative causes (i.e., rival explanations) • Too many evidences or causes, which are the primary causes • All based on the level of certainty you need for your evaluation • Stepwise process • Put yourself in the hardest critics, gather enough evidence to support your explanation • Repeat it until all remaining alternative explanations are ruled out. • Critical multiplism: triangle The harder people attack, the more solid your answers need to be.

5. Inferring causation: 8 strategies • 1: ask observers • Two possibilities • Ask actual or potential impactees • Ask indirect impactees (i.e., co-worker, parents...) • Design your interview questions to include causation questions • E.g., how much has your knowledge increased as a result of attending this program? – get primary cause • E.g., did anything else besides the program increase your knowledge in this area over the same period of time? – get other causes • E.g., please describe anything else that has happened to you or someone you know as a result of participating in this program? – get the causes which people know or believe were caused by the program.

6. Inferring causation: 8 strategies • 1: ask observers • Causation-rich questions tend to be leading (direct the respondent to answer in a particular way). Be careful about the wording when designing interview questions • The causation question is not just whether the program produced the effect but also what other factors enabled or inhibited the effect. • Individual might not be a reliable witness to answer the causation question, other evidence will be required to make justifiable causal inferences.

7. Inferring causation: 8 strategies • 1: ask observers • Methods • Questionnaires to identify the targeted groups (people who experienced substantial changes) • Using open-end to get more opinions • In-depth interview with the targeted groups • To find out causation.

8. Inferring causation: 8 strategies • 2. Check whether the content of the evaluand matches the outcome • Alcoholics treatment program  alcoholics avoid relapses • Check whether the strategies which alcoholics use to avoid relapses after the program, are the same as the strategies taught in the program

9. Inferring causation: 8 strategies • 3. Look for other telltale patterns that suggest one cause or another • Modus operandi method – look for evidence -- detective metaphor to describe the way in which potential causal explanations are identified and tested. • A silly example • Evidences: a naked man, dead; in the middle of the desert; personal belongings near by; half match at hand • Cause of his death

10. Inferring causation: 8 strategies • 4. Check whether the timing of outcomes makes sense • Common sense: • an outcome should appear only at the same time as or after whatever caused it – a considerable delay. • A further downstream the outcomes, the longer they should take to appear • Using timing to confirm or disconfirm causal links: • Is the outcome happened before the evaluation? Or Other downstream outcomes too early? • Is the timing of the outcomes logical to possible causes? • Do the further downstream outcomes in the logic model occur out of sequence? More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.

11. Inferring causation: 8 strategies • 4. Check whether the timing of outcomes makes sense • Example – a community health education on diet and exercise • Fairly immediate knowledge and skill gain: during or immediately after the intervention • A short delay (days or weeks) before the knowledge and skills are transformed into changing behavior • A moderate delay (weeks or months) before we see changes in individual health indicators (weight, cholesterol, blood pressure, etc.) • A long delay (months or years) to see changes on improvement on diabetes and heart diseases

12. Exercise Lab • Take grantsmanship workshop as one example • List the timeline potential outcomes (fairly immediate, a short delay, a moderate delay, a long delay) • Using timing strategies to confirm or disconfirm the cause links, state one page for how and why: • One month after the workshop, 3 proposals got grants • One year after the workshop, 3 proposals got grants • Three months after the workshop, some people write good proposals, but some are not. • Think about your own solution • Form a group and discuss

13. Inferring causation: 8 strategies • 5. Check whether the „dose“ is related logically to the „response“. • The dose-response idea • The more dose of drug, the better response later on • If more A (dose), then better B (response) • Compare the less dose with more dose (not overdose) to confirm or disconfirm the cause links • E.g., for performance evaluation project, if we found that performance had been improved dramatically in the unit where the system has been poorly implemented,  this system is not the cause of the performance improvement.

14. Inferring causation: 8 strategies • 6. Make comparisons with a „control“ or „comparison“ group • Divide the participants into different groups • control group (receive the evaluand) vs. Comparison group (receive no evaluand) • Sampling should be done carefully to make sure no systematic differences between groups • Sample size • randomization

15. Inferring causation: 8 strategies • 7. Control statistically for extraneous variables • When using control and comparison groups, try to exclude external variables and make two groups no systematic differences: • Statistical methods • Regression analysis • Try to identify other potential systematic differences • E.g., math improvement for students, how to sample students and think about other potential existing difference. Is the random sampling enough?

16. Inferring causation: 8 strategies • 8. Identify and check the underlying causal mechanism(s) • Try to look for an underlying mechanism to make the case for causation more or less convincing • Cigarette smoking  lung cancer • Correlation studies • Carcinogenic in cigarette causes cancer • Normally coming from literature.

17. Put them together • Do we need all the evidences we collect from 8 strategies? • How to select them? • Put yourself in the shoes of a tough critic, identify the most potential threatening rival explanation, then chose the types of evidence that will most quickly and cost-effectively confirm or dispel that rival explanation. • Go to next less tough rival explanation, ... • Continue, until you have amassed a body of evidence to provide you enough certainty to draw causal inferences

18. Exercise Lab • Grantsmanship workshop (p57) • Grantsmanship workshop strengthen local communities • For (evidences) • Against (evidences) • Other alternative causes (i.e., rival explanations) • Using strategies to confirm or disconfirm these evidences or causes • Putting them together • Form a group to discuss