Critical Appraisal

1. Critical Appraisal Dr. Chris Hall – Facilitator Dr. Dave Dyck R3 March 20/2003

2. Objectives: • Review study design and the advantages/ disadvantages of each • Review key concepts in hypothesis, measurement, and analysis • Article appraisal • Treatment articles • Diagnosis articles • Harm articles • Overviews/meta-analysis • Survive the next hour and still be able to smile

3. Study Design: • Ecological studies • Case Reports • Case Series • Cross-Sectional Studies • Case Control and Retrospective Cohort Studies • Prospective Cohort Studies • Randomized Controlled Trials

4. Ecological Studies: • Studies of a group rather than individual subjects • Supplies data on exposure and disease as a summary measure of the total population as an aggregate eg. Incidence studies • Berkson’s Bias: ie. The correlation between the variables is not the same on the individual level as it is for the group. Therefore you cannot link exposures to disease on an individual basis • Also, difficult to account for confounding variables

5. Case Reports • Submission of individual cases with rare or interesting findings • ++++ subject to bias (selection / submission and publication) • Should not infer causality or suggest practice change

6. Case Series: • A group of “consecutive cases” with unifying features • Selection bias = what constitutes a case, is it truly consecutive, response bias • Publication bias • Measurement bias (presence of ‘disease’ or exposure may be variable)

7. Cross Sectional Studies: • Ie. Prevalence study • Presence or absence of a specific disease compared with one or several variables within a defined population at a specific point in time

9. Cross Sectional Studies disadvantages: • Subject to selection bias (see HO) • Cause and effect cannot be determined (see HO) (ie. Don’t know whether the exposure occurs before the outcome or the outcome occurs before the exposure) • Temporal trends may be missed (seasonal variations) • Previous deaths, drop-outs, and migration are not counted; and short lived, transient outcomes are underrepresented. Thus, CSS are best suited to study chronic, non-fatal conditions.

10. Cross sectional studies – advantages: • Can do quickly • May provide enough of an association between an exposure/outcome to generate a hypothesis which can be studied by another method. • Useful for descriptive/analytical studies

11. Case Control Studies: • Starts now and goes back in time • Start with the outcome and ask or find out about prior exposure • Specific hypothesis usually tested • Select all cases of a specific disease during a certain time and select a number of controls who represent general population  then determine exposure to factor in each  odds ratio • May match controls to patients (but can never be sure of similar baseline states)

12. Case Control Study:

13. CCS cont • Odds ratio provides an estimate of the relative risk (esp when disease is rare) • Thus, use CCS only when disease is rare (< 10% of population) • As OR increases (>1)  greater risk • As OR decreases (<1)  reduced risk

14. CCS advantages: • Small # needed (good for rare diseases or when outcomes are rare or delayed) • Quick • Inexpensive • Can study many factors

15. CCS disadvantages: • Problems selecting/matching controls • Only an estimate of relative risk • No incidence rates • Biases (? Unequal ascertainment of exposure between cases and controls) • Ie recall bias= cases are more likely to remember exposure than controls • Selection bias = cases and controls should be selected according to predetermined, strict, objective criteria

16. Cohort Study (prospective) • Start with 2 groups free of disease and follow forward for a period of time • 1 group has the factor (eg. Smoking) the other group does not • Define 1 or more outcomes (eg. Lung CA) • Tabulate the # of persons who develop the outcome • Provides estimates of incidence, relative risk, and attributable risk

18. Relative risk / Attributable risk • Relative risk = measures the strength of association between exposure and disease • Attributable risk = measures the number of cases of disease that can be attributed to exposure • Given a constant relative risk, attributable risk rises with incidence of the disease in members of the population who are not exposed

20. Cohort Study • Cannot by itself establish causation, but can show an association between a factor and an outcome • Generally provides stronger evidence for causation than case control studies

21. Cohort Study advantages: • Lack of bias in factor • Uncovers natural history • Can study many diseases • Yields incidence rates, relative, and attributable risk • Allows for more control of confounding variables

22. Cohort Study Disadvantages: • Possible bias in ascertainment of disease. • Need large numbers and long follow-up • Easy to lose patients in follow-up (attrition of subjects). This may introduce bias if lost subjects are different from those who continue to be followed • Hard to maintain comparable follow-up for all levels of exposure

23. Cohort Study disadvantages cont. • Expensive • Locked into the factor(s) measured • Measurement bias (eg. Unblinded physician who looks harder for + outcomes in the exposed pt) • Confounding variables still present

24. Randomized Control Trials: • To test the hypothesis that an intervention (treatment or manipulation) makes a difference. • An experimental group is manipulated while a control group receives a placebo or standard procedure • All other conditions are kept the same between the groups

26. RCTs • Goals= • Prevention (to decrease risk of disease or death) • Therapeutic (decrease symptoms, prevent recurrences, decrease mortality) • Diagnostic (evaluate new diagnostic procedures)

27. RCT problems: • Ethical issues • Difficulty to test an intervention that is already widely used • Randomization • Blinding techniques (may be difficult due to common SE of drugs) • Control group (placebo, conventional tx, specific tx) • Subject selection and issues of generalizability • Are refusers different in some way

28. Causation:

29. Key Terms for diagnostic tests: • Sensitivity= proportion with the disease identified by the test • Specificity= proportion without the disease with a negative test

30. Sensitivity= a/a+c Specificity=d/b+d

31. Other key terms: • Positive Predictive Value= This is the probability of having the disease given a positive test (a/a+b) • Negative Predictive Value= The probability of not having the disease given a negative test (d/c+d)

32. Statistical Hypothesis: • Null Hypothesis • Hypothesis of no difference between a test group and a control group (ie. There is no association between the disease and the risk factor in the population) • Alternative Hypothesis • Hypothesis that there is some difference between a test group and control group

34. Measurements and Analysis: • Sampling bias = selecting a sample that does not truly represent the population • Sampling size = contributes to the credibility of “positive” studies and the power of “negative studies”. Increasing the sample size decreases the probability of making type I and type II errors.

35. Errors • Type I Error (alpha error) = the probability that a null hypothesis is considered false when it is actually true. (ie. Declaring an effect to be present when it is not) This probability is represented by the p value or alpha; the probability the difference is due to chance alone.

36. Errors cont. • Type II Error (Beta Error) = the probability of accepting a null hypothesis as true when it is actually false (ie. Declaring a difference/effect to be absent when it is present) • The probability that a difference truly exists • Reflects the power (1-Beta) of a study

38. Significance: • Statistical Significance: determination by a statistical test that there is evidence against the null hypothesis. • The level of significance depends on the values chosen for alpha error • Usually alpha<.05 and beta<.20 (studies rarely aim for power >80%)

39. Significance cont. • Clinical Significance: statistical significance is necessary but not sufficient for clinical significance which reflects the meaningfulness of the difference (eg. A statistically significant 1mm Hg BP reduction is not clinically significant) • Also includes such factors as cost, SE.

40. Other terms: • Accuracy= how closely a measurement approaches the true value • Reliability= how consistent or reproducible a measurement is when performed by different observers under the same conditions or the same observer under different conditions • Validity= describes the accuracy and reliability of a test (ie. The extent to which a measurement approaches what it is designed to measure)

41. Validity and Reliability

42. Appraising an article (JAMA): • 3 basic stages • 1) the validity – are the conclusions justified? • 2) the message – what are the results? • 3) the utility – can I generalise the findings to my patients?

43. Are the results valid? – (therapy article) • Primary guides • Was the assignment of patients to treatment randomized? • Were all patients who entered the trial properly accounted for and attributed at its conclusion? • Was follow-up complete? • Were patients analyzed in the groups to which they were randomized? Ie. Intention to treat analysis

44. Are the results valid? • Secondary guides: • Were patients, their clinicians, and study personnel “blind” to treatment? (avoids bias) • Were the groups similar at the start of the trial? (randomization not always effective if sample size small) • Aside from the experimental intervention, were the groups treated equally? (ie. Cointerventions)

45. What are the results? • How large was the treatment effect? • Relative risk reduction vs absolute risk reduction

46. Eg. • Baseline risk of death without therapy=20/100 = .20 = 20% (X = .20) • Risk with therapy reduced to 15/100 = .15 = 15% (Y = .15) • Absolute Risk Reduction = (X-Y) = .20-.15 = .05 (5%) • Relative Risk = (Y/X) = .15/.20 = .75 • Relative Risk Reduction = [1-(Y/X)] x 100% = [1-(.75)] x 100% = 25%

47. Number needed to treat = NNT • To calculate simply take the inverse of the absolute risk reduction • In last example= 1/.05 = 20 is the NNT

48. What are the results? Cont. • How precise was the estimate of treatment effect? • Use confidence intervals (CI) = a range of values reflecting the statistical precision of an estimate (eg. A 95% CI has a 95% chance of including the true value) • CI narrow as sample size increases eg. In last example of 100 patients with 20 pts dying in the control group and 15 in the tx group the 95%CI for the RRR was -38% - 59%. If 1000 patients were enrolled in each group with 200 dying in the controls and 150 in the tx group the 95% CI for the RRR is 9%-41%.

50. CI cont • If CI cross 0 they are generally unhelpful in making conclusions • When is the sample size big enough? • If the lower boundary of the CI is still clinically significant to you (in + studies) • (or if the upper CI boundary is not clinically significant in negative studies)