Appraisal of an RCT using a critical appraisal checklist

1. Appraisal of an RCT using a critical appraisal checklist Spink MJ, et al. Effectiveness of a multifaceted podiatry intervention to prevent falls in community dwelling older people with disabling foot pain: randomised controlled trial. BMJ 2011;342:d3411.

2. Is the included population explicitly described?Is the setting explicitly described? • The RCT should clearly explain how it recruited people, and the inclusion/exclusion criteria it used, so that you can decide if this is representative of the target population and generalisable to your practice • Examples of fair recruitment are: random recruitment/recruitment of consecutive patients in a treatment centre. However, an intervention study will need to have informed consent from participants, and so it may use volunteer recruitment/recruitment from advertising to ensure that it recruits enough people • Also remember: inclusion and exclusion criteria should be relevant for the study method and question — i.e., a study shouldn’t exclude people with certain weight or height, if this is not relevant to the study outcomes or question posed

3. In this study: Is the included population explicitly described? • Look in Methods section for details of recruited participants. (The results section should detail the actual people randomised from this recruited population group.) Here, participants were recruited from the treatment centre database and by advertisements. Inclusion and exclusion criteria are clearly defined and sensible for study method and question

4. In this study:Is the setting explicitly described? • Look in Methods section in this RCT for details of the setting. Sometimes this information may be in the results section …

5. Is the intervention and control explicitly described? • The study should describe the intervention and control in sufficient detail, such that these can be replicated • Watch out for statements such as “usual care” — the RCT should define what this involves

6. In this study:Is the intervention and control explicitly described? • Look in Methods section for details of the intervention and control etc…

7. Are the outcomes explicitly described? • The study may report on several different outcomes • A primary outcome is the prespecified outcome that is considered to be the most important outcome of the study. It is best to have only one primary outcome in a study • Secondary outcomes are additional outcomes of interest. However, the study is often not powered to detect differences between groups in these outcomes • All outcomes should be clearly described in sufficient detail, including the timing of measurement, so that they could be replicated

8. In this study:Are the outcomes explicitly described? • Yes, look in Methods section. This RCT also refers to another protocol paper with further information, which you should also read …

9. Are the outcome measurements valid? (objective, using standard/validated measures) • Outcomes can be continuous (vary on a continuum or scale, e.g., weight, pain score), or dichotomous (yes/no outcomes e.g., death, heart attack) • Clinically relevant outcomes include outcomes that are important to the patient, e.g., survival, symptom improvement. Ideally an RCT should report on clinically relevant outcomes • Proxy measures are sometimes used because these may be easier/cheaper to measure, or can be assessed in a shorter time frame, e.g., intraocular pressure in glaucoma trials, or HbA1c in diabetes trials. However, proxy measures do not always correlate with clinical outcomes • Also watch out for: • Reporting significant outcomes only — an RCT should report on all its prespecified outcomes whether they are significant or not • Reporting continuous outcomes on unvalidated scales. It can be difficult to compare results of these with other studies

10. In this study:Are the outcome measurements valid? • Look again at the outcomes section — particularly the primary outcomes • There may be further information in the introduction or discussion sections if you are not familiar with the topic …

11. Is the randomisation method described?Is the study adequately randomised? • If a study is successfully randomised, then it should be impossible to predict in advance which group each person will be assigned to. The ideal is that everyone has a known and equal chance of being in either group • The following allocation methods are not true randomisation methods, as it may be possible to “guess” which group each person will be in in advance, so watch out for these! • Date of birth, day of the week, medical record number, month of the year, or the order in which they are included in the study (alternate)

12. In this study:Is the randomisation method described?Is the study adequately randomised? • Look in Methods for details of randomisation used …

13. Is the level of blinding reported? Could the level of blinding affect the interpretation of the results? (observer bias, bias in analysis, use of objective or subjective outcome measures, etc) • Double-blinding — i.e., both patients and investigators are unaware of treatment allocation is ideal, and minimises measurement bias. However, sometimes this may not be possible, for example, where the intervention involves a physical treatment or surgery or where adverse effects of a drug clearly compromise blinding. Single-blinding (where either one of the patients or the assessors are unaware of treatment allocation) may, however, be possible and acceptable in these circumstances, depending on the study design • Also remember — if the outcome is highly objective (e.g., death) then the requirement for blinding may be less critical, because measurement bias should not really affect the result. However, if the outcome is subjective (e.g., self-rated symptoms or function) then blinding of the assessor may be crucial to avoid the influence of bias on the result

14. In this study:Is the level of blinding reported? • Generally check the Methods section for information on blinding applied; however, sometimes further clarification may be found in other parts of the review — e.g., this RCT gives further clarification in the discussion section • This RCT was single-blinded (assessor blinding). It was not possible to blind the patients because of the physical nature of the intervention … …

15. In this study:Could the level of blinding affect the interpretation of the results? • To assess whether the level of blinding affects the interpretation of the results, look again at the outcomes section, and whether you decided if the outcomes were objective or subjective • In this study, the primary outcome of proportion of fallers, multiple fallers, and falling rate are relatively objective measures and so single blinding applied should not affect the interpretation of these results. Secondary outcome measures are more subjective, however, and so level of blinding applied, particularly no patient blinding, may suffer from bias

16. Are both groups treated in exactly the same way apart from the intervention? • Bear in mind that this includes all analyses, tests, and contact with the study personnel, as well as any other allowed treatments in the study • This may not always be possible/ethical depending on the nature of the interventions; however, you should decide whether any differences in treatment during the study could affect the results

17. In this study:Are both groups treated in exactly the same way apart from the intervention? • Look again at the Methods (description of interventions) section. You should also look at the results section to check that both groups had similar assessments and follow-up

18. Are treatment and control groups similar at baseline?Are any baseline differences explained/accounted for at analysis? • If the randomisation process worked, then the groups should be similar at baseline (remember that the goal of randomisation is that everyone should have a similar and equal chance of being in either group) • The study should specify whether any differences between groups are statistically significant, because any significant differences between groups at the beginning of the study could potentially affect the results

19. In this study: Are treatment and control groups similar at baseline?Are any baseline differences explained/accounted for at analysis? • Yes. Look at the results section. RCTs often contain a table describing baseline characteristics, as in this case

20. Are all randomised people accounted for at the end of the RCT? Does the RCT present an intention-to-treat analysis?Are final results based on more than 80% of people randomised? • Most trials will suffer from some loss to follow-up for various reasons (including patient withdrawal due to safety/efficacy/consent, lost contact, etc). However, to avoid bias and maintain the validity of the initial randomisation, loss to follow-up should be minimal — generally less than 20% of the randomised population. However, it should be noted that if few patients have the outcome of interest, then even small losses can bias the results • An intention-to-treat analysis is where the groups are analysed based on the initial treatment to which they were randomised (not by treatment eventually administered or whether they finished the trial!). This type of analysis is generally considered optimal for most RCTs (with the possible exception of non-inferiority or equivalence trials)

21. In this study: Are all randomised people accounted for at the end of the RCT? Does the RCT present an intention-to-treat analysis?Are final results based on more than 80% of people randomised? • Yes. Look at the Methods and Results sections (to see what actually happened). RCTs often contain a figure which shows the flow of people through the study, as in this case

22. Are all clinically relevant outcomes reported? • This requires your judgement — look at what outcomes were reported, and how these answer the question posed by the study

23. Does the RCT report a direct statistical analysis between groups?Does the RCT report appropriate test statistics? • The RCT should compare similar results between groups directly. Indirect comparisons, for example, comparison with baseline within a group, are difficult to interpret • The appropriate type of test statistic presented will depend on what type of outcome was measured. For dichotomous outcomes, the results can be expressed in many ways, including: Relative Risk (RR) = risk of the outcome in the treatment group / risk of the outcome in the control group Absolute Risk Reduction (ARR) = risk of the outcome in the control group – risk of the outcome in the treatment group Relative Risk Reduction (RRR) = absolute risk reduction / risk of the outcome in the control group. Number Needed to Treat (NNT) = inverse of the ARR and is calculated as 1 / ARR. • The study should present a confidence interval or P value so that you can judge the statistical significance of the result. However, bear in mind that “statistically significant” and “clinically significant” are different!

24. In this study: Does the RCT report a direct statistical analysis between groups?Does the RCT report appropriate test statistics? • Yes, generally look first at the Methods section and then at the Results section … … …

25. In this study: Does the RCT report on the clinical relevance/importance of the results? • Yes, generally find this information in the discussion section of the RCT

26. In this study: Are the setting and intervention generalisable to routine clinical practice/care? • Again, this requires your judgement — the study may make suggestions but you will know best what is generalisable to your area of practice