1 / 19

External Validity of Trials

External Validity of Trials . Background. External or ecological validity refers to whether the results of the trial can be generalised to the general clinical population. Randomisation per se does NOT make a study externally valid. Validity Issues.

deron
Download Presentation

External Validity of Trials

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. External Validity of Trials

  2. Background • External or ecological validity refers to whether the results of the trial can be generalised to the general clinical population. • Randomisation per se does NOT make a study externally valid.

  3. Validity Issues • Are the ‘correct’ patients being included in the trials? • Are the ‘correct’ practitioners taking part in the study? • Are the ‘correct’ facilities being used?

  4. Trial Problems • Many if not most trials are undertaken in circumstances that are far removed from ‘usual’ clinical practice. • Usually clinicians running trials are content experts yet the results are to be applied by non-experts.

  5. Osteoporosis Trials • Most of the large osteoporosis trials with fracture as an outcome recruited participants from ‘bone’ clinics. • Patients under care of clinical experts but results are expected to be applied by clinicians who are not experts.

  6. Participants • Usually participants who go into trials are different from those who do not. • These ‘storm troopers’ of patients usually comply with instructions better than average. • Combined with different clinicians and patients results can be different from the ‘real’ world.

  7. % (n) with Took part in trial n = 1 409 Did not take part in trial, n = 1 814 1 risk factor 42.4 (597) 62.9 (1 141) .000 2 risk factors 40.9 (576) 29.4 (533) .000 3 risk factors 13 (183) 6.9 (126) .000 4 risk factors 3.8 (53) 0.8 (14) .000 Trial participants v non

  8. Type of fracture Took part in trial n = 1 409 Did not take part in trial, n = 1 814 P value Overall 6.5% 10.4% .000 Hip 1.4% 1.8% .405 Wrist 2.5% 3.8% .044 Crude risk of fracture

  9. Type of fracture OR (adjusted) 95% CI lower upper P value Overall 0.42 0.31 0.55 .000 Hip 0.50 0.28 0.90 .021 Wrist 0.52 0.34 0.79 .002 Risk of fracture (adjusted)

  10. Alendronate • Alendronate is a bisphosphonate for osteoporosis treatment. Large trials conducted in expert clinical centres showed it reduced fractures with few or no side-effects. • Things were different when it was licensed and ‘real’ clinicians used the drug.

  11. Alendronate problems • To take a bisphosphonate one needs to take it on an empty stomach with a large glass of water and remain upright for at least 2 hours. • Otherwise the drug can get affect the oesophagus and cause ulceration. • In real life this is what happened and led to a warning. Trial data gave to evidence this was a problem.

  12. The Wrong Solution • Often once the phase III trial has been finished companies finance a ‘real’ world non-randomised study. • This is usually nothing more than a marketing exercise and produces worthless data on ‘real’ world experience of the problem.

  13. A correct solution • Involves some form of randomisation to eliminate confounding. • For example one might undertake a cluster trial of giving the drug non-specialist physicians to use of their patients BEFORE the drug is widely available. • This will produce more robust data.

  14. Alternatively • We should try and design our trials to be as pragmatic as possible so that the results are widely generalisable. • This means recruiting non-expert clinicians and their patients. • Using a pragmatic design rather than an explanatory design.

  15. Back Pain Trials • An early RCT of chiropractic manipulation for back pain was criticised by physiotherapists that the chiropractic was undertaken in private practice. • May have been a ‘Bach’ effect.

  16. MRC back pain trial • To address the issue of nice premises the MRC back pain trial included an arm where patients were randomised to be treated on private premises or NHS premises. • Increased the costs of the trial hugely.

  17. Surgical Trials • Often surgical trials have poor external validity because the new technique is usually developed in a teaching hospital among ‘expert’ surgeons (e.g. laparoscopic methods). • It is unlikely the results of surgery undertaken in ‘top’ hospitals would be the same as a ‘bog-standard’ DGH.

  18. Big and Simple • Try and make the trial large, as it allows for various PRE-SPECIFIED sub-group analyses (specialists vs generalists). • Simple which means one can get as many in as possible.

  19. Summary • Most trials will NEVER be able to recruit exactly the same types of participants as those who receive treatment in clinical practice. • PRAGMATIC trials have the strongest external validity and usually one of these should be done before implementation.

More Related