1 / 12

Evaluation : what is it, and why do we need it ?

Evaluation : what is it, and why do we need it ?. Prof. Jeremy Wyatt DM FRCP, Director Acknowledgments: Doug Altman & David Spiegelhalter Chuck Friedman & Trish Greenhalgh. What is evaluation ?. Describing or measuring something Entails data collection and analysis

amandla
Download Presentation

Evaluation : what is it, and why do we need it ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation: what is it, and why do we need it ? Prof. Jeremy Wyatt DM FRCP, Director Acknowledgments: Doug Altman & David Spiegelhalter Chuck Friedman & Trish Greenhalgh

  2. What is evaluation ? • Describing or measuring something • Entails data collection and analysis • Implies a set of criteria or judgements to be made • Usually with: • A purpose – making a decision, answering a question… • People (the stakeholders) in mind

  3. Evaluation as an information-generating cycle 2. Design a study 1. Question 3. Collect data, analyse results 4. Make decision

  4. Why evaluate ? • To learn as we go along [formative] • To ensure our systems are safe & effective, solve more problems than they create • To inform decisions made by others [summative] • To publish, add to the evidence base • To account for money spent (cover our backs) • To persuade stakeholders: health professionals, politicians, organisations, patients…

  5. Some evaluation stakeholders Trade associations System developers System suppliers Regulators Evaluation funder System purchaser Health information system Evaluators Tax payers Academic peers Professional bodies System users Patients Users of similar systems Friends & family

  6. Kinds of evaluation study Evaluation studies Qualitative studies Quantitative studies Measurement studies Demonstration studies Reliability studies Descriptive studies Validity studies Correlational studies Comparative studies

  7. The politics of evaluation Society Advocates Opponents Evaluation study findings Risk takers Regulators Watch out for “fear of the clear” Laggards Innovators Practitioners Academics Public sector Private sector The media Vested interests

  8. Does telehealth work ? UK Whole System Demonstrators - covers 6000 pts with HF, COPD, DM “If used correctly” telehealth reduced: • Death rates by 45% • NHS resource usage by 15-20% • NHS tariff costs by 8% [But how many people used it “correctly” ?]

  9. Key Telehealth study questions Not “Does telehealth work ?” But: • Who wants telehealth ? • Who engages with telehealth ? • Who benefits from it ? • In what care pathways is it cost effective?

  10. Why bother with evaluation – can’t we just predict the results ? No, the real word is too messy / complex: • Bike Ed - carefully designed training campaign for boys – doubled injury risk (Carlin J 1998) • Weekly exercise programme for nurses to reduce back problems - no reduction, interfered with work planning (Skargren E 1999) • Toughened glass tankards to reduce alcohol-related injuries – randomised trial in 57 bars showed 60% rise (Warburton A 2000)

  11. Conclusion: an evaluation mindset • Be realistic: aim to be informative, not definitive • Tailor each study to the problem and resources • Collect information to address main stakeholder questions • Be: • Focused (always have a plan) • Open (to intended & unintended effects) • Flexible (prepared to change your plan) • Evaluation, like politics, is the art of the possible - so exploit opportunities, in lab and field

  12. Workshop:Can we make evaluation of health IT part of NHS routine business?

More Related