Systematising the process : the role of PSMs in informing model structure

1. Systematising the process : the role of PSMs in informing model structure Jim Chilcott Technical Director, ScHARR-TAG Suzy Paisley DoH Research Scientist in Evidence Synthesis Paul Tappenden Senior OR Analyst Consensus Working Group on the Use of Evidence in Economic Decision Models MRC HSRC Workshop 26 September 2005

2. Before we start…

3. Today’s presentation • Define ‘structuring’ within the modelling process • Describe current practice • Evidence used • Tasks undertaken • Discuss ‘What role problem structuring methods (PSMs)?’ Consensus Working Group on the Use of Evidence in Economic Decision Models

4. Defining model structuring • Disease and treatment pathway • Model scope • Modelling method Consensus Working Group on the Use of Evidence in Economic Decision Models

5. Current practice:HTA Guidelines HTA guidelines (n=32) None 17 (53%) Description 2 (6%) Justification 4 (13%) Sensitivity analysis 5 (16%) Not available 4 (13%) Consensus Working Group on the Use of Evidence in Economic Decision Models

6. Statement of problem Justification of approach Scope / perspective Structural assumptions Consistency Strategies / comparators Model type Time horizon Health states Cycle length Parsimony Current practice:Modelling guidelines (Philips et al.,2004) Consensus Working Group on the Use of Evidence in Economic Decision Models

7. Current practice:Reported (Cooper et al., 2005) HTA monographs (n=42) Diagram of model 26 (62%) Model structure and assumptions discussed 5 (12%) Consensus Working Group on the Use of Evidence in Economic Decision Models

8. Current practice:Observed • 8 case study projects • Description model structuring process • Short, semi-structured interviews • Examination of corresponding study reports Consensus Working Group on the Use of Evidence in Economic Decision Models

9. Evidence consulted Consensus Working Group on the Use of Evidence in Economic Decision Models

10. Reporting Consensus Working Group on the Use of Evidence in Economic Decision Models

11. Summary of themes • Model structure amended according to data availability • Clinician input (1st order validation) • Existing models consulted as 1st step • Iterative structuring process • Complexity of structure shifted both ways • Very limited reporting process of development • MSc dissertation provided most transparent reporting Consensus Working Group on the Use of Evidence in Economic Decision Models

12. Clinical expertise Published evidence Existing models Natural history Treatment pathway Modelling method Effectiveness evidence Current practice:Evidence used Consensus Working Group on the Use of Evidence in Economic Decision Models

13. Current practice:Tasks undertaken • Iteration • Consultation • Evaluation and comparison • Feedback and clarification • Decisions regarding simplicity, complexity, feasibility and inconsistency Consensus Working Group on the Use of Evidence in Economic Decision Models

14. Why PSMs? • Lack of transparency • Data modelling vs problem modelling • Validity of model (unquestioned question) • Issues that ‘can’t be modelled’ • ‘Not qualified to comment’ Consensus Working Group on the Use of Evidence in Economic Decision Models

15. Why PSMs? • Data led models vs evidence informed models • Use of expert opinion • Implicit use of PSM methods • Observed usefulness of action research process Consensus Working Group on the Use of Evidence in Economic Decision Models

16. ? PSMs / MSMs Messes Problems SODA SSM SCA Robustness Analysis Drama Theory Decision Conferencing Decision analysis Markov Queuing theory Simulation Game Theory Consensus Working Group on the Use of Evidence in Economic Decision Models

17. Characteristics of PSMs • Techniques to facilitate exploration / definition of a problem situation • Group based • Uncovering options • Enabling group action • Accepting uncertainty • Managing conflict • Facilitator - analyst Consensus Working Group on the Use of Evidence in Economic Decision Models

18. SODA Individual interviews => Merge maps => Group workshops => Commitment to action Cognitive maps Group map Expand/refine map Group ownership Consensus Working Group on the Use of Evidence in Economic Decision Models

19. SSM Consensus Working Group on the Use of Evidence in Economic Decision Models

20. Pros • Problem structuring, not problem solving • Gathering information, generating ideas • Retain flexibility in early stages • Demonstrate understanding of complexity, diversity and inconsistency • Prioritising and simplifying Consensus Working Group on the Use of Evidence in Economic Decision Models

21. Cons • Methods development required • Adaptation to HTA context • New skills • Ownership • Facilitation of group decision making • Reproducibility Consensus Working Group on the Use of Evidence in Economic Decision Models

22. Developing a mixed method Qualitative model structuring process to inform quantitative decision-making process. ‘Soft’ qualitative structuring process ‘Hard’ analytic methods Consensus Working Group on the Use of Evidence in Economic Decision Models