Understanding Elicitation Methods: Combining Expert Knowledge and Preferences

1. Elicitation Some introductory remarks by Tony O’Hagan

2. Welcome! • Welcome to the third CHEBS dissemination workshop • This forms part of our Focus Fortnight on “Elicitation” • Our format allows plenty of time for discussion of the issues raised in each talk, so please feel free to join in!

3. Two strands • The talks today address elicitation in two quite different contexts • Elicitation of expert knowledge in the form of probability distributions of unknown parameters • Elicitation of people’s preferences for health states, in order to formulate utilities • A key objective of this FF is to bring these two strands together, to see what each can learn from the other

4. Eliciting probabilities • Bayesian statistics concerns the updating of beliefs from prior to posterior • In principle, specification of prior beliefs is therefore a key component of any analysis • In practice it is rarely done seriously • Given sufficient data, it doesn’t matter much • It is difficult and time consuming to do properly • It doesn’t lend itself to nice papers in leading statistics journals

5. Not necessarily ‘prior’ • Eliciting expert opinion probabilistically does not have to be thought of as ‘prior’ to some data • Expert information is a crucial input to risk analyses, etc, where there is no prospect of getting more data • This is the area where elicitation methodology has been developed most

6. Elicitation in C-E analysis • Elicitation of prior beliefs for Bayesian analysis of clinical trial and other data • Often, sample sizes are not very large and prior information is appreciable • Elicitation of expert knowledge of inputs to economic models • Needed for sensitivity or value of information analysis • Fully Bayesian synthesis of information • Use posterior distributions for model inputs where possible

7. Challenges • Regulatory context • Elicitation will need to be robust, defensible • Correlations • How to elicit dependence? • Or how to structure/model the parameters so that the structure induces the right dependence?

8. Eliciting preferences • What is the ‘effectiveness’ for which ‘we’ are willing to pay? • We can only compare costs across the whole health care spectrum, and allocate budgets rationally, if effectiveness is on a common scale • Ultimately, we are willing to pay for health improvements that yield • Longer life and/or • Better quality of life

9. QALYs • The Quality-Adjusted Life Year seems to be the gold standard • Health providers’ willingness to pay can be reduced to how much they would pay for one QALY • Elicitation then involves asking people how they value different states of health • This is in order to know how to ‘adjust’ a life year • Various challenges seem to arise, both theoretical and practical

10. Questioning the QALY • Is two years of one person’s life worth one year of each of two people’s? • If two years of poor health is worth one year of perfect health, are four years of poor health worth two of perfect health? • Whose preferences/values? • Individuals vary wildly • Systematic differences between sectors of population • What does society or the health provider think?

11. Practical challenges • Choice of instrument: SG versus TTO • Health state is massively multidimensional • Can’t value every possible combination • Relevant dimensions depend on medical condition • How to quantify uncertainty in valuations • Deferred benefits valued differently from immediate benefits • Does that make sense from a societal perspective?

12. Some commonalities • Different instruments yield different answers • SG is also used to elicit individual probabilities • But probabilistic elicitation is more about distributions • A multidisciplinary perspective is needed • There is a lot of psychology involved • Questions of whose judgements are needed • Individual versus community • Results are subject to uncertainty • Always need to extrapolate from limited elicited data • Fundamental duality