Cost effectiveness in Human Services: simple complicated and complex considerations

1. Cost effectiveness in Human Services: simple complicated and complex considerations Towards a method that is useful and accurate Andrew Hawkins, ARTD Consultants • andrew.hawkins@artd.com.au

2. Valid public policy question • This presentation is based on a paper that • Challenges the view that human services programs exist in the real world as stable, replicable entities for which real world outcomes can be measured BUT • Accepts the validity of the question, are available funds better spent on program X or program Y? • Commonly expressed as ‘was program X cost effective’. • The paper does not suggest its a better method for evaluation and understanding (it isn’t!). It’s just a way of answering an important and recurring question.

3. A synthesis of ideas • Realist theory—stable, replicable programs do not exist other than as budget allocations • Chaos and Complexity theory—long term ‘real world’ outcomes of complex systems cannot be measured • Psychological theory—improved decision making and citizen behaviour change are the real intention of interventions • Cost effectiveness theory—the need for a common metric to measure the relative worth of interventions.

4. Realist theory • Programs exist in budget allocations • The interest in ‘program effectiveness’ only occurs because people assume programs can be deployed with similar effect in different situations, like drugs or bridges. • Programs do not emit a constant irresistible force that changes people • In the real world programs have their effects on intended recipients by being translated into actual interventions • Example: ‘Empowering local school principals’ or ‘Family case management’ • Realist theory focuses on identifying relatively stable Context Mechanism Outcome patterns

5. Program Intervention Immediate outcomes Person

6. Complexity theory • Interventions can be made into simple, complicated and complex systems • Chaos means that due to the problem of sensitivity to initial conditions you can’t make long term measures of simple systems. • The social world is a complex place—complex systems involve interdependence, non-linearity, unpredictability, emergent outcomes • You cannot measure the impact of a butterfly flapping its wings or predict the pattern of a snowflake-humans and their societies are infinitely more complex than these systems • A focus on real world outcomes assumes we live in a simple or complicated world where there is no chaos and things operate linearly—we know this is not true but persist with the illusion of control—statistical or otherwise!

7. Program Intervention Immediate outcomes Person The World Long term outcomes? Long term outcomes? Long term outcomes?

8. Psychological theory • Policy makers are notreally trying to generate outcomes, they are trying to change behaviour • The media and political process may demand outcomes—but these demands must not form the basis for decision making • The worth of programs is the extent to which they lead to interventions that better prepare people for the world—physically, emotionally, or spiritually. • Nobody can control what opportunities the world will give to people, or what decisions people will make—but people can be better prepared to take advantage of opportunities • Human behaviour may not be able to be predicted (it’s complex and chaotic!) but factors influencing behaviour change in positive directions can be understood • A starting point: the eight factors influencing behaviour

9. A start: behaviour change theories Three necessary and sufficient factors that influence whether a person performs a given behaviour: 1. The person has formed a strong positive intention (or made a commitment) to perform the behaviour. 2. There are no environmental constraints that make it impossible for the behaviour to occur. 3. The person has the skills necessary to perform the behaviour. There are five factors that influence the strength and direction of intention: 4. The person believes that the advantages (benefits and anticipated positive outcomes) of performing the behaviour outweigh the disadvantages (costs, anticipated negative outcomes); in other words that person has a positive attitude to performing the behaviour. 5. The person perceives more social (normative) pressure to perform the behaviour than to not perform the behaviour 6. The person perceives that performance of the behaviour is more consistent than inconsistent with his or her self-image, or that is performance does not violate personal standards that activate negative self sanctions. 7. The person’s emotion reaction to performing the behaviour is more positive than negative 8. The person perceives that he or she has the capabilities to perform the behaviour under a number of different circumstances; in other words, the person has perceived self efficacy to execute the behaviour in question.

10. Program Intervention Immediate outcomes Defined in terms of factors influencing behaviour Person The World Long term outcomes? Long term outcomes? Long term outcomes?

11. Cost effectiveness theory • Comparative analysis requires data on costs and outcomes. • Costs should not be aggregate but measured at the individual level—this is to allow calculation of the resources consumed and outcomes generated for individual people • A common metric is difficult to find in human services—there is no QALY or DALY. • All programs attempt to influence decision making and change behaviour in ways that promote human flourishing and reduce suffering • Cost effectiveness could look at the costs to shift important factors known to drive behaviour change • This approach requires us to ‘measure’ psychological states—this is itself problematic—but avoids trying to measure complexity

12. What we measure in this approach to cost effectiveness Focus of cost efficiency Program Intervention Immediate outcomes Measured in terms of effects on the common factors influencing behaviour Person What we ignore because it too complex to measure The World Long term outcomes? Long term outcomes? Long term outcomes?

13. Restatement of the problem Program funders deserve answers to questions about the costs and benefits of the programs they fund. • Were program funds well spent? • Which interventions are most cost effective? • What is the best use of available resources to maximise gains to society? Methods must provide the most accurate and useful information possible for decision making. Evidence based decision making will focus on what we can know, not what we want to know about programs and interventions.

14. Summary of the solution • Programs exist to solve real world problems. • Programs actually impact recipients as interventions that are designed to support people to make better decisions and change behaviour. • The real world or long term outcomes of interventions in complex systems cannot be measured. BUT • All interventions are designed to improve decision making and change behaviour • We may be able to measure the immediate effects of interventions on common environmental and psychological factors identified by research as important to behaviour change • Common outcomes metrics will allow interventions to be compared in terms of the costs and outcomes for individual people • Decision making can be guided by the incremental cost effectiveness ratio of different interventions.

15. Epilogue • While we wait for better measures of the effects of interventions in complex systems, an overarching question about the cost effectiveness of human services programs may be ‘what is the lowest cost way to maximise factors that generate behaviour change, reduce suffering and increase human flourishing?’