Aimee Rensford March 2013

1. Outcomes-Focussed Contracting: Accountability,learning, and new approaches to integrated contracting Aimee Rensford March 2013

2. “In the public sector, proposals are created and evaluated with the expectation that every one of them will be a great success – that anything short of completely solving a problem is a failure. Creating a civil culture in which people understand that things seldom work as expected, especially when trying to solve complex, intractable problems, would support a shift toward learning. This shift is not to avoid accountability but rather to make learning a focus of meaningful accountability” (Westley, Zimmerman, & Patton, 2006)

3. Outline • Key findings from my research into the dynamics of accountability and learning in contracting for health promotion services • How do we foster ‘meaningful accountability’? • 3 Principles to guide us • Integrated contracting – the application of ‘Results-Based Accountability’ • Recent developments with Integrated Contracting – Streamlined Contracting with NGOs Pilot

4. Contracts • Link financial resources to health service outputs and outcomes • Specify the nature of the performance required • Create an accountability relationship where providers are required to explain and take responsibility for their actions (or inaction)

5. Model of Accountability

6. Key questions • Classical or relational contracts? • What was the funder/provider relationship like? • Do the accountability mechanisms support providers to learn about and improve what they do?

7. Classical Contracts • So called ‘complete’ contracts – specification of promises, obligations, and processes for dispute resolution is highly specified • Costly to write and enforce (leading to high transaction costs) • Particularly hard for HP – neatly defined deliverables are inflexible and not suitable for ongoing programmes that may need to adapt over time

8. Relational Contracts • Provide flexibility and easy flow of information within a trusting contract relationship • Preferable when some elements of a service are unverifiable or hard to measure • Strong relationship between funder and provider is critical (but also leads to high transaction costs) • Balance between trust and accountability

9. Contracting relationship theory: Agency theory • Foundational component of NZ Public Sector reforms of the 1990s • Assumes self interest and that agents (providers) will act opportunistically and shirk as much as possible • External monitoring and reporting key to controlling providers and ensuring goal alignment • Criticised for assumptions made about provider’s intentions • Threat of sanctions typically comes after the service has been delivered

10. Stewardship Theory • Proposes that goals are aligned • Relationships built on trust, reputation, & collective goals • Less emphasis on the validation of performance measures – proposes that if providers are trusted they will tend to provide more accurate information • Criticised for painting an overly ‘rosy’ picture of providers intentions

11. Findings: Contracts • Service Specifications formed the basis for ongoing reporting, and included measures to assess performance • Typically focused on outputs • Funders found it hard to develop ‘good’ performance measures (esp. in health promotion context) • Flexibility to deviate from the service specs necessary

12. Provision of regular monitoring reports against SS • So basically yeah, [providers] have service specs, which can be of varying quality, and they're accountable to deliver against those service specs (Funder) • You report to the measures so it’s basically outputs like have you done this, have you done that, but there is an opportunity in there to write, to tell your story (Provider)

13. Reporting as a progress monitoring tool One of the positive aspects about reporting is that it also makes sure that you keep things on track and that you are actually delivering what you agreed to, so it also gives, so it gives you a chance to reflect, if you like, on how things are going, what processes, etcetera, etcetera. So it’s a bit of a sort of a snap shot of where you’re at, so it’s really important (Provider)

14. Flexibility to deviate from SS If [the programme] veers away from what was originally decided but there’s a rationale for that and good reason then I think that’s fine as long as it’s being transparent and being discussed. So I don’t know that there’s just one way for something to end up working (Funder)

15. Classical or relational?Agency or stewardship? • Strong focus on achievement of contracted outputs reflects a classical approach to contracting • Yet the flexibility afforded in the management of contracts suggests a relational approach • Funders and providers showed a preference towards partnership-based stewardship approaches

16. Findings: Learning and Evaluation • Some NGOs incorporated evaluation into their programme planning processes but this was not across the board • Role of the manager in provider organisations critical to establishing a learning culture • Barriers to learning: • Difficult to measure behaviour change • Lack of time • Lack of funding for evaluation • Lack of emphasis on discussing failures • Lack of evaluation capacity

17. I think barriers to learning are often around the fact that we’ve got outputs we have to report and you can get very caught up in the operational side of programming and ‘cause you're so busy and you’ve got to get your reports done, you’ve got to get some outputs happening and then you don’t perhaps think about the bigger picture to learn from what you’re doing (Provider)

18. I don’t think that as providers of nutrition and physical activity we’ve really latched on to continually improving. You get something going and then you keep it ticking on would be my impression. I don’t think we’ve quite got there with that. Like I mean for example no one seems to report failures and there must be, I’ve never been to a nutrition conference or anything where someone’s stood up and said actually don’t, this process or programme didn’t work (Funder)

19. The relationship between Accountability and Learning • There was clear consensus that the accountability mechanisms such as reporting, while useful for monitoring progress, were not enough to ensure that learning and improvement is taking place • Both funders and providers looked outside of the reports to judge whether providers are ‘doing a good job’

20. The degree to which learning and evaluation is integrated into service delivery depends heavily on providers and/or funders motivations to include it • The accountability mechanisms are not set up in a way that requires providers to report on learning and evaluation findings • This contributes to a de-prioritisation of learning and evaluation

21. So what? Why do anything about it? • Both funders and providers identified the need to integrate accountability for learning into the reporting process to ensure effectiveness and efficiency of public health services to achieve health outcomes • The quality of information used to assess performance is critical given the role it should play in the management of contracts, and in the policy cycle • A key purpose of accountability is the promotion of learning in pursuit of continuous improvement (Aucoin & Hentzman, 2000)

22. Where do we start..?

23. Principle 1 • Work together to include formal expectations for reporting on learning and evaluation in the contract service specifications • Helps to ‘legitimise’ time spent on learning, and helps to make it a priority from the beginning of the contracting process

24. Principle 2 • Support evaluation capacity building strategies for both funders and providers • This will help to improve the quality of the measures used to assess performance Examples of ECB Strategies: • Training: attending courses, workshops, and seminars on evaluation • Coaching and mentoring: building a relationship with an evaluation expert who provides individualised technical and professional support • Involvement in evaluation process: participating in all stages • Technical assistance: receiving help from an internal or external evaluator • Appreciative inquiry: using strengths-based, collaborative, narrative approach to learning that focuses on what ‘gives life’ to a programme

25. Principle 3 • Integrate learning and evaluation in a way that supports a ‘stewardship’ approach to contract management • Strong, trust-based relationship is critical • “In situations where trust is present the trusted tend to disclose more accurate, relevant and complete information, whereas the trusting feels less need to impose social controls in order to gain access to or influence over information” (Swift, 2001)

26. Further considerations • Providers have multiple accountability requirements and need to streamline these where possible • Realities of contracting timeframes, short term contracts • Increasing pressure on planning and funding teams and reduced FTE to manage contracts

27. Integrated Contracting across Govt • Longer term agreement • Common set of terms and conditions across agencies • Ideally brings together all funding that an NGO is receiving from any government funder • Provider and all of their funders around the table • Results Based Accountability approach to developing the contract and reporting template

28. RBA Refresher • 2 key aspects: • Population level accountability • High-level outcomes e.g. adult smoking rate • Influenced by a plethora of external factors -NGO can not be held accountable for impacting the outcomes by themselves • Performance level accountability • What were the actual results achieved by those who directly received your service? • How much did we do? • How well did we do it? • Was anyone better off?

29. Benefits of ICs • Ensures that reporting is not just on outputs but also progress towards outcomes and the quality of service delivery • Funders no longer working in isolation – better understanding of the other services delivered • Opportunities for service integration where appropriate to support the achievement of common outcomes • Opportunity to reduce/streamline compliance costs (e.g. audit)

30. Streamlined contracting with NGOs pilot • Part of Better Public Services Demonstration project, lead by Ministry of Business Innovation & Employment • Aims to create a contracting framework that will: • Reduce compliance burden on NGOs • Reduce duplication – in compliance costs and contract management processes • Uses an RBA approach to linking the services provided by an NGO to the Crown purchaser’s desired population and client specific outcomes • Interested in measuring (among other things) the contribution the NGO makes to achieving outcomes

31. Proposed contracting framework Framework Terms and Conditions (FTC) Standard terms and conditions governing the relationship between government and NGO providers Framework Agreement Management Plan Specifies the binding commitments between Government agencies when engaging an NGO Provider (collaboration, coordination & consistency). The provider is not party to this document. DECISION-SUPPORT TOOL (DST) Outcome Agreement (5) Outcome Agreement (4) Outcome Agreement (3) Outcome Agreement (2) This may be in the form of high trust, integrated or a standard agreement, as guided by the decision-support tool RBA OUTCOMES This may be in the form of high trust, integrated or a standard agreement, as guided by the decision-support tool RBA OUTCOMES Outcome Agreement Management Plan (OAMP) This may be in the form of high trust, integrated or a standard agreement, as guided by the decision-support tool RBA OUTCOMES Outcome Agreement Management Plan (OAMP) Outcome Agreement Management Plan (OAMP) OUTCOME AGRMT RBA OUTCOMES Outcome Agreement Management Plan (OAMP) OUTCOME AGREEMENT (OA) Based on a standard form template that can be used to describe a high trust, integrated or standard agreement, as guided by the decision-support tool RBA OUTCOMES Outcome Agreement Management Plan (OAMP) (optional non-contractual management plan)

32. DecisionSupportTool(DST)

33. Summary • There are opportunities make the reporting process a driver of learning and service improvement • Integrated contracts provide one way of doing this in a coordinated way – but can also be supported in a 1:1 traditional style contract • MBIE model may be used to inform ICs going forward

34. Questions? Aimee_Rensford@moh.govt.nz