Defeating ‘Benefits Fraud’ Project Challenge Sept 25 th 2008

1. Defeating ‘Benefits Fraud’Project Challenge Sept 25th 2008 Stephen Jenner stephen.jenner@cjit.gsi.gov.uk

2. Major change programmes - the rationale

3. But the track record is not good… “Implementation of IT systems has resulted in delay, confusion and inconvenience to the citizen and, in many cases poor value for money to the taxpayer.” Public Accounts Committee of the House of Commons, 2000 “Deficiencies in benefits capture bedevils nearly 50% of government projects” OGC Gateway News, 2003 “30-40% of systems to support business change delivery no benefits whatsoever.” OGC Successful Delivery toolkit, 2005 “the committee has no confidence that the amounts being assessed have any relationship to the benefits anticipated to be returned.” US Senate Appropriations Committee, 2007

4. And ‘horror’ stories abound… • The Dome visitors down by 5-6 million • Channel Tunnel – Eurostar passenger forecast in yr1 of full operations = 15.9m, actual = 2.9m • Bangkok skytrain – demand 2.5 times over-estimated.

5. And research indicates this is a widespread problem… • 2002 Mott MacDonald study for HM Treasury: • 2% benefits optimism bias BUT and • it’s a BIG BUT – most did not either quantify the benefits and/or evaluate the benefits realised. • Flyvbjerg et al (2005) Demand forecasting in road and rail projects found: • The average overestimation in rail projects was 106% • Road projects - around half were inaccurate by +/-20% • No improvement in accuracy over the last 30 years.

6. And it’s concluded… “There is a demonstrated, systemic tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors…appraisers tend to overstate benefits, and underestimate timings and costs.” HM Treasury Green Book “Delusional optimism: we overemphasise projects’ potential benefits and underestimate likely costs, spinning success scenarios while ignoring the possibility of mistakes.”Lovallo and Kahneman “forecasters generally do a poor job of estimating the demand for transport infrastructure projects”. Flyvbjerg et al

7. The explanations… • The technical explanation - our investment appraisal tools are inadequate in relation to future uncertainty? • The psychological explanation – Optimism bias, derived from cognitive biases? • The economic explanation – approval creates work for project managers, suppliers etc • The political explanation – “Strategic misrepresentation”?

8. The explanations - Strategic Misrepresentation? “the planned, systematic, deliberate misstatement of costs and benefits to get projects approved.” In short, “that is lying”. And that’s because, “Lying pays off, or at least economic agents believe it does.” Flyvbjerg et al “Figures don’t lie, but liars can figure” Sharpe and Keelin 38% of respondents in one survey openly admitted to overstating benefits to get funding with the traditional investment appraisal process being, “seen as a ritual that must be overcome before any project can begin” Ward

9. The result - Business Cases contain ‘assumptions masquerading as facts’

10. Examples of ‘Benefits Fraud’ or being, ‘economical with the actualité’ • Double counting • Inconsistent valuation • Over-estimation of impact • Under/over valuation of impact • Efficiency ‘savings’ or buying vouchers

11. Key messages – 3 challenges… • Ensuring benefits claimed are robust and realisable • Capturing all forms of value created • Realising benefits and creating value.

12. 1.1 Ensuring Benefits forecasts are robust and realisable – the Benefits Eligibility Framework • The set of rules about what benefits can and can’t be claimed and how they should be valued • Ensures a methodologically sound approach to measuring and valuing benefits • Allows a consistent approach across the Portfolio – a level playing field and so • Enables comparisons over time and between projects “Use a benefits framework to define business benefits” Source: Gartner, ‘Show Me the Money: Advanced Practices in Benefits Realisation’

13. 1.2 Independent review – a ‘fool’s errand’ • Ayerssuggests an, “’Advocatus Diaboli’…whose job it is to poke holes in pet projects. These professional “No” men could be an antidote to overconfidence bias.” • Davidson Frame proposes the use of “murder boards” – a cross departmental panel charged with pulling a proposal apart to, “make sure that arguments in support of project ideas do not have built into them the seeds of their own destruction.” • Steve Jenner - I suggest a ‘fool’ to ask the questions others dare to ask and identify those assumptions that masquerade as facts.

14. Triangulation Use more than one appraisal method, or value lens: combine financial metrics with multi-criteria analysis and decision conferencing, and examine ‘Attractiveness’ in the context of ‘Achievability’. 1.3 Triangulation

15. 1.4 Validation in the CJS IT portfolio… Project View of Benefits Benefits Contributor Business Case Development 1-Page Summary Benefits Report Project Live Service If funding approved Measuring & Tracking: Initial Baselining Benefits Eligibility Framework Benefits Evidence & Revised Forecast CJIT VMO Check alignment with BEF e.g. double counting, cost avoidance Project Benefits Lead and Benefits Realisation Lead agree Efficiency & Effectiveness benefits Benefit Realisation Plan: Police Benefits Validation with Recipients Benefit Realisation Plan: CPS Benefits Recipients Measuring & Tracking: Benefits Realisation Benefit Realisation Plan: Courts Benefit Realisation Plan: Corrections Benefits Validation with Efficiency Planners Benefit Realisation Leads and Efficiency Planners agree Efficiency Benefits that contribute to Departmental Efficiency Plans CPS Efficiency Plan Efficiency Planners DCA Efficiency Plan HO Efficiency Plan Confidence Benefits Validation with Strategic Planners Victims & Witnesses Strategic Planners agree Efficiency & Effectiveness Benefits that contribute to Strategic Targets & priorities Strategic Planners Offences Brought To Justice (OBTJ) Enforcement

16. Value Management 2 - Capturing all Forms of Value Political Efficiency Effectiveness Foundation Social Value Categories of Value

17. Understanding the logic that underpins an investment Hypothesis (IT Functionality) Root Cause (Level 2) Root Cause (Level 1) Consequences Targets Initiative Cost and measurement data underpins consequence

18. 3. Realising benefits and creating value • Leadership – re-setting the standards of expected behaviour • Who writes the business case? • Transparency & Accountability - Regular stage gates with formal re-commitment to the cost/benefit case • REAL post implementation reviews - Track results and use them to provide an evidence base for future forecasts • Managing benefits from an enterprise-perspective and asking - Is that the best we can do? And…

19. ‘Clear line of Sight’ from Strategic Intent to Value creation

20. Thank you – Questions? Stephen.Jenner@cjit.gsi.gov.uk