Monitoring and Evaluation in the Building and Construction Sector

1. Monitoring and Evaluation in the Building and Construction Sector A Brave New World Anne Duncan Manager, Sector Trends and Performance

2. Who Am I? • I am the Manager of Sector Trends and Performance in Sector Policy at the Department of Building and Housing • I am experienced researcher and evaluator and member of the AES • I have a strong background in social and health research and evaluation, with a primary focus around programme evaluation and outcomes.

3. Who Are We? The Department of Building and Housing was set up in 2004 and is the New Zealand government agency responsible for improving building quality and housing availability in New Zealand. The Government's medium-term priorities for the sector are to: • support rebuilding and recovery in Canterbury • provide effectively for those most in need of housing assistance through the right products and services • develop innovative policy and approaches to housing and building supply and affordability • work in partnership with third party providers to transform the social housing sector and improve availability of social housing • develop a sector skills strategy with agreed productivity goals and targets • implement agreed changes from the Building Act 2004 review • achieve greater efficiency, effectiveness and alignment of trade licensing boards in the sector • improve transparency of the cost to deliver Crown-funded programmes.

4. Our Big Issue

5. Weathertight – or not? • 42,000 homes between 1992 and 2008 were affected by trapped water rotting house structure. • PWC estimates costs at $11 billion Government response: • Set up the Department of Building and Housing in 2004 • Set up Weathertight Homes Remediation Service (WHRS) • Review of the Building Act (BAR)

6. Why Does it Matter? • A home is not just a house. It is where we live, enjoy our family, share our lives. • Building and construction shapes the way our cities look and function. • Buildings impact on our sense of community and cohesiveness • Buildings provide vital services that keep us working. Buildings are us!

8. The Response Now Needs to be Evaluated • So how do we know if this response has worked, and, if so, to what degree, where and for whom? • Can we use traditional evaluation approaches in this work? If not, why not, and what approach should we use?

9. Traditional Evaluation of Building and Construction Pass

10. Traditional Evaluation of Building and Construction Fail

11. Impacts of Weathertightness Which is the Pass and which is the Fail?

12. New kinds of evaluation in response to new challenges • Understanding of the function of evaluation and how it can add value in this situation. • Using evaluation to both educate and enable participants to take more responsibility for the work they deliver. • Using evaluation to bridge the relationship gap across government, industry leaders and businesses, providers of building and construction services, and the public – the end-users of these services.

13. How Can Evaluation Work When Everyone has a Different Definition of Success? Challenges: Evaluation of regulation interests a wide range of people: • Government – concerned with cost-effectiveness and whether policy objectives have been achieved or not • Industry and business – concerned with safety and standards • Those working in the sector – concerned with income and job security • The public are concerned about spending their money wisely, either in residential or commercial investment.

14. The Main Focus: The Building Act Review The BAR was intended to address a range of concerns about the existing building consent process: • Delays and high costs to consumers, builders, consent authorities • Regulatory oversight is not matched to building risk levels • Risk-averse regulatory approach is believed to lead to delays, and to discourage innovation • Inconsistencies in regulatory approaches and requirements across territorial authorities • Need for better accountability by building providers, and better informed decisions by purchasers

15. Intervention logic

16. Building Act Review – Aims Review of the Building Act 2004 to reduce the costs, but not the quality, of the building control system, to achieve the following results: • Quality homes and building are produced through a business enabling and efficient regulatory framework • Consumers can make informed decisions and have confidence in transacting in the building and housing market • Homes and building are produced cost effectively by a productive sector who have the right skills and knowledge • The regulatory system is administered in an efficient and cost effective manner.”

17. Workstreams • Accountability – everyone knows their rights and responsibilities • Consumers – consumers take ownership • Sector skills and productivity – the sector has the skills to do the job right the first time • Regulatory system design – the system is nationally consistent and efficient • Stakeholder engagement and communications – people know what they need to know

18. BAR as a System

19. Measures Accountability • Implementation of nationally consistent streamlined process • Time and cost of consent processing, by risk category Consumers: • Confidence in providers • Use of formal contracting • Satisfaction with building results • Volume of disputes • Level of Building Act or Building Code enquiries

20. Measures Sector Skills and Productivity • Number of Licensed Building Practitioners by license class and region, as proportion of requirements • LBP success rate in passing building inspections • Level of rework required Regulatory System Design • Clearer Building Code guidance • Improved access to Building Code System information • Implementation of risk-based consenting system • Time and cost of consent processing under risk-based approach • Building quality changes under risk-based approach: numbers of consent refusals, numbers of failed or repeat inspections, proportion of applications considered deficient

22. Ay, there’s the rub

23. Measures Sector Skills and Productivity Number of Licensed Building Practitioners by license class and region, as proportion of requirements LBP success rate in passing building inspections Level of rework required Regulatory System Design Clearer Building Code guidance Improved access to Building Code System information Implementation of risk-based consenting system Time and cost of consent processing under risk-based approach Building quality changes under risk-based approach: numbers of consent refusals, numbers of failed or repeat inspections, proportion of applications considered deficient

24. Co-Dependency It is easy to identify from this list that none of these areas exists in a vacuum. Any changes made to one could impact on any or all of the rest. They are inter-related to the point of co-dependency!

25. Complexity • The programmes are inter-related to such an extent that it seems to me impossible to separate out the impacts of any one of them • The programmes were being led by individual work stream leaders who did not have, as I saw it, a systems overview. That is, I wasn’t convinced that they understood the complex set of interactions and inter-impacts (is there such a word?) that were going on. • The measures we identified seemed to simplistic. I wasn’t convinced that they could capture the reality of the situation and tell us what, if anything, was changing. Once I had identified these issues, it became clear that we needed to consider the BAR as a complex adaptive system – something that was changing and adapting all the time. As articulated by Patton, if we focus too much on measuring simply the goals of programme in evaluation, we will miss the unanticipated effects. [1] [1]Patton, M., Q. (2008) Utilization-Focused Evaluation 4th Edition. Sage Publications: United States of America

26. Risks • Interdependencies of workstreams • Political landscape • Cost • Scope Main risk: Interdependency Evaluation, generally speaking, can only really follow the horse and cart.

28. Risks (cont) • Political landscape: Working with 79 Building Consent Authorities (BCAs) who each have, to some extent, their own way of doing things. Part of the BAR is to bring them together into a small number of regional authorities who operate consistently. This will help! • Cost and Scope are usual project risks that can be managed. The key mitigation is to ensure you are aware of costs and scope as you progress so you can respond early.

29. Where Are We Now? • Baseline Data – so we know where we have come from • Key statistics dashboard: Monitoring baseline data monthly on processing of building consents, number of Licensed Building Practitioners, consumer awareness and consumer attitudes etc. • Presented in a regular Indicators Report

31. Other Work • Consumer satisfaction survey NRB has been contracted to conduct a telephone survey of consumer satisfaction with building services. We hope to repeat this survey on a yearly basis and see the changes from the BAR reflected in increased customer satisfaction. • Scanning changes in the consenting process Working through the websites of current Building Consent Authorities to monitor any proactive changes to their consenting systems and collect baseline data. Those who are taking a proactive change are likely to be approached for interviews. • Case studies Auckland and Christchurch are piloting ‘risk based consenting’. We will use case studies to evaluate the pilot and make any changes before the process is rolled out nationally.

32. And Next? • Onwards and Upwards into the Wild Blue Yonder!

33. I’ll let you know how we go. Thank you and any questions? Anne.Duncan@dbh.govt.nz