Tensions of Measurement: community investment activities in housing associations

1. Tensions of Measurement: community investment activities in housing associations David Mullins and Vanessa Wilkes, TSRC Housing Studies Association Conference 15th April 2011

2. Summary • Sector Context • Evidence Base • Levels of Measurement • 5 levels of measurement • Questions and Issues • Tensions • External Drivers • Internal Drivers • Conclusions • Further Research

3. Sector Context • Well established industry, large enterprises (50% market share in social housing) • Entrenched hybridity (social & commercial) • Core business highly regulated & measured • Non-core ‘community investment’ (CI) activities symbol of independence • Legitimacy & control issues brought to fore – 2008 Housing & Regeneration Bill & NHF Audit • New tools emerging to measure social impact of CI activities (in-house & off the shelf)

4. Evidence Base • Scoping study – emerging approaches to measurement of social performance (2010) • literature review & checklist (17 tools to set priorities, measure outcomes or link these to corporate goals) • 7 Case studies (organisation/tool based: 3 in-house, 4 outsourced) • 2 workshops (now 3, plus follow up tracking e.g. of first sector specific tool) • NHF Neighbourhood Audits 2008, 2011 • Sector overview of CI (64% coverage by stock) • Inputs/Outputs (£435mil CI in 6,800 projects 2008; leverage 1.6:2.7) • Linked PhD • Impacts & outcomes, depth case studies • More on this later!

5. The levels of impact measurement OECD definition of impact measurement “the impact of all changes, positive or negative, attributable to the intervention” • Also – intended and unintended effects and long-term as well as short-term Original argument (Zappala and Lyons 2009): • 3 levels of impact measurement • Whole sector • Individual organisations • Individual projects • Tools, frameworks and methodologies not easily transferred between levels • Framework developed in relation to housing study (Mullins 2010) to 5 levels

6. 5 levels of impact measurement • Sector level • Organisational level • Team level • Project level • Individual Transferability between levels is dependent on: • How the organisation adopts and uses the tool • Having the necessary skills and expertise • Relationship between internal and external data needs • A corporate awareness of what is required from impact measurement • The understanding and support by staff • Maximising the usage of the data at more than one level

7. Questions and Issues Whole sector • Is it possible or feasible to measure impact? • Dependent on collective approach and understanding • Expensive • Housing sector – disparate projects and circumstances • Draw on information from level 2 Organisation / Team / Project • Reason / motivation • External requirement? • Internal legitimisation / accountability? • A learning tool? • Availability of skills and expertise • Not necessarily new methodologies but alternative application • Interpretative and subjective decisions • Draw on information from levels below

8. External Tensions External Drivers Emerging Tensions Data for Funders Conflicts with internal needs Multiple monitoring Timescales Mission Drift Sector promotion ‘more than a good story’ Limited no of PIs obscure complexities of context & impact • Funding • Sector Promotion • Dances around regulation • Common outcomes / community budgeting • Wider third sector policy influences (e.g. promotion of SROI)

9. Internal Tensions Internal Drivers Emerging Tensions Commercial & Social Goals Social Dividend from commercial success Tendency toward harder measures – leverage Contribute to core business Attribution problems – leads to focus of CI on tenants Corporate v Project Levels Who sets the goals? which CI outcomes get reported? • Accountability & Reporting (variety of audiences) • Boards & Trustees • Management • Making sure people enjoy the projects • Benchmarking • Organisational Power Play • Promoting visibility of CI within corporate scorecards • Aligning CI activities to corporate goals

10. Conclusions • Early days but likely to grow • High Expectations in advance of evidence! • Tensions around types of measures and uses made of the data (both external & internal) • Is it worth it? (practical issues, costs motivation/training) • Underlying tensions reflect hybrid and large scale of these organisations & competing interests (e.g. Core business v CI)

11. Further Research • The role of community investment measurement tools within the organisation • How is community investment integrated and understood? • What measurement tools are used, what is their strategic fit within the organisation? • Data considerations (internal and external) • How is data reported and used? • What level of understanding does it provide? • The practical implications of measurement • Does impact measurement cause organisations to rethink their mission, goals or values? • What are the operational issues and practical considerations which affect measurement, such as length of contracts

12. Thanks for Listening Please Keep in Touch David Mullins D.W.Mullins@bham.ac.uk Vanessa Wilkes vew930@bham.ac.uk