Customer Expectations & WTP For Improvements in Service from Electricity DNOs

1. Customer Expectations & WTP For Improvements in Service from Electricity DNOs Pilot Debrief February 2008

2. Agenda • Fieldwork Methodology • Interviewer Feedback • Stated Preference Methodology • Stated Preference Findings • Summary of Recommended Changes

3. Fieldwork Methodology • Objectives of pilot were to test: • the recruitment process • the survey hit rate • the clarity and flow of the questionnaire • the appropriateness of the language used • the accuracy of all routings • ease of use of the show material • design and understanding of the stated preference exercises • the interview duration • Approach: • Businesses: 40 CATI phone-post/email-phone (41 achieved) • Residential: 100 CAPI in-home interviews (99 achieved) • Quotas: • Business: • 10 EPN (11 achieved), 10 SPN (10 achieved), 20 LPN (20 achieved) • Mix of size (20 small; 15 medium; 6 large) & sector • Minimum 10% with outage experience (39% achieved) • Residential: • 25 EPN (25 achieved), 25 SPN (25 achieved), 50 LPN (49 achieved) • Mix of age & SEG

4. General Feedback: Interview Duration DOMESTIC Mean = 32 minutes BUSINESS Mean = 43 minutes

5. Problems with clarity & flow: Question 34: ‘did you find each of the levels of service described easy to follow?’… if you said ‘no’, Question 35 then said ‘which levels weren’t clear to you?’; seem like separate issues Need showcard for income question Problems with routing: Q22b; if they say don’t know should skip subsequent question General comments on what would make it easier Concern that people will think we are trying to sell them services for an electricity company. Also, many people less aware of Ofgem than of Ofwat or the FSA, so they were very cautious. Maybe a more elaborate explanation of the survey might help. Interview duration: Longer than budgeted for in most instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates) and current incentive (due to length) Other: Some issues with defining rural versus urban – we will use a postcode dbase to do so in the mainstage Add all screening questions (eg age, SEG) to main questionnaire Add question about number of children 5+ General Feedback: Comments on Domestic Questionnaire (non SP)

6. Problems with clarity & flow: Some people assumed it was a sales call despite the current intro. Need to further emphasize that Ofgem are a regulatory body and that the research is looking at improvements that could be made to the services provided to their company RQ13 – need a “don’t know code Q23: some said “both”; add this code? Q26: needs a don’t know or can’t remember Q40: needs re-phrasing so it doesn’t suggest anything will be sent; needs to separate leaflet and email; needs to be multicode Problems with routing: none Interview duration: Longer than budgeted for in most instances; cannot be done within current fee (due both to time required for each and impact of length on hit rates) Other: Some issues with defining rural versus urban – again, we will use a postcode dbase to do so in the mainstage General Feedback: Comments on Business Questionnaire (non SP)

7. Stated Preference Methodology • Objective: to obtain estimates of customers’ willingness-to-pay for improved service levels (or willingness-to-accept for reduced service levels) • No revealed preference evidence, so need to rely on stated preferences • Customers are not offered different service levels and prices in real world • Two possible stated preference approaches: • Contingent Valuation questions (CV) – ask respondents directly questions about valuations • Discrete choice experiments (SP DCE) – elicit valuations through tradeoffs • Based on hypothetical choices • Essential that price is one of the attributes

8. Example Choice: Experiment 1 (Domestic)

9. CV vs SP DCE

10. Attributes Tested for Domestic Customers Cost also included in all experiments

11. Attributes Tested for Business Customers Cost also included in all experiments

12. Stated Preference Methodology • Potential problem with separation of attributes across experiments is that results may over-estimate total wtp • Therefore we also include a higher-level experiment, including all attributes to test total wtp • We also include a series of CV questions to elicit total wtp for best service • The higher-level experiment is simplified so that ‘group’ of attributes are varied simultaneously • Two levels tested • Lowest service level • Highest service level • Still complicated

13. Example higher-level choice: 6 choices

14. Stated Preference Methodology – Analysis • Analysis based on principles of utility maximisation • ‘Utility’ assigned to each alternative, made up of: • Attributes of the alternative • Characteristics of the respondent, e.g. income, etc. • Assumption that error distribution is extreme value (iid) leads to logit model • Maximum likelihood techniques used to obtain values of unknown parameters • Find the βsthat optimise explanation of the model • Model outputs • Coefficient estimates (and t-statistics) • Model fit statistics

15. Parameters to be estimated Random error reflecting influence of parameters not included, differences between individuals/responses Characteristics of alternative/individual Utility Specification

16. Logit Model Formulation Assumption that random terms () are independent and identically distributed across alternatives (IID) and that they vary with an Extreme-Value distribution

17. Stated Preference Methodology – Analysis • Coefficients used to: • Determine the relative importance of attributes • Determine the monetary value for attributes • Specify utility functions for prediction models • Monetary values for attributes identified in lower-level games • May be adjusted by values for groups of attributes identified by higher-level game or by CV question • This is a judgement call

18. Stated Preference – Did it Work? • Trading behaviour • Background Questions on respondent understanding • Model Results

19. Most domestic respondents were willing to choose new service options in at least one of the choices • Domestic – London • Domestic – Other DNOs

20. Most business respondents were willing to choose new service options in at least one of the choices • Business – London • Business – Other DNOs

21. Examining choices at different cost levels implies that many domestic customers were willing to pay 15% increases in DUOS for service improvements Other DNOs Experiment 1 Experiment 2 Experiment 3

22. A wider range of costs was investigated in the higher-level experiment with large numbers of domestic users choosing alternatives with large cost increases

23. Examining choices at different cost levels implies that many business customers were willing to pay 15% increases in DUOS for service improvements Other DNOs Experiment 1 Experiment 2 Experiment 3

24. A wider range of costs was investigated in the higher-level experiment with large numbers of business users choosing alternatives with large cost increases

25. Most respondents understood the experiments • ‘Were you able to make the comparisons in the choices we presented to you’ • Domestic • 48 of 49 respondent in London reported they were able to make the comparisons • 47 out of 50 respondents in other DNOs reported they were able to make the comparisons • Business • 19 of 20 respondents in London reported they were able to make the comparisons • 20 of 21 respondents in other DNOs reported they were able to make the comparisons

26. A sizeable proportion of domestic customers felt that the levels of service in the choices were unrealistic, and the experiments were difficult • ‘Were the levels of service we have been asking about in the choices realistic?’ • London, 11 of 49 domestic respondents felt the service levels were not realistic or too difficult • A 1 in 10 year electricity cut is not realistic because we are having electricity cut 6 times in a year • I don’t think all the changes will be possible within the costs stated • It is seriously insane to raise the bills again and again • No electricity company would cut emissions, as they are not efficient enough to replace the current system • There were too many options and everything was too similar • I don’t believe the whole concept, it’s not clear enough • They are all the same • Too much complicated • Other, 8 of 50 domestic respondents felt the service levels were not realistic or too difficult • Because they are not achievable, ie the cost of putting all these cables underground isn’t going to be economically viable • The last thing I want is for my bill to go up • They are talking about increasing expenditure by certain percentages, but the original expenditure is not listed… so 15% increase of what? • The comparisons are not obvious, got to really read in order to give an answer • The different choices or packages didn’t always seem logical. • Too much to absorb all at one for the average consumer who is having enough problems paying their bills

27. Most business respondents felt that the levels of service in the choices were realistic • ‘Were the levels of service we have been asking about in the choices realistic? • London, 3 of 20 respondents felt the service levels were not realistic • 7 power cuts in 10 years is outrageous • Because we had 3 power cuts in the last 2 years of long duration • They do not employ enough people, so realistically it is not going to happen • Other, 4 of 21 respondents felt the service levels were not realistic • Because some reductions are not able to be guaranteed. There are elements outside your control, i.e. the power cut we experienced was due to accidental cutting through power cables. • Some options offered more for less….doesn’t make sense. • Typical of today’s climate, people want all these things, but are not prepared to pay for them. Realistically, these improvements would probably cost too much. • Some of the improvements would be very difficult to achieve.

28. Model results are encouraging • Domestic Customer Models • Results are very encouraging • Cost coefficient is significantly negative • Most other coefficients have intuitive signs, many are significant • Resulting wtp values look reasonable • Business Customer Models • Many statistically insignificant terms, probably a result of small sample sizes and heterogeneity of sample, e.g. in terms of size of business • Expect model to be improved with larger sample size in main survey • Most important variables: • Power cuts • Restoration of supply • Reduction of carbon emissions

29. Interviewer Feedback on SP • “Very complex” • “Too much to consider” • “Consider reducing and rotating over 2 interviews to make each survey shorter” • “Maybe cut down on the sets of choices; if possible, 6 instead of 8. Choice sets are quite long” • “Most respondents found the exercise difficult or complex at first but then adapted to it. It is a bit difficult to keep them interested in the survey till the end” • “The interview should be shortened if possible” • “It’s a lot of reading”

30. Conclusions • Subject area is abstract and the choices are complex, but – despite some concerns on the part of interviewers – respondents seem to have been able to undertake the SP experiments • Some, mostly minor, revisions required for main survey • Main area for review is cost levels • Mainstage interviews: • 2,100 domestic face-to-face interviews (150 interviews conducted in each of the 14 DNO territories) • 1050 CATI business interviews (75 per DNO territory)

31. Recommended Changes • Review cost levels • Pilot cost levels: • Fewer reductions and more increases for Experiments 2 and 3, e.g. 2 reductions, 6 increases and base • Recommend that we incorporate larger cost increases • Review wording of attributes and levels to simplify as much as is possible (may not be much scope for this) • Include ‘better than now’ and ‘worse than now’ descriptors, where appropriate • Review ‘Network resilience to flooding’ definition • Complaint that it was difficult to make comparisons when base level is not defined • Reduce number of choices in lower-level experiments from 8 per experiment to 6 per experiment

32. Other Recommendations • Minor Text Changes • Compensation for Multiple Interruptions: minor error in wording ‘which will entitles you…’, plus (£x) in text to be removed • Network Resilience to Major Storms: include a ‘,’ separator in the numbers, e.g. 180,000 rather than 180000 • Minor Changes to Questions/Routing • Slightly reword Q34 and/or Q35 (domestic) / Q46 and/or Q47 (business) • Amend routing for Q22(domestic) / Q33 (business) “don’t knows” • Use postcode dbase to define rural & urban • Add age & SEG data from screener to main questionnaire (domestic only) • Add question about number of children 5+ (domestic only) • RQ13 (business) – need a “don’t know code • Q18 (domestic)/Q23 (business): some said “both”; add this code? • Q20 (domestic)/Q26 (business): needs a don’t know or can’t remember • Q40 (business only): needs re-phrasing so it doesn’t suggest anything will be sent; needs to separate leaflet and email; needs to be multicode • Expand explanation of survey & of Ofgem • Reduce the length of the questionnaire or increase fees & incentives