TF-MSP May 2012

1. TF-MSP May 2012 NREN Surveys for Assessing the Service Portfolio and Evaluating User Experiences/Demands (brief overview) Lajos Balint lajos.balint@niif.hu NIIFI TF-CPR meeting Reykjavik, 20.05.2012

2. TF-MSP meeting at CARnet Present:fi, uk, dk, gr, nl, hu, fr, hr, ie, be, lv, si, nordunet, dante, terena 9 May: NRENs Brokering of Services for their customers 10 May: Client Surveys / User Satisfaction Assessment / Methodologies The big questions: What? Why? When? Who? How?

3. FUNET Surveys- Harri Kuusisto • Bi-annual survey – third one ended just now (42 questions) (+ feedback from annual Funet conf. + web pages eg.for polls) • Respondent coverage: admin, tech, security, AAI • 69-72-68% responses, web based (72% = 65 orgs, 100 answers) (from 80 orgs, 400 potential respondents) (universities better response: 88%) • Topical coverage: quality, satisfaction, requirements wrt. network, service portfolio, features, security, AAI, admin. • Comparison with earlier responses (trends, new requirements …) • NPS first time: % of 9-10 pts - % of 0-6 pts = 65,5 % (good, but …) • Definite difference between university vs. polytechnics answers • Completion: 15…60 mins (39 crossings + 3 short texts) • Specific reports automatically generated

4. CARNet Surveys - Barbara Kolarek • Annual survey since 1997 (contractual obligation for the members!) • Appointed representative at each respondent • 241 / 432 and 1383 / 1798 organisations / sites (academic and schoolnet) • Structure: like TERENA Compendium (but also qualification/evaluation - 6 thematic areas) • Output: automatic graphs (go to web, CARnet depts, institutes, ministry) • Forward and backward information • 150 questions (80% of respondees respond but mostly incomplete answers!)

5. SURFnet Surveys - Walter van Dijk • Annual survey (responded by ICT mgmt of connected 156 institutions) • Since 2009, standard set of questions, 10 mins to answer online • About 80 responses(~50%, at each category between 40 and 60 %) • Acutal work outsourced • Mostly quality and satisfaction related qualitative questions (”quantified measures”) • NPS = 34 % (= 36% - 2%), universities just about 20 % … (but average 2011 score is 8.2!) • Corollaries: more consultancy, attention, support, excellence needed + campus level activities to be improved • Uptake of 14 services also measured (from 100% SURFcert to 20% SURFconnect)

6. HEANET Surveys - Peter O'Halloran • 65 clients, 1M users incl.schools • Diverse voices and perspectives (Universities, institutes, schools, others) • 33 products/services(connectivity + hosting incl.MM + MW + contractmgmt) • Multiple ways of communicating with customers (Board, requirements, reviews, polls, WGs, conf., WSs, NOC …) • Goal: evaluation/improvement: performance, services (evaluate+introduce), issues, common knowledge • All clients (one per each), e-mail, anonymous to the public • Define (Sep 11), Collect (Oct 11), Communicate (Nov 11 – March 12!) • Method: Survey-Monkey, consolidated by XL sheets and graphs • Questions about: HEAnet, services, NOC, user satisfaction • 68 % response rate (58% very satified, 37% satisfied, 5% neutral wrt. quality) (38% strongly satisfied, 57% satisfied, 5% neutral with value for money) (+ many more results) • Challenges/issues mentioned (by priority order): Budget/resources, mobility, cloud, demand for new services, data, others • Continue, simplify, third party (?), timing (?), engage non-respondents

7. NIIF/HungarNet Surveys - Lajos Balint • Picture is less pleasing than with the previous presentations (Responses: sometimes valueless, sometimes nothing new) (+ many outside surveys - TERENA, DANTE, EU projects …) • No personnel for surveying, no budget for outsourcing – difficulties • Dilemmas and crucial aspects - motivation, frequency, exploitation - coverage (thematic/respondent), granularity, complexity • Survey I: Wide SP coverage, external request, single execution (2010-11) 2700 invited units (Y/N + annotation) - 90 moderate responses (~3%) (however well exploitable output) • Survey II: Narrow (VoIP+VC) coverage, biannual execution (commensurable results) 20-30 invited institutes - 80-100 % high value responses (expert, responsible, motivated respondents)

8. BELNET Surveys – Koen Schelkens • 102 from 187 (55%) responses • Completing: ~30 mins • Overall Net Promoter Score: NPS = 65 % (68-3) • Customer Experience Index: >85 % • Satisfaction results: excellent • Strengths and weaknesses evaluated • SWOT analysis performed • Strategic conclusions derived • Many more questions involved (percentage results on importance and satisfaction derived)

9. Experiences - corollaries • Result depends on motivation-devotion (both sides) • Proper objective and careful preparation allow success • Experts prefer demanding surveys, unlike non-experts • Even very low (<10%) response level can be usable • Repeated surveys – commensurability is a crucial • Interactive on-line surveys should have multilevel memory • Allocate time-energy-cost also to due post-processing • Consider overall cost/benefit (resources vs. influence) Successful survey not yet guarantees attained impact!

10. Consequences for TF-CPR • Surveying is just one way of communicating with customers (Board, requirements, reviews, polls, WGs, conf., WSs, NOC …) CPR can/should well complement other ways of interaction • Common goal: improving: performance, services (evaluate+introduce), issues, common knowledge While surveys ask, CPR should inform the users/stakeholders • Diverse target groups need different handling when surveyed (research vs. education … communities, experts vs. non-experts …) CPR also should apply different (audience-dependent) approaches • Small results also can help a lot (low response rate in surveying – narrow segment of potential reachable community) Gradual extension of target community leads to stable CPR impact • Allocate time-energy-cost to all activity components (preparation - execution - post-processing both in surveying and CPR) Duly prepare-disseminate-evaluate CPR materials and effects • Consider overall cost/benefit also wrt. CPR resource spending vs. influence Successfully completed CPR actions not yet guarantee attained impact!

11. Conclusion • common goals • complementary activities • equivalent principles • same target communities • similar problems and solutions … Another reason why TF-MSP and TF-CPR should cooperate 