1 / 22

Measuring ROI on ACSI Surveys

Tim Evans Social Security Administration Office of Electronic Services tim.evans@ssa.gov. Measuring ROI on ACSI Surveys. Introduction. ROI on your surveys not directly attributable to your survey expenditures—you may be validating prior work Ways of using ACSI data for ROI purposes

malachi-hood
Download Presentation

Measuring ROI on ACSI Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tim Evans Social Security Administration Office of Electronic Services tim.evans@ssa.gov Measuring ROI on ACSI Surveys

  2. Introduction ROI on your surveys not directly attributable to your survey expenditures—you may be validating prior work Ways of using ACSI data for ROI purposes Identify user behavior that explicitly shows your site’s value Identify data that shows how you can improve your site to increase its value Identify and focus your (limited) resource priorities

  3. SSA and ACSI Began Q1/04 Four current surveys Main site FAQ’s site Retirement Planner site Internet Social Security Benefit Application--ISBA (not published) Two more surveys planned Business Services On-line (employer wage reporting) Disability (largest single group of surveyed visitors say they came for disability info)

  4. SSA ROI for ACSI Surveys--Summary Main survey shows high value + return on 2003 site redesign FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ’s availability Retirement Planner survey shows high value of Planner + the opportunity to focus our resources ISBA Survey helps clarify resource priorities in limited-budget environment

  5. Main Site Survey, Q3/04http://www.socialsecurity.gov Overall 75 Content 80 Functionality 76 Look & Feel 79 Navigation 73 Search 74 Site Performance 83

  6. Compare to ForeSee Results Benchmarks Overall Satisfaction: +5 vis a vis Q2/04 Gov’t benchmark; +4 vis a vis overall Look & Feel: +4/+3 Content: +2/+2 Navigation: +5/+3

  7. 2003 Site Redesign Site-wide adoption of CSS Style Sheets for common look & feel (colors, fonts, etc.) Added common header with top nav-links Primarily implemented via common DreamWeaver template for easy propagation site wide Context: ~80 SSA webmasters with central, but informal governance

  8. SSA FAQ’s Survey, Q3/04http://ssa-custhelp.ssa.gov/cgi-bin/ssa.cfg/php/enduser/entry.php Overall Score 74 Content 81 Functionality 77 Look & Feel 79 Navigation 75 Search 75 Site Performance 84

  9. Custom Questions Results Reveal ROI--#1 If there were no FAQ’s, how would you have contacted Social Security to ask your question? (N=1273) Send e-mail 16% Browse web site 8 Call 800 number 39 Call local SSA office 22 Visit local SSA office 10 Write SSA 1 Would not have contacted SSA 4

  10. Custom Questions Results Reveal ROI--#2 • After reading our answer(s), what do you plan to do next? (N=487) • Nothing; question answered 36% • Nothing; question not answered 4 • Send SSA e-mail 6 • Call SSA 800 number 14 • Call SSA local office 9 • Visit local SSA office 22 • Try different question 4 • Look elsewhere on site 3

  11. Key Comparisons for 2 Questions Send SSA e-mail 16% 6% -62% Call SSA 800 number 39 14 -65 Call SSA local office 22 9 -59 Visit local SSA office 10 22 +120

  12. ROI—Reduction Use of High-Cost Support Channels Forester Research Estimates Cost of Services Knowledge-Based Self Service: $1.17/incident Telephone support: $33/incident E-mail support: $10/incident SSA Results, shown by ACSI data 65% reduction in calls to 800 number 59% reduction in calls to local office 62% reduction in e-mails 120% increase in local office visits

  13. SSA Retirement Planner Survey, Q3/04http://www.socialsecurity.gov/r&m1.htm Overall Score 77 Content 82 Functionality 80 Look & Feel 81 Navigation 77 Site Performance 85 Tasks/Transactions 78

  14. SSA Retirement Planner Survey, Q3/04--Futures Likely to Return 85 Recommend 83 File on-line 61 What’s wrong with this picture? Is it good news or bad?

  15. Retirement Planner—Custom Question How do you plan to file for retirement benefits? (7/12-8/9 data; N=1127) On line 39% Face to face 37 Telephone 4 On line w/SSA help 2 More recent data shows ‘on line w/help’ running ~13%, ‘on line’ ~30%--totaling more than 40% (not enough data for conclusions)

  16. Pew Internet Studyhttp://www.pewinternet.org/PPF/r/128/report_display.asp How Americans Get in Touch with Government (May, 2004) Telephone 40% Web 24 In person 13 E-mail 11 Write a letter 10

  17. Pew Internet Study—Cont’d Why Americans contact Government Conduct Transactions 30% Get specific information 25 Register opinions 19 Help w/specific problems 11 For complex, problem-solving contacts, 62% prefer phone/in person

  18. What Does This Tell Us About ROI for Retirement Planner? With ~40% interested in filing on line, SSA is doing quite well (compare Pew 30% transactions figure) Futures Score shows opportunity to improve on-line application, as Boomers age and interest in on-line filing rises New ‘On-line w/Help’ Numbers strong indicator for future, bearing out Pew conclusions on complex contacts

  19. Internet Social Security Benefit Application (ISBA) Retirement/spouse applicants vastly more satisfied (>2:1) than disability applicants Far more (5:1) retirement/spouse applicants successfully complete on-line application Lowest scores in Functionality, Tasks/Transactions; cut across board, especially among disability applicants Disability applicants have far lower level of Internet experience (5:1 “beginner”), and may need additional help Expectations count: level of Internet experience inversely proportional to satisfaction

  20. ISBA and ROI Lessons Retirement/spouse applicants >80% of ISBA customers; universe will grow as Boomers age and demand service (Pew data cited earlier) Disability applicants have lower Internet experience, need more support via high-cost channels The whole disability application process differs substantially from retirement application process With resources limited, best use lies in improving overall Functionality, Tasks/Transactions All applicants will benefit from general improvements in the on-line application

  21. Wrap ROI on your surveys not directly attributable to your survey expenditures Ways of using ACSI data for ROI purposes Identify user behavior that explicitly shows value of your site Identify data that shows how you can improve your site to increase its value Identify and focus your resource priorities

  22. SSA Conclusions Main survey shows high value + return on 2003 site redesign FAQ survey shows clear reduction in use of high-cost support channels as direct result of FAQ’s availability Retirement Planner survey shows high value of Planner + the opportunity to focus our resources ISBA Survey helps clarify resource priorities in limited-budget environment

More Related