Conducting Statewide Customer Satisfaction Surveys

1. Conducting Statewide Customer Satisfaction Surveys Brad Sickles USDOL / ETA ETA Performance Accountability Team

2. Measuring Customer Satisfaction • Participant Customer Satisfaction • Employer Customer Satisfaction Customer ratings on each of the three core questions regarding satisfaction are weighted to account for regional differences and reported on a 1 to 100 scale. Computation of the ACSI is discussed in TEGL No. 6-00. The American Customer Satisfaction Index (ACSI) is used to report statewide results to the Department.

3. The Core ACSI Questions 1. Utilizing a scale of 1 to 10 where “1” means “Very Dissatisfied” and “10” means “Very Satisfied” what is your overall satisfaction with the services provided from ________? 2. Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? “1” now means “Falls Short of Your Expectations” and “10” means “Exceeds Your Expectations.” 3. Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? “1” now means “Not very close to the Ideal” and “10" means “Very Close to the Ideal.”

4. Participant Customer Satisfaction • The weighted average of participant ratings on each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. • Who: Exiters • When: 1to 60 days after date of hard exit or date a soft exit was determined • How: Telephone survey • Required completions: 500 statewide • Required response rate: 50% first year; 70% following years

5. Employer Customer Satisfaction • The weighted average of employer ratings on each of the three questions regarding overall satisfaction reported on a 1 to 100 scale. • Who: Employers who receive a substantial service. • When: 1to 60 days after date of the completion of a service. For employers who listed a job order where no referrals were made, contact should occur 30 to 60 days after a job order was listed. • How: Telephone survey • Required completions: 500 statewide • Required response rate: 50% first year; 70% following years

7. Capturing Customer Satisfaction Results to Assess Local WIBs • States may use the ACSI questions and approach to measure satisfaction at the local level • States may also develop their own customer satisfaction instruments and approach for assessing local WIBs • Approaches must be applied uniformly and consistently to all WIBs • TEGL 6-00 outlines parameters for collecting customer satisfaction results

8. Revisions to TEGL No. 6-00 Surveys to Obtain State Level Results… • Response rates: 50% for the first year of data collection; 70% for following years. • Response levels below the specified minimums will invalidate the results. • States are required to determine the appropriate sample sizes. States are no longer required to draw minimum sample sizes of 1,000. • Respondents must be told responding to the survey is voluntary and the information will be kept confidential. • States must create a process for creating an up-to-date customer list, capturing information on each customer’s address and telephone number.

9. Revisions to TEGL No. 6-00 Surveys to Obtain State Level Results… • Disregard earlier guidance that stated the sampling frame only included participants and employers eligible for the surveys who had valid contact information. • A survey is complete when valid answers are provided by the respondents for each of the core ACSI questions. Valid answers are responses 1 through 10. • The calculation of the ACSI score includes only the results from complete surveys. • The introductory statements for both the participant survey and the employer survey were revised to better set the context for the interviews.

10. Revisions to TEGL No. 6-00 Surveys to Obtain State Level Results: Additions to Ensure Adequate Response Rates • Local program staff must inform the customer during the registration process about the importance of satisfying customers and the possibility of being contacted for information on his or her experience with the services. • The interview should be limited to 15 minutes or less. • A minimum of five follow-up attempts is required, involving various times of the day before closing the record.

11. Revisions to TEGL No. 6-00 Surveys to Obtain State Level Results: Additions to Ensure Adequate Response Rates • Local programs should collect alternate contact information. • A letter in advance of the survey should be sent out informing the customer that he or she can expect to be contacted about his or her satisfaction with the services.

12. Creating Questions to Supplement the ACSI Customer Satisfaction Not the Same as Customer Service • Customer service is measured by standards set by you or your agency. • Customer satisfaction is measured by the customer’s standards for the services, regardless of whether they make sense.

13. Convenience Accessibility Ease of use Timeliness Safety Reliability Accuracy Thoroughness Fairness Appropriateness Attractiveness and cleanliness Courteous Professional Attentive Friendly Helpful Knowledgeable Prompt Informative Honest Candid Drivers of Satisfaction Customer Service Characteristic Service Quality Measures

14. Customers define quality Variety of services Customization of services Convenience of services Timeliness of services Continually identify factors that influence satisfaction Continually identify expectations and set customer service standards well above these expectations (to “delight” customers) Characteristics of Firms With High Customer Satisfaction

15. Remember your survey's purpose: to provide you feedback on the level of satisfaction. If you have doubts about a question, discard it. State your questions simply. Stay focused when writing questions; avoid vague areas. If a question can be misinterpreted by the respondent, chances are that it will be. Include only one subject per question. Avoid questions that lead respondents. Consider optional ways to ask questions that deal with sensitive areas. Writing Effective Questions

16. Create questionnaire items to determine the key drivers of satisfaction… Writing Effective Questions

17. Answer options need to be mutually exclusive and exhaustive. Keep open‑ended questions to a minimum. Respondents interpret time-oriented responses differently, such as “never,” “sometimes,” and “always.” Consider a "don't know" responses when you create answer sets. Create meaningful scales for rating services. Constructing Effective Response Items

18. The first series of questions should be easy for the respondent to understand and should capture his or her attention and interest. Start with general questions and gradually shift to more specific questions. Group questions in logical sections and sequence sections or questions in a well thought out order. Introduce each new section with a summary statement so that participants have an opportunity to adjust to the new set of questions. Ordering Questions

19. Position questions about personal or sensitive issues toward the end of the questionnaire. The order of similar items on a list can bias results. Randomly or alphabetically order items and indicate in the instructions how they are ordered, reducing the likelihood that respondents will see the first items as most important of the group. Put demographic questions at the end of the survey form. Seek to minimize the number of times the a “skip rule” is followed as confusion can result with too many skips or when skips become overly complicated. Ordering Questions

20. Fowler, F. J., Jr. Survey Research Methods (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc., 1993. Hayes, Bob E. Measuring Customer Satisfaction: Development and Use of Questionnaires. Milwaukee: ASQC Quality Press,1993. Payne, S. L. The Art of Asking Questions. Princeton, NJ: Princeton University Press, 1951. Schuman, H and Presser, S. Questions and Answers in Attitude Surveys: Experiments in Question Form, Wording, and Context. New York: Academic Press,1981. Simply Better! The Voice of the Customer, Second Edition. Washington: U.S. Department of Labor, Employment and Training Administration, 1998. Sudman, S. and Bradburn, N. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco, CA: Jossey‑Bass Inc., 1982 Vavra, T. G. Improving Your Measurement of Customer Satisfaction. Milwaukee: ASQC Quality Press, 1997. Resources