Insights from customer needs and satisfaction at UCF’s FEEDS program

1. Insights from customer needs and satisfaction at UCF’s FEEDS program Results from the Voice of the Customer component of a Six Sigma methodology project. Roberto Champney1 Alain Gaumier2 Naomi Morris1 1University of Central Florida 2Optimum Consulting, Inc.

2. Objectives • Project Background and Description • Six-Sigma methodology overview • Voice of the Customer (VOC) • Results and Insights From VOC • Recommendations

3. Project Background • FEEDS- Florida Engineering Education Delivery System. • Created in 1982 by the Florida State Legislature • The system has evolved from a tape system in 1982 to a CD and web-based education option in its current form. • The University of Central Florida (UCF) is one of 12 universities in the state that participates in the program, and holds the largest enrollment, 38% (2002-2003).

4. Project Background • ASQ (American Society For Quality) Community Good Works Project. • Improve communities through the use of quality tools and technologies. • Provides skills transfer to non-traditional quality practitioners. • Project Goals • Document the fall 2003 version of the FEEDS system. • Identify Priority Process Improvements.

5. Six-Sigma methodology overviewDMAIC

6. Capability Long-Term Yield Standard 3 Sigma 93.32% Historical Standard 4 Sigma 99.38% Current Standard 6 Sigma 99.99999% New Standard Classical View of Performance • Practical Meaning of “99% Good”: • 20,000 lost articles of mail per hour • 15 minutes of unsafe drinking water each day • 5,000 incorrect surgical operations per week • 2 short or long landings at most major airports each day • 200,000 wrong drug prescriptions each year • No electricity for almost 7 hours each month

7. Voice of the Customer (VOC) • Technically there are numerous ways to enhance the system, yet there has been little attempt to focus this enhancement to represent the needs of its users. • Identify customers and their needs. • Collect and analyze reactive data. • Consider proactive approaches. • Convert collected data into customer needs. • Sort out the most important attributes “ctq”. • Obtain specifications for the critical-to-quality characteristics.

8. Voice of the Customer (VOC) • Conducted a student focus group. • Conducted several interviews with FEEDS students to: • Capture major needs categories and issues. • Further understand FEEDS users. • Developed a survey concerning the satisfaction and needs of FEEDS student users. • End of Semester FEEDS Student Online Survey. • Set adequate performance levels. • Correlations to identify key drivers. • Align opportunities for improvement with FEEDS UCF priorities.

9. The Surveys • Two surveys over a two semester period • Interview comments were used to create the survey questions: • 30 and 39 questions • Some demographics. • gender, study program, program level. • Three types of questions (5-point likert scale and open ended). • Satisfaction with current attributes of the system. • Usefulness and Importance of particular attributes of the system (hypothetical and current). • Comments on particular issues with the system.

10. The Survey (Sample Questions) • Overall how satisfied are you with the FEEDS system?    • Overall how effective are FEEDS courses compared to live courses? • How satisfied are you with the quality of the video stream (vision and audio) on your FEEDS courses? • Audio • Image • How satisfied are you with the communication with FEEDS professors?

11. The Survey (Validating Sample) • Response Rate: 338 (1st survey) and (149 2nd Survey) • Mass email, and site banner. • Percentages represent actual FEEDS student body distribution

12. Insights From VOC

13. Insights From VOCOverall Satisfaction Comments69% and 54% of respondents gave comments • First Survey • 30 % positive comments • 41 % negative comments • Second Survey • 39% positive comments • 19% negative comments Note: one respondent may have contributed to more than one category of comments.

14. Insights From VOC • On Spring ’04 UCF FEEDS upgraded their stream format.

15. Insights From VOC • Prioritize future efforts using differences between ‘received’ and ‘desired’ • See SERVQUAL survey method. • Delivery of Service Quality," Journal of Marketing, April 1988, pp. 35-48. Parasuraman, Zeithaml and Berry, "SERVQUAL: A Multiple-Item Scale for Measuring Customer Perceptions of Service Quality," Journal of Retailing, Spring 1988, pp. 12-40

16. Insights From VOC • How often are you able to attend a live class? • 29% to 39% always (4 & 5 scores) • For what reasons are you not able to ALWAYS attend a live class?

17. Insights From VOC • Please indicate all the mediums with which you communicate with FEEDS professors? (Check the two most frequent)

18. Insights From VOC • How long does it takes on average to get a response from your professor? • How satisfied are you with how long it takes to get a response from your professor? • 55 % satisfied (4&5 score)

19. Insights From VOC In the event that you could have access to the class notes before the class session, what would be the best way to gain access to them? Where do you regularly watch the FEEDS stream or CD? (Check the two most frequent) • Home 126 • Work 47 • Library 17 • Computer Lab 18 • Other 8 (Beach, Starbucks, Outdoors)

20. Insights From VOC • How often have you experienced missing material from a FEEDS stream? • In the event that you have experienced missing material from a FEEDS stream, what part was missing?

21. VOC Instrument Recommendations • ~5 interviews will identify most of the problems. • Don’t get carried away with questions, you’ll have to make sense of them later. • Everything is important or useful. • Be careful with these types of questions. • Ex. Teacher assistant meetings: • Usefulness score: 4.07 • Attend score: 2.79

22. VOC Instrument Recommendations • Open ended comments are most valuable, but very time consuming. • Use at least two persons for categorization. • Provide enough space for comments. • 255 characters is not enough • Use online surveys, don’t send attachment surveys

23. Insight Summary • Identify Student Strata • 100% remote students. • Miss student interaction. • Miss out on after class talk. • Value recording of ‘pre-class’ and ‘break’ periods • Convenience students • Register for FEEDS but attend live classes. • Use FEEDS as backup or study aid. • Gender differences • Females more than males value the interaction with other students and professors (per comments). • Study Program differences • Teacher Assistant meetings, and problem solving groups.

24. Insight Summary • User satisfaction and system effectiveness are correlated. • Effectiveness is driving satisfaction. • Communication with professors is critical to satisfaction and effectiveness. • ~1/3 of students regularly attend live classes • Email and in-person communication is used mostly for communication between FEEDS students and professors.

25. Insight Summary • Provide access to class notes before class, when possible. • Consider scanning professor hand notes if possible. • Chalk/White Board is not appropriate for this medium. • Live students should see the same image as Remote students. • They’ll become the ‘directors’ of the class production.

26. Insight Summary • Standardize equipment and training. • Encourage (enforce) professor training. • Consider standardizing method of delivery and procedures (i.e. all use WebCT, all email assignments). • Use one location for files, content, or materials access. • Need commitment from professors.

27. Insight Summary • Computer demos are particularly difficult to follow (unable to read buttons and options). • Most of students’ problems with the ‘stream’ are with accessing it due to technical difficulties. • Provide content online, consider avoiding mailings (they consume resources).

28. Insight Summary • Other online services or university sections confused as part of FEEDS (i.e. WebCT). • Apply VOC methodology with other stakeholders • Professors, Administrators, etc.

29. Questions ?