Video Enhanced VR for Teaching Restaurant Skills to Children with Autism

1. Video Enhanced VR for Teaching Restaurant Skills to Children with Autism

2. AbstractWhile Virtual reality has been used for various training applications, few studies have measured its effectiveness in teaching social interactions. This research usedweb delivered gamingtechnology to createvirtual worlds where children with Autism Spectrum Disorder(ASD)interactedwith avatars to practicerestaurant social skills.Videos of similar real world situationswithin the virtual sequencesreinforced the lessons and aided in generalization, a known problem for individuals with ASD. Before training, two unknown restaurant social skills were identified for each of five children with ASD, aged 7 to 16.All children who completed VR training correctly performed two new social skills in a virtual restaurant. In post-training real world restaurants, these children exhibited at least one new appropriate social interaction and 75% demonstrated 2 new social skills.

3. Learning Steps for Autism • Define a common social skilldeficit-eating in a restaurant • Divide social skill into discreet steps(1) Wait to be seated, (2) Answer hostess questions, (3) Follow hostess to table, (4) Wait for server and read menu, (4) Order food, (5) Table conversation • Define measurableactions in each stepEx: Step 1 includes determining if there is a hostess or you seat yourself, waiting in the appropriate place, recognizing the hostess, and waiting for others already in line • Identify meaning behind each step - Theory of Mind • Show a range of appropriate responses for each step • Use non-threatening practice of steps with avatars in VR • Display video of real people in same step within VR lesson • Reward or correct at each step based on child’s actions • Vary scene/avatar actions to teach variations • Allow user control of steps for different learning abilities • Vary avatar responses to avoid patterns and teach concepts

4. Program Design • Text with video explanation of each stepWhat to do (Question/How to Respond) Explain the social action or exchangeWhy? -Explain the meaning behind the actionsSee it! -See a video of real people doing the stepOops… What to do if something goes wrongShow possible variations of ‘What to do’ • Virtual world practice in pizza restaurantThree windows on screen: Interactive VR world with avatars Question/answer and video/correction window Displays responses child can pick If child picks correct response, shows video If child picks wrong response, corrects/repeats step What to do/Why? Guidance • Multi-player practice with other people over web

5. Program Details • Player moves in virtual worlds with mouse or keyboard directional keys • Avatar program tracks and verbally responds to different actions of the player • Avatars in virtual world include player, a friend the player is eating with, a hostess, a waitress, and other patrons at different tables • Visual cues. Example: Colors visually separate window functions; Error/reward statements in red, questions with choices in blue, inactive window in black • Videos at each step are repeatable with button control • Avatar spoken words have both text displayed at window bottom and audio • Print out of step information available for guidance if child prefers hard copy • Text with video explanation program was developed with Flash • VR practice world was created and delivered over web with Wild Tangent Platform • Multi-player VR practice was created and delivered over web with Adobe Atmosphere • VR actions are controlled with JAVA programs • Graphics used Wild Tangent world creation toolkit, Atmosphere world creation toolkit, 3d StudioMax, Poser, and Photoshop • Video production was done in a real a restaurant during lunch hour. Waitress, hostess, and friend were scripted actors. The regular restaurant clientele was part of the filming background. Fourteen action and seventeen avatar phrase variations were used • Video processing was done with Premier

6. Study Design • Six children ages 7 to 16 with ASD chosenAll in special education programs and recommended by therapists.Child 01(Age 11, Male, Asian)-Child 02(Age 7, Male, White)-Child 03(Age 9, Female, White)-Child 04(Age 16, Male, Black)-Child 05(Age 9, Male, White)-Child 06(12, Female, Black). • Pre-training testing in real restaurants on six skill stepsIndividual testing was done in multiple lunch or dinner visits. Restaurants varied and depended on parent recommendation. Interactions were not scripted. Actions and latency were recorded by 2 observers. Correct response required that 90% of all parts of a step be done correctly.Child had to display inappropriate behavior at more than chance level for 2 or more steps to be included in study. • Five children practiced correct social actions on computerTraining was approximately one hour, two to three times over a 2 week period. Training first involved a treasure hunt game to practice navigation in virtual space. All children mastered control in no more than two plays. Each child then completed the Text, VR Pizza world, and web training programs. After training, each child assessed his knowledge with a written questionnaire. All children came back approximately one week later and repeated the three training programs and questionnaire. • Post-training testing in real restaurants on six skillsPre-training testing was repeated in the same restaurants and data collected as before. Appropriate skill responses varied between visits because hostess, waitress, patrons, and other details varied, creating different required social interactions. Two restaurant visits were done per child with an average of 3-day separation between visits.

10. Conclusions • 4 children completed studyChild 03 knew all skills in pre-tests. Child 06 was removed because of family conflicts • All learned 2 new skills in virtual worlds at 100% accuracyFour subjects mastered all 6 skill steps to 100% accuracy on all computer training • All did 1 new skill in real world with 90% accuracy75% did 2 new skills in real world with 90% accuracySkill retention was strongest when the restaurant visit was close in time frame to computer practice. Child 05 did a third computer training between restaurant visits after missing Step 1 in first post-training and subsequently did all skills correctly • Mixed results for one child who generalized only 1 skillChild 04 was severely autistic (CARS 34) and missed all skills in pre-training. In first post-training, he demonstrated 100% accuracy in 5 of 6 steps. In second restaurant visit, music and the unexpected presence of teacher at next table distracted child. Computer refresher and retest could not be completed before child moved to another state • ProblemsReal world actions did not always match discreet step divisions. Example: child might order correctly but stare at ceiling rather than waitress (was given 50% accuracy). Observers required that each action be completely acceptable by normal social standards • Graphs of results and details of pilot study available from do2learn@do2learn.com