Implementation and Evaluation of a Second Language Acquisition-based Programming Course

2. Implementation and Evaluation of a Second Language Acquisition-based Programming Course Christina Frederick, Lulu Sun, Caroline Liron, Matthew Verleger, Rachel Cunningham and Paula SanJuanEspejo Embry-Riddle Aeronautical University

3. Overview • The Project • Background and Method • Results of the First Year Evaluation • Discussion

4. The Project • Applying Second Language Acquisition to Facilitate a Blended Learning of Programming Languages • National Science Foundation, Division of Engineering Education and Centers, grant number EEC 1441825

5. Project Elements • Use SLA techniques in a blended learning class to teach introductory programming using MATLAB. • Techniques include: • Use of visuals and pictures at lower fluency levels • Facilitated discussion boards • Think, Pair, Share exercises • Self-paced videos that provide information in smaller chunks, including vocabulary • Embedded self-quizzes in video allowing for practice and playback

6. Background • VanRoy (2003) and others (Solomon, 2004; Wynn, 2015)have compared programming language learning to regular language learning • Shared elements include: vocabulary, syntax, punctuation and sometimes unique alphabets • As well, learners go through stages of fluency before achieving true proficiency • Some K-12 school districts are moving toward counting programming instruction as fulfilling a language requirement.

7. Importance • Can we use SLA techniques that have been successful in language learning and apply them to the teaching of a programming language? • Only 2% of high school students learn programming, but it has been identified as a critical skill (Partovi, 2013) • The current project hopes to develop and test materials that facilitate programming language acquisition at the college level in the hopes of exporting best practices to the K12 environment

8. Method • Implemented SLA-based techniques in 6 blended learning, introduction to programming classes • Fall 2015, Spring 2016 • Comparison group was 9 sections of same class taught in normal blended format • 3 instructors, all of whom taught at least 1 SLA and 1 non-SLA section each semester • Language used was MATLAB • 380 Students (1/3 in SLA-based class, 2/3 in non-SLA)

9. Measures to Assess Quality/Effectiveness of SLA Techniques • Student Engagement: Measured perceptions of enjoyment, competence, importance, pressure-tension and usefulness • Used the Intrinsic Motivation Inventory (McCauley et al., 1987) • Administered at Week 1 and at the end of the course in both sections • As well it was given after students were taught data types, input/output, conditional statements and loops • For these topics, SLA students used supplemental videos designed specifically for them

10. Measures • TLX (NASA, Hart & Staveland, 1988) • Measures cognitive, physical, temporal, performance demands, as well as perceived effort and frustration • Given at same points of time as IMI • End of Course Evaluations • Instructor ratings of course content, organization of class, student learning and instructor interaction • Also for Spring 2016, SLA students answered extra questions about their experience

11. Measures • Grades • We compared grades on homeworks, projects, exams and final grade across SLA and Non-SLA sections • Use of LMS and Video Content • Examined how often students utilized course materials and for how long

12. Results: What have we found so far? • Significant Differences in Course Engagement • In week 1, SLA > N-SLA in Effort/Importance of class • After Data Types Video (week 2) SLA > N-SLA on competence and usefulness and SLA < N-SLA on pressure/tension • After I/O video, SLA > N-SLA in enjoyment and competence • After Loops video, SLA > N-SLA on competence • No end of course differences were evidenced • Workload Differences • In week 1, SLA > N-SLA in physical demand, temporal demand and effort • In week 2 after data types, SLA < N-SLA on mental demand and frustration • No End of Course differences were evidenced

13. Engagement

14. End of Course Evaluations • No differences in course evaluation outcomes by instructor • This allowed us to examine SLA/Non-SLA class differences, without controlling for instructor • There were no significant mean differences between SLA and Non-SLA classes on course evaluation items

15. Course Evaluation Items SLA SectionsNon-SLA Sections Mean N STD Mean N STD Instructor Evaluation Items Overall course clarity 3.4240 5 .11675 3.4663 8 .15399 Overall content, structure and organization of class 3.2600 5 .08337 3.2962 8 .15784 Overall learning outcomes 3.4440 5 .11866 3.4613 8 .10869 Overall student instructor interaction 3.5460 5 .11887 3.5975 8 .11273 Student Perception Items I knew course was hybrid 4.3100 3 .22539 4.3300 5 .17161 I knew what was meant by a hybrid course 4.0633 3 .33020 4.0160 5 .31405 Online activities helped me learn 3.7600 3 .29547 3.7380 5 .33722 Time spent compared to other classes 4.4167 3 .04163 4.4220 5 .19741 I would re-enroll in a hybrid course 3.3400 3 .34395 3.5300 5 .41719 The video modules helped my learning 3.1700 3 .15716 3.0557 7 .33510 The instructional approach used (SLA vs. Non-SLA) helped my learning 3.1367 3 .23671 3.0057 7 .28124 I found the teaching techniques engaging 2.6600 3 .19157 2.7943 7 .34684 I would take another class that used the same techniques 2.6233 3 .07572 2.6414 7 .27667 All items used a 5 point scale with 1=strongly disagree to 5=strongly agree Note: Some items were only asked in SLA sections, or only asked during one semester, thus the difference in ‘N’

16. Student Perception Items (Cont’d) MEAN N STD The Think/Pair/Share format helped my learning 3.0633 3 .16258 The online discussion board helped my learning 2.3067 3 .21548 The program writing problem in the quizzes helped test my understanding of the material 2.9500 3 .20075 The SLA format provided a simple and easy to understand environment 2.8733 3 .01155 The comments provided after each online quiz question helped me understand the material 3.2233 3 .02887 All items used a 5 point scale with 1=strongly disagree to 5=strongly agree Note: Some items were only asked in SLA sections, or only asked during one semester, thus the difference in ‘N’

17. Course Grades Overall Lab Scores Hypothesis: SLA Students will score higher than non-SLA students Results: F (1,328) = 2.282, p=.07, NS Note: While these differences may not be significant, to students they may be important! A 78 is a ‘C’ while an 82 is a ‘B’!

18. Course Grades Overall Exam Scores Hypothesis: SLA Students will score higher than non-SLA students Results: F (1,328) = 1.837, p=.09, NS Note: Again, while not significant, at many institutions this difference is the distinction between a C- and a C grade.

19. Course Grades Overall Project Scores Hypothesis: SLA Students will score higher than non-SLA students Results: Mann-Whitney U = 12141.5, p=.33, ns

20. Course Grades Final Grade Hypothesis: SLA Students will score higher than non-SLA students Results: Mann-Whitney U = 127467.5, p=.80, NS

21. Summary • Year 1 of the Project is complete • Year 2 will begin in fall 2016 • One goal is to enhance response rates for surveys (highest response was 113 students of 378, 30%) • At this time, results are inconclusive about overall effectiveness of the SLA-based teaching approach, however when differences occurred, they did favor the SLA taught class sections • It may be that year 2 will show more differences, as instructors will have had a year of familiarity and practice with techniques

22. Thank You! Special Thanks to Austin Yazel who assisted with data analysis