Using intelligent tutors to enhance student learning of application programming interfaces

1. Using intelligent tutorsto enhance student learningof application programming interfaces Aniket Dahotre, Vasanth Krishnamoorthy, Matt Corley, Chris Scaffidi School of Electrical Engineering and Computer Science Oregon State University

2. Application Programming Interface (API) “The most serious problem facing instructors who try to teach Java—or any modern industrial-strength language for that matter—is the problem of scale. While such languages may themselves be reasonable simple, writing any useful programs requires the use of classes supplied as application programmer interfaces (APIs) along with the language. For modern languages, such API collections are vast. The existence of these huge libraries makes it difficult for students and teachers to learn the language without suffering from conceptual overload.” --ACM Java Task Force, 2004 Introduction Approach  Evaluation  Conclusion

3. Examples(where API here = task-focused collection of methods) Introduction Approach  Evaluation  Conclusion

4. Particularly challenging in project-based learning curricula N students N / x different projects y * N / x different APIs 1 professor + 2 teaching assistants y * N / x >> 3 Students end up struggling with many APIs on their own. http://www.flickr.com/photos/kchichester/3388004334/ Introduction Approach  Evaluation  Conclusion

5. Related Work • Empirical studies • Novice programmers search for code examples when they struggle with using an API • One survey: 40% of programmers complained about lack of examples; 28% about understanding documentation [Robillard 2010] • Novel tools • Can search for code examples, copy them into the programmer’s program, and help to customize them • Programmers sometimes don’t understand code they copy Introduction Approach  Evaluation  Conclusion

6. Approach • Let the professors continue to focus on teaching students to use common APIs • Most commonly-used APIs • APIs most closely related to the course’s core concepts • Provide intelligent tutoring systems to help students learn the rest of the “specialized” APIs • Delegate creation of intelligent API tutors to TAs • Provide a tool so TAs they use existing code examples • Store intelligent API tutors on website, so they can accumulate over multiple terms Introduction  Approach  Evaluation  Conclusion

7. Current paper • How to create intelligent API tutors? • Can intelligent API tutors match textbook instruction? Introduction  Approach  Evaluation  Conclusion

8. How to create intelligent API tutors? 2. ApiTutor uses keywords to retrieve web pages about topic Search engines Teacher tool 1. TA enters topic keywords Repository 3. ApiTutor identifies, parses, compares, and filters code examples Teaching assistant 4. ApiTutor presents examples to TA for further filtering, organizing, and annotating 5. ApiTutor publishes intelligent tutor to repository website Introduction  Approach  Evaluation  Conclusion

9. 1. TA enters topic keywords Introduction  Approach  Evaluation  Conclusion

10. 2. ApiTutor uses keywords to retrieve web pages about topic Bing Yahoo Website Teacher tool … <code> try { Class.forName(”sun.jdbc.odbc.JdbcOdbcDriver”); //Or any other driver } catch(Exception x){ System.out.println( “Unable to load the driver class!” ); } </code> … Introduction  Approach  Evaluation  Conclusion

11. 3. ApiTutor identifies, parses, compares, and filters code examples • Extract candidate code snippet from <code> or <pre> tag • Parse snippet into abstract syntax tree • Represent each snippet as TFIDF vector • Vector entries correspond to names of packages imported, names of classes referenced, names of methods invoked, and list of classes for variables instantiated • Compute similarity matrix (all pairwise dot products) • Select largest elements of matrix Introduction  Approach  Evaluation  Conclusion

12. 4. ApiTutor presents examples to TA for filtering, organizing, and annotating Introduction  Approach  Evaluation  Conclusion

13. 5. ApiTutor publishes intelligent tutor to repository website Student chooses a topic from the list Hint messages stored by teacher Method term selected by teacher Student interacts with the adaptive quizzes and examples Introduction  Approach  Evaluation  Conclusion 13

14. Can intelligent API tutors match textbook instruction? • Laboratory study • 10 computer science students • Undergraduate or master’s • Two randomly-selected tasks per participant • Five possibilities: StringBuffer, FileWriter, HashMap, XML DocumentBuilder, and JDBC Connection classes • Structure of each task • Part 1: learn about API from intelligent tutor or textbook (randomly selected) • Part 2: three-question fill-in-the-blank quiz Introduction  Approach  Evaluation  Conclusion

15. Results Introduction  Approach  Evaluation  Conclusion

16. Results • Average scores • 45% with textbook • 93% with intelligent API tutors • 2-Factor ANOVA • Textbook vs intelligent API tutor: significant at P<0.01 • Among the 5 API tasks: significant at P<0.01 • No interaction at P<0.10 (intelligent API tutor roughly equally effective across tasks) Introduction  Approach  Evaluation  Conclusion

17. Results • Average scores • 45% with textbook • 93% with intelligent API tutors • Significant at P<0.01 • Time that students chose to study before quiz • 15 minutes with textbook • 10 minutes with intelligent API tutors  Better scores in less time Introduction  Approach  Evaluation  Conclusion

18. Conclusions… and more questions • How to create intelligent API tutors? • Mining examples from the web seems to work well • Can intelligent API tutors match textbook instruction? • Yes—in fact, intelligent API tutors outperformed textbook • … sparking more questions… • Will TAs be able to create intelligent API tutors this way? • And will their intelligent API tutors perform so well? • And will the approach work for a broad range of APIs? • And will it actually lead to learning if used in practice? Introduction  Approach  Evaluation  Conclusion

19. What next? • Generalize to support other APIs and languages • Test, test, test • Have TAs create the intelligent tutors • For dozens of APIs • Deploy and evaluate • In the classroom • In distance learning • In on-the-job training Introduction  Approach  Evaluation  Conclusion

20. Questions? Ultimate goal: Enabling students to learn specialized APIs on demand http://www.flickr.com/photos/robertcupisz/2540742831/ Introduction  Approach  Evaluation  Conclusion

21. Protocol details • 15 minutes to read the API’s training materials • Allowed to move on to the quiz earlier if desired • Told students to focus on class and method names • Provided at the start of the experiment with an example of a quiz so they would be prepared for fill-in-the-blank • No human tutoring at all • Computer vs textbook Introduction  Approach  Evaluation  Conclusion