1 / 45

Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course

Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course. John Whitmer, Ed.D. & Kathy Fernandes Academic Technology Services California State University, Office of the Chancellor CATS 2013, Sonoma State University February 19, 2013. Outline. Context

Download Presentation

Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Using Learner Analytics to Understand Student Achievement in a Large Enrollment Hybrid Course John Whitmer, Ed.D. & Kathy FernandesAcademic Technology Services California State University, Office of the Chancellor CATS 2013, Sonoma State University February 19, 2013

  2. Outline • Context • Learning Analytics, Methods & Tools • Findings • Conclusions & Next Steps • Becoming a Data Scientist (pending time & interest)

  3. Why are you here? • To increase my knowledge of “Learning Analytics” • To get ideas about how I can become a “Data Scientist” (or practice analytics) • To learn about the results of this study • Because this was the closest room

  4. 1. Context

  5. Case Study: Intro to Religious Studies • Undergraduate, introductory, high demand • Redesigned to hybrid delivery format through “academy eLearning program” • Enrollment: 373 students (54% increase on largest section) • Highest LMS (Vista) usage entire campus Fall 2010 (>250k hits) • Bimodal outcomes: • 10% increase on final exam • 7% & 11% increase in DWF • Why? Can’t tell with aggregated data 54 F’s

  6. Guiding Questions • How is student LMS use related to academic achievement in a single course section? • How does that finding compare to the relationship of achievement with traditional student characteristic variables? • How are these relationships different for “at-risk” students (URM & Pell-eligible)? • What data sources, variables and methods are most useful to answer these questions?

  7. It takes a village … • Chico “Graduation Initiative” endorsement • Former CIO (Bill Post included in project portfolio) • Registrar approval • Active participation: • Kathy Fernandes, Director Academic Technology • William A. Allen, Director of Institutional Research • Scott Kodai, Manager Distributed Learning Technologies and Classroom Services

  8. Predict the trend: Grade w/Mean LMS Hits What will the trend look like?

  9. Strong Trend: Grade w/Mean LMS Hits

  10. 2. Learner analytics, Methods & TOols

  11. Learner Analytics “ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)

  12. How? Knowledge & Skills Involved http://www.drewconway.com/zia/?p=2378

  13. Data Extracts & Sources • LMS Data • SQL queries on WebCT Vista web server log file • Student Data • Query against Peoplesoft data • ERSS data elements (common to CSU) • Required Institutional Research • Course Data • Query against Peoplesoft data

  14. Tools Used

  15. Methods at a Glance • Clean/filter/transform/reduce data (70% effort) • Descriptive / exploratory analysis (20% effort) • Statistical analysis (10% effort) • Factor analysis • Correlation single variables • Regression multiple variables; partial & complete

  16. Final data set: 72,000 records (-73%)

  17. Variables

  18. 3. Findings

  19. Correlation: Student Char. w/Final Grade Scatterplot of HS GPA vs. Course Grade

  20. LMS Use Variables Student Characteristic Variables Enrollment Status First in Family to Attend College Gender HS GPA Major-College Pell Eligible URM and Pell-Eligibility Interaction Under-Represented Minority URM and Gender Interaction • Administrative Activities (calendar, announcements) • Assessment Activities (quiz, homework, assignments, grade center) • Content Activities(web hits, PDF, content pages) • Engagement Activities (discussion, mail)

  21. Predict the trend • LMS use and final grade is _______ compared to student characteristics and final grade: • 50% smaller • 25% smaller • the same • 200% larger • 400% larger

  22. Predict the trend • LMS use and final grade is _______ compared to student characteristics and final grade: • 50% smaller • 25% smaller • the same • 200% larger • 400% larger

  23. Correlation LMS Use w/Final Grade Scatterplot of Assessment Activity Hits vs. Course Grade

  24. Conclusion: LMS Use Variables better Predictors than Student Characteristics LMS Use Variables18% Average(r = 0.35–0.48)Explanation of change in final grade Student Characteristic Variables 4% Average(r = -0.11–0.31) Explanation of change in final grade >

  25. SmallestLMS Use Variable(Administrative Activities) r = 0.35 Largest Student Characteristic (HS GPA) r = 0.31 >

  26. Combined Variables Regression Final Grade by LMS Use & Student Characteristic Variables LMS Use Variables25% (r2=0.25)Explanation of change in final grade Student Characteristic Variables +10%(r2=0.35) Explanation of change in final grade >

  27. Predict the trend • LMS use and final grade is ______ for “at-risk”* students? • 50% smaller than not at-risk students • 20% smaller than not at-risk students • No difference than not at-risk students • 20% larger than not at-risk students • 100% larger than not at-risk students *at-risk = BOTH under-represented minority and Pell-eligible

  28. Predict the trend • LMS use and final grade is ______ for “at-risk”* students? • 50% smaller than not at-risk students • 20% smaller than not at-risk students • No difference than not at-risk students • 20% larger than not at-risk students • 100% larger than not at-risk students *at-risk = BOTH under-represented minority and Pell-eligible

  29. At-Risk Students: “Over-Working Gap”

  30. Question 3 Results:Regressionby “At Risk” Population Subsamples

  31. What caused that difference?

  32. Activities by Pell and Grade Extra effort in content-related activities

  33. Previous Studies Relating LMS Use to Course Grade

  34. 4. Conclusions & Next Steps

  35. Conclusions • At the course level, LMS use better predictor of academic achievement than student demographics (what do, not who are). • Small strength magnitude of complete model demonstrates relevance of data, but suggests that better methods could produce stronger results. • LMS data requires extensive filtering to be useful; student variables need pre-screening for missing data.

  36. More Conclusions • LMS use frequency is a proxy for effort. Not a very complex indicator. • Student demographic measures need revision for utility in Postmodern era (importance to student, more frequent sampling, etc.). • LMS effectiveness for at-risk students may be caused by non-technical barriers. Need additional research!

  37. CSU Learner Analytics Collaborative Projects Moodle: mCURL (Moodle Common Use Reporting & Learning Analytics) • 8 CSU & 2 UC Campuses • Building common SQL queries for accurate & comparative use metrics • 2 rounds of data collection already completed and discussed Blackboard: Analytics for Learn Pilot • 3 CSU Campuses • Bb Learn Analytics product available “off the shelf”; defined and integrated with Peoplesoft • Building common queries & campus-specific reporting

  38. 5. So you wanna be a data scientist?

  39. Choose your domain(s) http://www.drewconway.com/zia/?p=2378

  40. Potential Resources • Math/Stats: campus classes • MOOC resources (search “Data”, “Statistics”, or “Analytics”) • Society for Learning Analytics Research (http://www.solaresearch.org/) • Learning Analytics Resources (http://johnwhitmer.net/resources/)

  41. Feedback? Questions? Kathy Fernandes kfernandes@csuchico.edu John Whitmer jwhitmer@calstate.edu Complete monographhttp://bit.ly/15ijySP Twitter: johncwhitmer

  42. Bonus Slides

  43. Missing Data On Critical Indicators

  44. LMS Use Consistent across Categories Factor Analysis of LMS Use Categories

  45. Ideas & Feedback Potential for improved LMS analysis methods: • time series analysis • social learning • activity patterns • discourse content analysis Group students by broader identity, with unique variables: • Continuing student (Current college GPA, URM, etc. • First-time freshman (HS GPA, SAT/Act, etc)

More Related