Evaluation and effectiveness of ALA interventions: A systematic review

1. Evaluation and effectiveness of ALA interventions: A systematic review Joanne Smith Psychology University of Exeter j.r.smith@exeter.ac.uk

2. Background Title to go here • At present, ~20 million university students in the EU • But ~7 million will never complete their degree • Similar figures for the US (39% non-completers) • But certain sub-groups are over-represented • E.g., mature age students, BME students • And universities vary in their retention rates • E.g., some record drop-out rates as high as 43% • Thus, room & opportunity to improve retention and attainment via active intervention Body copy to go here

3. Background Title to go here • One way to do this is by using learning analytics (LA) • LA integrates various types of data, statistical analysis, and predictive modelling to inform intervention in the way students learn, teachers teach, and institutions design curricula (Williams, 2014) • Core premise: use LA to identify students at relative risk of failing or withdrawing  make students + teachers aware of risk  intervention  resolve problem Body copy to go here

4. Background Title to go here • Despite increasing popularity + investment in LA programs, few empirically tested and evaluated • Does the evidence base support investment? • RESEARCH AIM: To systematically review the evidence on the efficacy of LA predictor variables + effectiveness of LA interventions in terms of retention and attainment Body copy to go here

5. Method Title to go here • Literature search strategy: • Comprehensive search of 13 EBSCO-host databases + Google Scholar using a number of search terms • Quality of research appraised with Quality Assessment Tool for Quantitative Studies (QAQTS) • Studies categorized as “strong”, “moderate” or “weak” depending on: • Participant selection method • Study design • Confounding variables • Researcher blinding • Data collection methods • Participant withdrawals and drop-outs Body copy to go here

6. Title to go here Inclusion Criteria • Article reported empirical studies testing predictor value of LA initiatives, algorithms, +/or models • Article reported effectiveness of LA initiatives in terms of retention, achievement +/or overall student success • The full text was available • The article was in English • The article had undergone scientific peer review • The article had been published since 2000 Body copy to go here

7. “Traditional” LA models Title to go here • Early LA models relied on fixed factors to generate single set of predictions in a specific timeframe • E.g., high school GPA, SES • 9 studies reviewed • Considerable evidence base supporting value of demographic and academic history variables in predicting retention, graduation, and test performance • But quality of evidence only moderate Body copy to go here

8. Title to go here Body copy to go here

9. Title to go here Body copy to go here

10. “Dynamic” LA models Title to go here • “traditional” models limited re ongoing assessments of student risk factors • i.e., models unable to incorporate more fine-grained and shifting info in their predictions • More recent research has focused more on dynamic models that make predictions based on fluid information • E.g., performance, course engagement over time • 10 studies reviewed Body copy to go here

11. Title to go here Body copy to go here

12. Title to go here Body copy to go here

13. “Dynamic” LA models Title to go here • Studies demonstrate predictive value of social and demographic and also academic history factors… • but also emphasise importance of using student activity and engagement in VLE • Important to consider how VLE activity is measured • i.e., needs to be more than just number of times accessed • Overall, stronger research quality (90% mod or strong) Body copy to go here

14. LA interventions Title to go here • Emerging evidence base supports the predictive power of LA but can this knowledge be used in interventions • Only identified 4 studies that evaluated the effectiveness of LA interventions • Basic assumption: identify at-risk students & analyse individual risk factors  disseminate info increase awareness of learning issues intervention • Some studies also added practical advice +/or offer of academic support Body copy to go here

15. Title to go here Body copy to go here

16. LA interventions Title to go here • (small) evidence base highlights promise of interventions based on LA for academic performance • All 4 studies reported post-intervention increases in grades +/or course activity • However, results on student withdrawal rates varied • 1 study found increased withdrawal • But might indicate strategic behavior • Overall, quality of evaluation research needs to be improved Body copy to go here

17. Conclusions Title to go here • Three key factors in predicting student “success” • Social/demographic factors • Academic history • Engagement, usually on a VLE • However, overall, evidence to support LA is limited due to small number of studies and issues of research quality • Reluctance to publish? • Generalisability? • Underlying mechanisms? • Translating good predictive models into concrete interventions? • Who should change? Body copy to go here

18. Recommendations Title to go here studies should employ experimental research methods (e.g., RCTs) over correlation and cohort pre-post designs outcomes should be broken down by population characteristics to allow for moderators/mediators to be identified staff and student experience of initiatives should be studied interventions should target student behaviour/activity as well as the educational environment standardised assessments of intervention programs should be developed generalisability of LA interventions should be assessed greater transparency via publication is needed Body copy to go here

19. Title to go here Body copy to go here Questions Sonderlund, A. L., & Smith, J. R. (2017). Evaluation and effectiveness of ALA interventions: A systematic review. Manuscript submitted for publication.