Enhancing ODL Students' Success through Risk Profiling and Prediction: The Case of Unisa

1. HANLIE LIEBENBERG • Senior Specialist: Institutional Research, Unisa • PROF GEORGE SUBOTZKY • Executive Director: Information & Strategic Analysis, Unisa • DION VAN ZYL • Manager: Information Services, Unisa • Presented at: • NADEOSAConference, Johannesburg, 30 August 2011 Enhancing ODL Students' Success through Risk Profiling and Prediction: The Case of Unisa

2. Acknowledgements • The efforts of numerous DISA staff members in gathering and preparing information is acknowledged • In particular, the help and support of Robert Lightbody, admin Asst/caregiver to Prof Subotzky, was invaluable in preparing this presentation

3. Overview • Background, Key Challenges & Research Problem • Unisa Student Success Framework • Unisa Conceptual & Predictive Models of Success • Unisa Tracking System • Data Analysis Challenges • Segmented Profiling: Categorising Student Risk

4. Background • Whilst various theoretical models contribute towards understanding the various dimensions impacting on student success, utilising actionable intelligence to inform effective interventions remains daunting • This challenge is particularly formidable at Unisa, which now has +340 000 mainly non-traditional, older, part-time, underprepared students • They face challenging socio-economic circumstances, particular work-related and domestic responsibilities, which impede on student success

5. Research Problem • To address this, Unisa recently developed a student support & success framework, comprising 4 elements: • Conceptual model • Predictive model • Student support interventions • Evaluating impact • Critical challenge: moving from conceptual model of student success to profiling, tracking, assessing and predicting risks to success

6. Key Challenges • Key concerns and critical questions that arose in developing an integrated Student Support and Success Framework in the Unisa context • More particularly, the process of moving from the conceptual modelling of student success – a necessary first step – to the detailed student profiling, tracking and predictive modelling of risks upon which effective interventions are based • Key challenge: translating and operationalising relevant constructs of the high-level conceptual model to create a comprehensive student profile, tracking system and predictive model which retains sufficient complexity but remains practicable 

7. The Challenge of Translating Theory into Practice A theory that could fully explain every aspect of the attrition process would contain so many constructs that it would become unwieldy if not unmanageable. Such situations call for the use of theoretical models which are simplified versions of reality that strip away the minute details to concentrate on factors that are assumed or deduced to be important. ... Models can be judged by their usefulness. A model of the attrition process should contain sufficient constructs to explain what is undoubtedly a complex process and yet sufficiently simple to be understandable and useable. It should be able to explain collected descriptive data, and it should provide a framework against which predictions can be hazarded and judgements made about potential interventions. Kember(1989: 279-280)

8. Operationalising the Conceptual Model • This implies: • Identifying and defining all academic and non-academic variables needed for construct measurement, segmentation, profiling and predictive modelling; • Utilisingsuitable data gathering methods that yield consistent, complete and unbiased data; and • Applying appropriate advanced statistical analysis that can identify complex underlying multivariate dynamic relationships between variables and constructs

9. Elements of the Unisa Student Success Framework • Extensive literature review & conceptual modeling of all factors affecting success in Unisa context • Comprehensive profiling, tracking and intelligence gathering culminating in predictive model of student risks/success Incrementally implementing an institution-wide Student Support Framework Evaluating impact over time

10. SHAPING CONDITIONS: (predictable as well as uncertain) • Social structure, macro & meso shifts: globalisation, political economy, policy; National/local culture & climate • Personal /biographical micro shifts TRANSFORMED STUDENT IDENTITY & ATTRIBUTES: • STUDENT • IDENTITY & ATTRIBUTES: • Situated agent: SES, demographics • Capital: cultural, intellectual, emotional, attitudinal • Habitus: perceptions, dispositions, discourse, expectations • Domains: • Intra-personal • Inter-personal • Modalities: • Attribution • Locus of control • Self-efficacy • Processes: • Informed responsibility & ‘choice’ • Ontological/epistemological dev. • Managing risks/opportunities/ uncertainty: Integration, adaptation, socialisation & negotiation FIT FIT FIT FIT FIT FIT • THE STUDENT WALK: • Multiple, mutually constitutive interactions between student, institution & networks • Managing complexity/ uncertainty/ unpredictability/risks/opportunities • Institutional requirements known & mastered by student • Student known by institution through tracking, profiling & prediction Retention/Progression/Positive experience Success Choice, Admission Learning activities Course success Gradua-tion Employ-ment/ citizenship FIT FIT FIT FIT FIT FIT TRANSFORMED INSTITUTIONAL IDENTITY & ATTRIBUTES: • INSTITUTIONAL • IDENTITY & ATTRIBUTES: • Situated organisation: history, location, strategic identity, culture, demographics • Capital: cultural, intellectual, attitudinal • Habitus: perceptions, dispositions, discourse, expectations • Domains: • Academic • Operational • Social • Processes: • Informed responsibility & choice • Managing risks/opportunities: • Transformation, change management, org. learning, integration & adaptation • Modalities: • Attribution • Locus of control • Self-efficacy • SHAPING CONDITIONS: (predictable as well as he uncertain) • Social structure, macro & meso shifts: globalisation, internationalisation, political economy, technology, social demand • HE/ODL trends, policy • Institutional biography & shifts; Strategy, business model & architecture, culture & climate, politics & power relations

11. Key Constructs of the Predictive Model • Students' inter-personal attributes: • Demographics and past socio-economic status, including educational and family background and exposure to role models; • Current socio-economic status and life circumstances, measured by the constructs of time and opportunity to study and stability in life circumstances and support for study; • Students' intra-personal attributes: • Academic readiness and ability; • Metacognitive skills; • Psychological attributes and outcomes of other processes; • Institutional services, practices & culture: • The quality of academic and administrative services; • Institutional culture and practices; • Integration, engagement and transformation: • Students' effective management of their life circumstances and mitigation of risks as well as meeting learning expectations and utilising opportunities; • The institution's effective management of academic and support processes and mitigation of risks.

12. Student as Situated Agent Intra-Personal Inter-Personal Psycho-logical Attributes & Outcomes • Background: • Demographics • Past SES • Educ. Background • Family Background • Role Models • Current SES & Life Circumstances: • Time & Opportunity • Stability & Support Academic Readiness & Ability Meta-Cognitive Skills Student Walk Integration, Engagement & Transformation Success • Student’s Effective Management of: • Life Circumstances & Risks • Learning Expectations & Opportunities Course Success Formative Assessment Graduation Fit: Academic Choices & Activities Utilisation of Admin/ Support Services Fit with Institutional Culture & Practices • Satisfaction • Graduateness • Institution’s Effective Management of: • Academic & Support Processes/Risks • Student Profile/Risk & Communication Institution as Situated Agent Institutional Services, Practices & Culture Quality of Academic Services Social: Institutional Culture & Practices Quality of Admin Services

13. Senate STLSC Student Success Forum School/College TLSC Academic Department Professional Structures Admin Structures DSAR DSAA TSDL DCCAD Lecturer/Supervisor/Online Mentor/Tutors/Regions SMPPD DISA Library Library Student Support Coordinator USGS Dean of Stud. Academic Affective Admin TRACKING SYSTEM Profiling, Tracking & Predicting Risk at the level of Student/Module/Qualification/Institution Operational Processes Communication/Engagement Student Information • Application/Registration • Study Material • Assessment Management • Finance • HR • College/School/Department/Regions • E-Tutor/F2F Tutor/Online Mentor • Counsellor • Call Centre • Admin Department • Tutorial Attendance • myUnisa/Library • Student Course Evaluation • Applications/Registration • HEMIS • Assessment Performance/Scores • Academic Readiness Self-Assessment • Student Profile Survey • Student Satisfaction Survey • Exit/Tracer Surveys • ICMAs

14. Student Profile Design Challenges • Considerations were given specifically to question response formats and scaling • Initial draft survey questionnaire consisting of over 100 questions based on key constructs identified in conceptual and predictive models • Throughout design process, imperative to ensure alignment between questionnaire items, measurements and constructs • Final version comprising approximately 50 questions derived • Methodological and practical issues had considered in operationalising the instrument

15. Data Analysis Challenges • All questions were designed within demands of data • Develop single continuous scale measure for each construct that is uni-dimensional, can discriminate across full spectrum of students and is valid/reliable • Four steps in the construction of scale measures, namely: • Item selection • Examination of the empirical relationships of items • Combining of items into a scale measure; and • Validating the scale measure

16. Risk Categories: Key Element of Segmented Student Profiling • This involved distilling 3 primary student-related cluster constructs from the predictive model, namely: • Academic ability • Psychological attributes/metacognitive skills and • Life circumstances • Effective engagement with the institution (construct left out of the initial risk categorisation, as this involves complex measurement through, for example, student engagement surveys) • A good example of deriving simplified, but meaningful measurable constructs out of the complexity of the full predictive model • The challenge was to define risk categories which could be measured on appropriate scales. Three approaches were explored

17. Hypotheses • If sufficient engagement, integration & transformation is achieved, this will generate: • Greater utilisation of support services • Sufficient fit between students' choices, behaviours, transforming attributes & performance and institutional communications, practices, expectations and culture • In turn, this will generate greater success in: • Formative assessment, course success, graduation, student satisfaction and required graduate attributes

18. Risk Model

19. Reflection on process so far Challenge 1: Identifying relevant measures Step 3 Analyses, Interpretation and Reporting Challenge 3: Data analyses Step 1 Project Design • Use of different multivariate techniques • “While identifying relevant variables explaining and protecting success is the point of departure, the real challenge, in light of the complexities involved, is determining the combined effects of and relationships between different predictor variables.” (Subotzky & Prinsloo, Distance Education 32/2, 2011) Research Process • Translation of conceptual ideas into meaningful questions/variables for profiling, tracking & risk/success prediction • Defining of measurable constructs & risk/success categories • Scaling considerations • Definition of risk categories Step 2 Data Collection Quantitative Challenge 2: Methodological and operational considerations • Tracking system • Survey design (data gathering method; timing & frequency; incentives

20. Questions