Naj Shaik PhD shaik@ad.uiuc University of Illinois at Urbana-Champaign

1. An Alternative Theoretical Framework to Analyze Failures in Decision Making: Application to Student Dropout in Asynchronous Learning Environments Naj Shaik PhD shaik@ad.uiuc.edu University of Illinois at Urbana-Champaign Ninth Sloan-C International Conference on Asynchronous Learning Networks Nov 14-16, 2003

2. Student dropout is rarely the result of a single point of decision failure but typically occur when the system breaks down at multiple points along the chain • Adapted from James Reason. Human Error. Cambridge University Press, 1990. • Retention is everybody's business • Provost, Virginia Commonwealth University

3. Retention Rates Historical Data • Between 1880 & 1980 the rate of retention in US was around 55% (Tinto, 1982) which is higher than Australia and UK. Current Research • Traditional Students: Second year retention rate • 80% for member institutions, 87% for selective institutions and 69% for less selective institutions (Consortium for Student Retention Data Exchange 2001). • 74% for private and 72% for public 4-year institutions (ACT 2001). • Non-Traditional Students: Second year retention rate • Retention rates for distance education programs are lower than those for the traditional on-campus programs and courses (Dutton, Dutton, and Perry, 2001; Terry, 2001; Carr, 2000).

4. Significance of Retention Issue • A number of federal and state agencies request the reporting of retention data. • U.S. Department of Education plans to emphasize raising retention rates in higher education while working with Congress to reauthorize the Higher Education Act (reported in Chronicle of Higher Education, Dec 2002). • It is used as an indicator of academic quality in U.S. News and World Report annual college rankings (Graham and Morse, 1998). • A low retention rate reflect poorly on a program and impact program promotion and recruitment efforts.

5. Conventional Theoretical Models / Frameworks • Over the last 3-decades a number of frameworks have been proposed to explain the student dropout phenomena based on student behavior. • Sociological Model (SM): Spady • Student Integration Model (SIM): Tinto • Social & Personal Beliefs Model (SPBM): Fishbein & Azjen • Student Attrition Model (SAM): Bean • Integrated Retention Model (IRM): Cabrera et. al. • Congruence Model (CM): Boshier • Chain of Response Model (CRM): Cross • Identity Theory Model (ITM): Witte et al. • Composite Persistence Model (CPM): Rovai • Rational Choice Model (RCM): Manski & Wise

6. Student Integration Model (SIM) Student Integration Model is based on • Durkheim (1951) theory of suicide • Van Gennep (1960) theory of rites of passage • Predictive version of Spady sociological model. • Individual pre-entry college attributes (family background, skill and ability, prior schooling) form individual goals and commitments. • The individual’s goals and commitments interact over time with institutional experiences (the formal and informal academic and social systems). • The extent to which the individual becomes academically and socially integrated into the academic and social systems of an institution over time determines the decision to dropout. • The absence of successful integration arises due to incongruence (odds with the institution) and isolation (disconnected from the institution).

7. Enhancements to Student Integration Model • Student Attrition Model (Bean): importance of external variables such as finances, family responsibility and support etc for non-traditional students. • Congruence Model (Boshier): Influence of environmental variables such as …. • Integrated Retention Model (Cabrera et al.): an integrated framework based on the overlapping factors between SIM (Tinto) & SAM (Bean). • Identity Theory (Witte et al.): non-traditional student’s identity development and its influence on student behavior. • Composite Persistence Model (Rovai): includes learning environment, nature of teaching and learning etc to the evolving framework.

8. Synthesis: Conventional “Fit” Models • Dropout is explained through student-institution “FIT" by looking at student, institutional, and environmental variables. • Absence of successful integration is attributed to incongruence (odds with the institution) and isolation (disconnected from the institution). (Tinto) • Student Responsibility: “ … either low goal commitment or low institutional commitment can lead to dropout.” (Tinto, 1976. p.96).

9. Observations: Conventional “Fit” Models • Incongruence: “… students must disassociate … from membership in the communities of the past, [such as] …. family, the local high school, [etc.] …(Tinto, 1987, p.95)” to become fully incorporated with the college community. • Implications: Students to “commit a form of cultural suicide to be academically successful” (Tierney, 1999, p.85). • Transition: The reference to theory of rights of passage is inconsistent with anthropological basis. • Rights of passage signifies movement within the same culture and not between cultures (Tierney 1992). • Integration / Fit appeals to commonsense but hides the complexity and over simplifies the issue (Draper, 2003).

10. Observations: Conventional “Fit” Models • Empirical Support (Draper, 2003) • Research is based on weakly consistent evidence. • No comparative research testing competing frameworks. • Absence of controlled experiments. • Absence of a psychometric-based standardized instrument for data collection. • Absence of standard protocols to administer instrument. • Generalization not possible due to data constraints. • In the absence of a common metrics valid time-series and cross-program comparisons on dropout rates are difficult to interpret (National Research Council, 1996). • The list of explanatory variables is growing as we look at the number of suggested enhancements to the ‘Fit’ framework.

11. Alternative Theoretical Framework • Based on cognitive architecture and draws on the field of cognitive psychology particularly the contributions of • Rasmussen: Skills- Rules- & Knowledge Framework • Norman: Error Taxonomy • Reason: Generic Error Modeling System • The dropout phenomenon is described as attributed to divergent origins from technological failure to managerial oversight to organizational weakness. • The goal is to highlight the importance of organizational environment relative to individual performance.

12. Rasmussen’s Step Ladder Framework Interpretation Evaluation IdentificationGoal Selection ObservationProcedure Selection Problem  Activation  Execution Problem Solved • Represents constraints in the work environment. • Describes three levels of human behavior in terms of increasing order of cognitive processing. • Comprises a three level hierarchy defined by skilled-based, rule-based and knowledge-based behavior. • At the lower level actions are more specific and rigid while at the higher level actions are more general and flexible.

13. Rasmussen’s Step Ladder Framework • Skill-Based (SB) corresponds to effortless routine actions that take place as smooth, automated and highly integrated patterns of behavior (driving under normal weather conditions). • Rule-Based (RB) is applicable to tacking unfamiliar anticipated problems. Solutions to these problems are governed by stored rules which are learned by explicit training and by experience (driving on a slippery road). • Knowledge-Based (KB) is applicable to unfamiliar unanticipated problems. It requires reasoning to diagnose and solve problems, identify deep features about a situation, adapt plans and responses to the needs of the situation in the absence of pre-packaged solutions (driving during rush hours on Chicago lake shore in winter).

14. Reason: Generic Error Modeling System Error Types: Slips, Lapses, and Mistakes • Slips relate to observable actions associated with attention failures (Skill-Based behavior). • Lapses involve memory failures (Skill-Based behavior). • Mistakes associated with complex problem solving and relate to a plan that is inadequate to achieve the objective (Rule-Based or Knowledge-Based) Error Source: Latent and Active Errors • Latent errors are removed from the direct control of the front-line operators and relate to organization level decisions, communication processes, business practices and rules, deficiencies in training etc. • Active errors include operator errors at system front end. Error Chain: Swiss cheese metaphor.

15. Reason: Generic Error Modeling System Error Chain & Swiss cheese metaphor • Numerous factors behind any single decision failure. • A single failure analyzed out of context might seem benign. • Swiss Cheese metaphor • to explain the influence of latent factors on decision failures. • need to work backwards to uncover latent factors on decision failures. • often a potential error chain is broken by redundancy in the system (feedback mechanisms). • Organizational Influences (latent errors)  Unsafe Supervision (latent error)  Preconditions for unsafe acts (latent error)  Unsafe acts (active failures).

16. Reason’s “Swiss Cheese” Metaphor Organizational Influences Management Hierarchy Organization Culture Organization Climate Organizational Processes Institutional Policies Institutional Procedures Rewards Structure Nature of Inter-departmental Cooperation Line Management Deficiencies Instructional Technology Resources Human Resource Constraints Budget Constraints Business Rules Business Practices Work Environment Front-end System Users User’s Knowledge, Experience, Training, & Workload Team Work Demands of the task Latent Factors @ Blunt End Active Triggers @ Sharp End

17. Reason’s “Swiss Cheese” Metaphor Blunt End Sharp End Latent Conditions Organizational Factors Latent Conditions Latent & Active Conditions Line Management Factors Supervision & Control Triggers Individual / Team Factors Preconditions Active Conditions Active Conditions Gaps or Weakness in System Defenses System Defenses Working

18. Reason’s “Swiss Cheese” Metaphor Blunt End Sharp End Latent Conditions Organizational Factors Latent Conditions Latent & Active Conditions Line Management Factors Supervision & Control Triggers Individual / Team Factors Preconditions Active Conditions Active Conditions Failure in Decision Making Trajectory of system failure when Loopholes in the firewall of the system lineup

19. Student Dropout: Processes Inventory • Recruitment • Admissions / Registration • Orientation • Advising • Teaching & Learning Environment

20. Recruitment Process • Identification of Recruits • direct mailings, loading test score data, high school and community campus visits, and campus events. • graduate students recruitment is decentralized and may involve separate recruiting efforts. • international students • Managing potential recruits information • Assigning and Managing recruiters (includes personal interviews with the prospects) • Developing a customized communication plan

21. Latent Failures: Recruitment Process Non-standardized Protocols • Personal interview with the recruits • Nature and scope of student communication plan • Address needs of special students groups: gender, ethnicity, athletes, economic status etc. • GPA computations for prospective graduate student (which degree? undergraduate, graduate, multiple degrees) Data / information requirements (in a fragmented system) • Availability of test scores and percentiles • Data needs of recruiters, admission councilors, departments, special groups on campus. • Type and scope of tracking regarding the interventions / special programs

22. Latent Failures (Culture): Recruitment Management emphasis is on enrolment relative to retention even thought it is not a sound economic policy. A small decrease in student attrition can result in a significant gain in revenue to the institution. • A public institution with 2000 freshman enrollment and a dropout rate of 30% can save $1 million for a 10% decrease in the dropout (Levitz, Noel & Richter 1999). • 1% increase in the freshman retention will result in a $500,000 gain in additional revenue to the institution (Johnson, 1997) • Community college with an enrollment of 15,000 students, and a full-time equivalent student bringing in state funds of $5,000, a 1% decrease in retention increases the tuition revenue by $750,000 (Dr. Jing Luan, Cabrillo College)

23. Admission / Registration Process • Evaluation of prospective student application • auto evaluate based on the criteria • flag applicants that do not meet the criteria • Review by administrators (initial review) • administrative officers • college / department • Look for other factors and background information • Review Committees (liberal arts, engineering, etc) • Look for additional details • Admission based on guidelines • Hold for dean’s review • Admission decision • Number of subjective factors are involved in admission decisions

24. Latent Failures (Culture): Registration Registration procedures to mask some attrition • British Open University beginning students register on a temporary basis. If they withdraw within the three months of starting the program, their official registration will not show up on the university records (Guri-Rosenblit, 1999). Definition • Definition of a dropout varies widely among institutions. The terms retention, attrition, departure, withdrawal are sometimes used interchangeably. • Time-to-degree completion or average time to graduation is less accurate due to an increasing number of non-traditional students and the stop-out phenomena.

25. Orientation Process Agencies and Scope • University level, College level, & Department level • Special interest groups / programs • Minority Affairs Program • Athletics Program • Honor’s program • Time Duration (1-2 days) Processes • Faculty: Meet in small groups with faculty. • Peers: Interact with peers and other freshmen. • Learn about the institution’s academic standard. • Support Services: Become familiar with services. • Advisor: Meeting to map out academic strategies. • Placement: Complete placement testing (if needed ). • Pre-Registration: Select fall classes, and register.

26. Advising Process • Decentralized process and varies from college to college. • Categories of academic advisors • Academic Professional (AP) • Faculty • Graduate Student • Advising Types • New Students • Continuing Students • Formal advising relationship with a faculty or AP • Informal advising relationship • Advisor and Student responsibilities • Student to advisor ratio • UIUC college: between 7 to 60 students per advisor • Information system to support Advising

27. Latent Failures: Advising • 60% of students reported that they are not satisfied with the quality of advising received at their college (National Survey reported in Cuseo 2003; Astin 1993). • Insufficient information, getting wrong information relating to instruction, financial aid and billing were some of the negative comments (Survey by Heverly 1999). • Need for better Advising Formats & Response Time • 24 X 7 Online with 24-hour response • Face-to-face regular, evening and weekend hours • Student to advisor ratio & quality of student advising • UIUC college: between 7 to 60 students per advisor • Decentralized / Fragmented Advising System • Sharing of student data and advisor notes across campus advising agencies. • Degree Audit Reporting System (DARS) not accessible to students.

28. Latent Failures (Culture): Advising • Faculty contracts do not emphasize the importance of advising as a faculty responsibility (Teague & Grites, 1980). • Reward structure • Often blocks the ability to reward faculty who are genuinely committed to advising” (Creamer & Scott, 2000) • Only 12% of postsecondary institutions offered incentives or rewards that recognize outstanding advising of first-year students (Survey: Policy Center on the First Year of College, 2003). • Faculty promotion and tenure - advising is typically recognized by giving it only minor consideration (Habley, 1988). • Supplementary programs for (at-risk and ethnic) students groups but none for faculty.

29. Teaching & Learning Environment Who is teaching the course • Research faculty • Adjunct faculty • Graduate student Course delivery format • Asynchronous online courses (no instructor contact) • Mixed format online courses (with instructor contact) • Face-to-face format Instructional Design • Course structure & role of instructor • Assessment and feedback • Support for different learning styles • Peer interactions & instructor interactions • Virtual community support

30. Latent Failures: Learning Environment Institutional commitment to teaching • Four-year and junior colleges: teaching emphasis. • Universities: research emphasis relative to teaching. Instructional Design • Rigid course structure with pre-determined course topics. • Assessment – multiple choice items graded by TA. • Little support for different learning styles Nature of instruction • Freshman courses with large class size often taught by non-tenured faculty / graduate students. • Corresponding course taught without any instructor • Online courses with all content online and no scheduled instructor contact. • Online courses with scheduled lecture / contact hours taught by untrained professionals.

31. Latent Failures (Culture): Faculty Research Rational Choice Model • The decision to enroll is a decision to initiate an experiment, a possible outcome of which is dropout. Lowering dropout levels would not necessarily make society better off. (: Manski, 1989). • Graduation decision is part of the overall labor market optimizing problem based on an assessment of returns from graduation and the cost of persistence to the student. (DeBrock et al. 1996). • Dropout does not necessarily reflect a lack of ability on the part of the student. Dropout cannot be treated as a failure on the part of the educational system. Attrition is recognised as a perfectly normal condition within tertiary education. (Tinto1982; Bean 1980)

32. Application to Student Dropout: Structural Layer Organizational Influences Management Hierarchy Regulations: State & Federal Organizational Culture: Research Emphasis Institutional Priorities: Sports & Entertainment Rewards Structure: Emphasis on Research Enrollment Management: Focus on Recruitment Accountability: Dropout Line Management Deficiencies Information System: Decentralized and Fragmented Human Resource Constraints: Temporary Staff & Graduate TAs Budget Constraints: Quality of Advising Pedagogy: Converting traditional courses to online format Front-end System UsersInstructor Knowledge, Experience, Training, & Workload Absence of Team Work & Inter-department Cooperation Learning Environment Latent Factors @ Blunt End Active Triggers @ Sharp End

33. Application to Student Dropout Blunt End Sharp End Decentralized / Fragmented Information System Vague Procedures Latent Conditions Insufficient Tutoring Services Organizational Factors Latent Conditions Advisor (Expertise) Latent & Active Conditions Line Management Factors Supervision & Control Triggers Individual / Team Factors Preconditions Active Conditions Non-traditional Student Active Conditions State & Federal Regulations Tenured Faculty with Research Focus Institutional Culture: Recruitment Emphasis Immature Freshman Incomplete Student Plan Support Services: Social Forced Online Pedagogy (Traditional Course) Large Class Size Student Dropout Trajectory of system failure when Loopholes in the firewall of the system lineup

34. Application to Student Dropout: Process Layer Triggers Incomplete information about the Prospect Incomplete Communication Plan Staff/Recruiter Workload Institutional factors Recruitment Over-whelmed with Advising Information Admission Decentralized / fragmented Information System Orientation Immature Freshman Over-worked Athlete Advising Asynchronous Online Course Learning Environment Decentralized Information System Vague Recruitment Guidelines Instructor Experience (TA) Federal & State Regulations Placement Credits Transfer Credits Support Services Freshman Course (large class size) Student Dropout Trajectory of system failure when Loopholes in the firewall of the system lineup System Defenses working

35. Application to Student Dropout: Process Layer Triggers Incomplete information about the Prospect Incomplete Communication Plan Staff/Recruiter Workload Institutional factors Recruitment Over-whelmed with Advising Information Admission Decentralized / fragmented Information System Orientation Immature Freshman Over-worked Athlete Advising Learning Environment w/: Instructor as Coach Small class-size Team-based Peer Learning Relevant Projects Instructor Interactions Assessment & Feedback Decentralized Information System Vague Recruitment Guidelines Placement Credits Transfer Credits Support Services Student Completes Course System Defenses working when Loopholes in the firewall of the system gets deflected

36. Interventions: National Level Federal & State Initiatives • National priority (mission is to educate). • Address accountability and legal liability issues. • Appropriations and student performance. • Student Attrition Management System (Shaik, 2003a). • Culture of Retention (Shaik, 2003b). Student Attrition Management System • Based on standardized metrics and failures typology. • Centralized national database on student attrition. • Online web-based system with a user friendly interface. • Reporting (voluntary and mandatory) of dropout incidents at (institutional) process level. • Recovery management framework.

37. Interventions: Organization Level Assessment of work environment • Methodologies: • Cognitive Task Analysis (What is); • Cognitive Work Analysis (What should be); • Fault-tree Methodology; • Goal: identify latent errors & redesign environment. Reward structure • Goal: to encourage retention by faculty and staff. Research • Validated and replicable using time-series and cross-program data from the national database. • Reports on dropout to go beyond data collection. • Pattern Analysis to identify the causes and to discover robust patterns which can then be used as forecasts from operational and reporting data.

38. Selected References • Bean, J. P. (1980). Dropouts and turnover: The synthesis and test of a causal model of student attrition. Research in Higher Education, 12, 155-187. • Boshier, R. (1973). Educational participation and dropout: A theoretical model. Adult Education, 23, 255–282. • Cabrera, A. F., Nora, A., and Castaneda, M. B. (1993). College persistence: Structural equations modeling test of an integrated model of student retention. Journal of Higher Education, 64(2), 123-139. • Cross, K. P. (1981). Adults as learners. San Francisco: Jossey-Bass. • Draper, S. (2003). Tinto’s model of student retention. URL: http://www.psy.gla.ac.uk/~steve/localed/tinto.html • Fishbein, M. and Ajzen, I (1975). Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. Reading, MA: Addison-Wesley. • Rasmussen, J. (1983). Skills, Rules and Knowledge; Signals, Signs and Symbols and Other Distinctions in Human Performance Models. IEEE Transactions on System, Man and Cybernetics. SMC-13 (3). 257-266. • Norman, D. (2002). The Design of Everyday Things. Basic Books. • Reason J. (1990). Human Error. Cambridge University Press. • Shaik, N. (2003a). Student Attrition Management System: An Organization with Memory – A Conceptual Framework . Working paper, Division of Academic Outreach, University of Illinois. • Shaik, N. (2003b). Culture of Student Retention: Alternative Framework. Working paper, Division of Academic Outreach, University of Illinois.

39. Selected References • Spady, W. G., (1971) Drop-outs from Higher Education: An Interdisciplinary Review & Synthesis, Interchange, 64-85. • Seidman, A. (1996). Retention Revisited: RET = E Id + (E + I + C)Iv. College and University, 71(4), 18-20. • Tinto, V. (1993). Leaving College: Rethinking the Causes and Cures of Student Attrition, Chicago: University of Chicago Press. • Vicente, K. and Rasmussen, J. (1992). Ecological Interface Design: Theoretical Foundations. IEEE Transactions on Systems, Man, and Cybernetics, 22(4), 589-606. • Virginia Commonwealth University (2000) Retention 2000 Recommendations. http://www.students.vcu.edu/dsa/retention2000/advising • Witte J. W., Forbes, S. E. & Witte, M. M (2000). Identity Theory and Persistence: A Tentative Synthesis of Tinto, Erikson, and Houle. Journal of Integrative Psyhcology. Vol 2.