A First Look at Retention

1. A First Look at Retention Following changes in a two-semester Business Statistics Sequence at Duquesne University Amy L. Phelps, phelpsa@duq.edu Speaking About Teaching Business Statistics eConference September 2012

2. Private Urban Campus www.business.duq.edu Duquesne is located on a secluded, private 48-acre campus in the heart of Pittsburgh. Duquesne is one of America’s leading Catholic institutions offering liberal and professional education since 1878. The Palumbo Business School graduates about 300 undergraduate students a year. The two-semester statistics sequence is required of all business students.

3. Palumbo School of Business

4. AACSB Assurance of Learning Standards, From AACSB White Paper, 2007: • AOL focus is the Program (not student, faculty, course…) • “By measuring learning the school can • evaluate its students’ success • use the measures to plan improvement efforts • provide feedback and guidance for individual students”

5. AACSB Assurance of Learning Standards • Prescribed process (AACSB, 2007): • Definition of learning goals and objectives • Alignment of curricula with adopted goals • Identification of instruments and measures to assess learning • Collection, analyzing, and dissemination of assessment information • Using assessment information for continuous improvement…

6. What is a ‘Program’ 1. The undergraduate Business curriculum • Eg. The Business Core courses • Align with mission statement of the School of Business 2. An undergraduate Business major • Eg. The economics major • Align with the mission of the Economics major 3. A collective set of courses with a common area of study • Eg. An assessment of the two required Stat courses • Align with specific learning objectives of the course

7. Related Literature • Statistical reform movement leading to GAISE • Copious assessment ff first course in stats • CAOS exam (DelMas) • Statistical literacy, pre/post, first course • Sisto (JSM, 2006) and Phelps (JSM, 2007) applied the CAOS to second semester business statistics • sig improvement in scores • Retention ff gap in time • Tintle et al (SERJ, 2012) • First course, pre/post/4mos lag • Compared traditional to randomized-based curriculum

8. Our History of Bstat Assess • CAOS Exam (DelMas) tracked pre/post • Hovered around 50% with stat II sig higher • CCPE: Duquesne exam aligned with LO • Stat I – sig ↑ year 3 • Stat II – hovered in low 40% • ETS: Commercially marketed national exam • 6 Prob and Stat ques: 12% to 97% students answered correctly • Statistics Retention Exam created by myself in alignment of school and program goals

9. First Stat retention assessmentAdministered Fall2009-Spring2011 • 18 MC questions • Concepts more than 70% answered correctly • The normal distn is defined by mean and sd • identified correct interpretation of CI for mean • identified correct interpretation of margin of error, p • identified correct scatterplot for given reg equation • correctly calculated joint prod from contingency table • Concepts less than 50% answered correctly • Use median when data are skewed • When data are skewed right, mean > median • As n increases, the length of CI decreases • identifying which histogram has larger sd

10. Written short answers • 2 short-answer, problem solving • Two-sample t-test • ~1/3 rejected a non-sig result • ~1/4 gave incorrect/inconsistent reasoning • ~1/4 inconsistent written conclusion based on whether they accepted or rejected the hypothesis • Simple linear regression • ~20% identified sig linpred with correct reasoning • Wide variations interpreting R2 • Much difficulty using Excel coefficients to create the least squares equation

11. Proposed curriculum changesAdministered in 2010-11 • We launched a revised curriculum attempting to reduce theory and increase application using real-world examples • ~Half the faculty used the traditional curriculum, half used the revised curriculum • Observed changes in retention exam scores administered Fall/Spring 2011-2012

12. Traditional curriculum • Business Statistics I • Descriptive statistics • (not including least squares regression) • Probability, Probability distributions, Expected values, random variables etc… • Normal distribution, Central Limit Theorem • Sampling distribution for mean • Confidence interval and intro to hypothesis testing • Business Statistics II • Review one-sample ttest, two-sample ttest, paired ttest, Chi-square tests, SLR, MLR, ANOVA • While technology was used, it was not formally taught, hand calculations were still assessed on written exams

13. Revised Curriculum • Business Statistics I • Descriptive statistics including least squares equ • Probability Rules presented through applications of contingency table • Normal distribution, CLT, Sampling distributions for means and proportions • One and two sample inference • Chi-square test of independence  Intention to prepare students for broad applications and case studies using statistical modeling More emphasis on written interpretations through lab exercises and teaching technology (Excel, StatCrunch)

14. Revised Curriculum • Business Statistics II • The second course emphasizing modeling through applications of cases • One and two sample inference is revisited with applications focused on using independent variable(s) to model/predict dependent variable • SLR/MLR, ANOVA, > 2x2 table chi-square are introduced as cases and datasets become more advanced and modeling makes sense. • Introduction to Analytics Bigger and more real-life datasets, emphasizing problem-solving and communication of results. Bi-weekly lab written reports.

15. Conceptually • Our thought was to give them the tools in the first semester • Foster understanding in the second semester through review, applications and cases • Emphasize written and verbal communication  Deeper understanding Ability to thoughtfully advance through the scientific method and communicate results.

16. The retention exam • Ideally, Sophomores take the two course sequence and then in their Junior year they take Global Economics either in the Fall or Spring • Ideally, they take the retention exam following a one (summer) or two semester break (summer plus fall).

17. Methods • We compared student performance on the retention exam between those taking the stats II between the traditional and revised • The 2011-12 retention exam was reduced to 12 MC, the two-sample t-test and SLR word problems remained the same. • Data from the Fall (2011), Spring (2012) and Fall 2012 retention exams were combined

18. Results

19. Changes in specific questions • Still collecting and analyzing • MC • Not much change • Maybe in identifying the use of median when data are skewed: from 25%  38 • Identifying stat sig slope from Excel output from 50%  57% • Short answer • Still working on that but • very few inconsistent conclusion in 2 sample ttest • fewer stating not sig predictor b/c R^2 is low

20. Accounting for Semester Lag • Regression analysis including semester lag (lag = 1-6 semesters) between stats II and retention exam Pred Score = 0.668 – 0.192(Curr) – 0.02(Sem) Adj R2 = 20.0%

21. Current Plan Diagram 1. Establish the assessment plan Define learning goals (6, linked to mission stmt)  identify outcome methods  implement methods 2. Implement & manage the assessment plan Methods  Resultsanalysis curr changes 3. Periodically update the assessment plan