Dr. Stephen Landry, CIO Dr. Eric Fountain, IT Assessment Coordinator

1. Using External Data Sources to Increase Confidence in Assessment Conclusions Dr. Stephen Landry, CIO Dr. Eric Fountain, IT Assessment Coordinator Danielle Mirliss, Instructional Designer

2. Who We Are … • Seton Hall University • Mid sized, Catholic affiliated university in suburban NJ, 15 miles from New York City. • 4,400 Undergraduates, 50% residential. • Two campuses: Main Campus in South Orange, NJ and Law School in Newark, NJ. • Approx. 350 Full Time Faculty on the South Orange Campus. • Research Intensive (Research II) Institution.

3. Institutional Change @ SHU • SHU made a major commitment to institutional change in its 1996 Strategic Plan • Seton Hall became a ubiquitous computing campus, providing laptop computers to the learning community as part of tuition and fees • Seton Hall increased enrollment while increasing academic requirements for admission • Seton Hall added new programs and faculty • Seton Hall increased emphasis on assessment and accountability

4. Ubiquitous Computing @ SHU • Seton Hall University’s Mobile Computing Program is designed to ensure all members of the learning community have access to learning resources which are increasingly online. • Program has three interdependent components: • Access: The University licenses students the use of a laptop as part of their tuition and fees; • Curricular Integration: The University provides support and incentives to faculty to use technology in T&L; • Services and Support: Network (Wired and Wireless), Help Desk, Laptop Repairs, Insurance, Training, etc.

5. Implementing Ubiquitous Computing 1994 – 1996: Development of SHU Strategic Plan 1995 – 1997: Mobile Computing Pilot Project with small cohorts of students in Business School 1997: Mobile Computing Pilot Project expands to 300 incoming Freshman 1998: Mobile Computing Program launched for all incoming Freshman 2001: Mobile Computing Program expanded to select graduate programs; campus wireless network implemented 2002: SHU graduates first mobile class 2004: Tablet PC Pilot Project

6. Mobile Computing and Curriculum Development • Seton Hall recognized that access to technology is necessary for the changes envisioned in the learning environment, but not a sufficient condition. • Teaching, Learning and Technology Center • Provides support and training for faculty in their use of technology in their teaching • Internal Grant Programs: • Curriculum Development Initiative • Faculty Innovation Grants • Implementation of Classroom and Network Technology • Teaching, Learning, and Technology Roundtable

7. Mobile Computingand Assessment • In 1997 Seton Hall recognized it had a unique assessment opportunity • Mobile Computing Assessment Program has two objectives: • To assess the impact of ubiquitous computing on the learning environment • To provide prompt feedback to the Mobile Computing Program planning team to address potential implementation issues with the program

8. Assessment Methodology and Resources

9. Foundations of the Assessment Movement Research Traditions Contributing to the Assessment Movement: • Student Learning in College • Retention and Student Behavior • Evaluation and Scientific Management • Mastery Learning

10. Assessment Process Cycle • Reflect • Plan • Act • Observe A B D C

12. National Studies Studies Focused on Students: • National Survey of Student Engagement (http://www.iub.edu/~nsse/) • Higher Education Research Institute (http://www.gseis.ucla.edu/heri/heri.html • CIRP Freshman Survey: profile of entering freshman class. • CSS College Student Survey: follow up of CIRP cohorts. • YFCY Your First College Year • Flashlight (http://www.tltgroup.org/programs/flashlight.html) Studies Focused on Faculty: • Faculty Survey of Student Engagement (http://www.indiana.edu/~nsse/fsse/index.htm)

13. National Studies Studies Focused on Technology: • EDUCAUSE Core Data Survey (http://www.educause.edu/apps/coredata/index.asp?bhcp=1) • Campus Computing Project (http://www.campuscomputing.net/) • Cost of Supporting Technology Services (http://www.costsproject.org/)

14. Assessment in the Teaching, Learning and Technology Center

15. Current Initiatives • Mobile Computing Assessment Team • Project Assessment • TLTR-Assessment Committee

16. Mobile Computing Assessment Team • Cross-Campus, Cross-Discipline • Provost’s Office • Institutional Research • Faculty • TLTC • Multiple Projects • Mobile Computing Assessment Survey • Extant Data Analysis • Large-Course Redesign • Classroom Study • Multi-Campus

17. Mobile Computing Survey Research Questions R1. What is the effect of the Mobile Computing Program on student satisfaction? R2. What is the effect of the Mobile Computing Program on technology use? R3. What is the effect of the Mobile Computing Program on desired educational outcomes?

18. Results The typical Seton Hall University student, whether representing all respondents, or particular groups of students (groups identified by gender, race/ethnicity, and residence status), is generally: • Attracted to Seton Hall University by the availability of technology at the University and the infusion of technology in the curriculum, as suggested by student reported positive influences on the decision to attend the University, • Satisfied with the Mobile Computing Program as well as with the laptop and support services once enrolled at the University, • Making good use of the technology available to all students, at least in terms of certain types of technology use. • Perceiving a substantive positive impact of Seton Hall University’s Mobile Computing Program on the learning environment.

19. Course Level Assessment Strategies Web-based Online Courses Mixed mode Hybrid Courses Enhanced Courses with electronic resources and exercises

20. Hybrid Course Assessment • Collaborative effort between Provost’s Office, faculty, and TLTC • Student and Faculty focus groups • Development and implementation of pilot survey

21. Tablet PC Highlights

22. MCAT – Future Plans Expand Faculty Participation Expand Membership to Include Student Affairs and Student Government Organize and Publicize Results Approach Question of Classroom Use FlashLight Comparison Studies

23. Using External Data Sources to Increase Confidence in Assessment Conclusions An Example from Seton Hall University using results from the 2003 National Survey of Student Engagement

24. What NSSE Measures • Student exposure to and participation in educational practices that promote engagement and student learning • Interaction with faculty in & out of the classroom • Active & collaborative learning • High expectations and academic challenge • Enriching educational experiences • Student perception of a supportive campus environment helping them to succeed

25. NSSE Questions - Active & Collaborative Learning

26. NSSE 2003 - Active & Collaborative LearningSeton Hall Freshmen vs Peers & National

27. NSSE Questions - Faculty Student Interaction

28. NSSE 2003 – Faculty Student Interaction Seton Hall Freshmen vs Peers & National

29. NSSE Technology Qs • Technology Use • SHU StudentsSHU Carnegie Peers • Time online for academic - 6+ hours / week • 55% 2003 SHU FR40% Doct-Intensives FR • 51% 2003 SHU SR44% Doct-Intensives SR • Time online for any reason - 6+ hours / week • 88% 2003 SHU FR77% Doct-Intensives FR • 81% 2003 SHU SR72% Doct-Intensives SR

30. NSSE Technology Qs • Technology Use& Active Learning • SHU StudentsSHU Carnegie Peers • How often required to use IT - % Very Often • 59%2003 SHU FR49% Doct-Intensives FR • 61% 2003 SHU SR56% Doct-Intensives SR • Used IT to make presentations - % Very Often • 47% 2003 SHU FR34% Doct-Intensives FR • 56% 2003 SHU SR53% Doct-Intensives SR

31. NSSE Technology Qs • Technology Use& Faculty Contact • SHU StudentsSHU Carnegie Peers • Email ask instructor to clarify assign - % Very Often • 51%2003 SHU FR30% Doct-Intensives FR • 57% 2003 SHU SR39% Doct-Intensives SR • Email to express ideas to instructor - % Very Often • 22% 2003 SHU FR 13% Doct-Intensives FR • 28% 2003 SHU SR 16% Doct-Intensives SR

32. NSSE Technology Qs • Technology Use& Active/Collaborative Learning • SHU StudentsSHU Carnegie Peers • Work in teams in class using IT - % Very Often • 27 %2003 SHU FR14% Doct-Intensives FR • 25% 2003 SHU SR19% Doct-Intensives SR • Communicate with classmates on-line to complete academic work - % Very Often • 51% 2003 SHU FR 27% Doct-Intensives FR • 49% 2003 SHU SR 39% Doct-Intensives SR

33. NSSE Technology Qs • Technology Use & Active Learning / Reflection • SHU StudentsSHU Carnegie Peers • Gain new insights into course material from on-line discussions - % Quite a bit+ • 43% 2003 SHU FR37% Doct-Intensives FR • 31% 2003 SHU SR32% Doct-Intensives SR • Express ideas to instructor by email not comfortable saying in class - % Very Often • 22% 2003 SHU FR 13% Doct-Intensives FR • 28% 2003 SHU SR 16% Doct-Intensives SR

34. Link to movie SHU NSSE & technology results are consistent and together suggest that SHU investments in IT are helping build educational environments at SHU that are increasingly engaging, challenging, learning community oriented, reflective and “active”. Janet Easterling Institutional Research Associate Seton Hall University

35. Conclusions

36. Questions? Contacts: Dr. Stephen Landry, CIO – landryst@shu.edu Dr. Eric Fountain, IT Assessment – fountaer@shu.edu Danielle Mirliss, Instructional Designer – mirlisda@shu.edu

37. Resources Internet Resources for Higher Education Outcomes Assessment - http://www2.acs.ncsu.edu/UPA/assmt/resource.htm#nsse AAHE Assessment Forum - http://www.aahe.org/assessment/assess_links.htm Assessment in Higher Education - http://ahe.cqu.edu.au/