Evaluating Academic Challenge Beyond the NSSE

1. Evaluating Academic Challenge Beyond the NSSE Jim Purcell, Associate Vice Chancellor Oklahoma State Regents for Higher Education

2. Legacy of Assessment • Letter from the Six Nations to William & Mary College

3. We are convinc'd, therefore, that you mean to do us Good by your Proposal; and we thank you heartily. But you, who are wise, must know that different Nations have different Conceptions of things; and you will therefore not take it amiss, if our Ideas of this kind of Education happen not to be the same with yours. We have had some Experience of it. Several of our Young People were formerly brought up at the Colleges of the Northern Provinces; they were instructed in all your Sciences; but, when they came back to us, they were bad Runners, ignorant of every means of living in the Woods, unable to bear either Cold or Hunger, knew neither how to build a Cabin, take a Deer, or kill an Enemy, spoke our Language imperfectly, were therefore neither fit for Hunters, Warriors, nor Counselors; they were totally good for nothing.

4. We are, however, not the less oblig'd by your kind Offer, tho' we decline accepting it; and, to show our grateful Sense of it, if the Gentlemen of Virginia will send us a Dozen of their Sons, we will take care of their Education; instruct them in all we know, and make Men of them.

5. Knowing is half the Battle • GI Joe Cartoon Character and Action Figure • Know Thyself • Socrates • “Every new adjustment is a crisis in self-esteem.”

6. Assessment is a Mirror • Assessment is a reflection of reality – not reality Now we see but a poor reflection as in a mirror. Now I know in part; then I shall know fully . . . 1 Corinthians 13:12

7. JoHari Window Known by Self Unknown by Self Open Area Blind Area Known by Others Unknown Area Hidden Area Unknown by Others Joseph Luft and Harry Ingham

8. JoHari Window Known by Self Unknown by Self Open Area Blind Area Known by Others Unknown Area Hidden Area Unknown by Others Joseph Luft and Harry Ingham

9. Georgia College & State University

10. Student Engagement • Notional value: We like the idea, but do not take action. • Cost for caring about student engagement: • Investment of time, study, money, and risk. • What are we doing? • Is it working? • Important to take risk: Pause to think day.

13. The NSSE first Reading • Asked the questions that were at the core of the institutional mission • Sense of Place • Sense of Activity • Sense of Interaction • Provoked faculty interest • Enabled institutional conversation

14. 1= never 2= sometimes 3= often 4= very often National Liberal Arts Georgia College 1st-year 1st-year 1st-year 3.00 2.75 2.81 Seniors Seniors Seniors 3.06 3.16 3.30 Selected NSSE findings Asked questions in class or contributed to class discussions. From NSSE 2000

15. 1= never 2= sometimes 3= often 4= very often National Liberal Arts Georgia College 1st-year 1st-year 1st-year 2.44 2.42 2.28 Seniors Seniors Seniors 2.49 2.64 2.43 Selected NSSE findings Worked with other students on projects during class. From NSSE 2000

16. 1= never 2= sometimes 3= often 4= very often National Liberal Arts Georgia College 1st-year 1st-year 1st-year 2.61 2.39 2.31 Seniors Seniors Seniors 2.71 2.76 2.83 Selected NSSE findings Worked with other students on assignments outside of class. From NSSE 2000

19. Faculty Response? • Bogus: The survey was not administered correctly • Bad Survey: The survey failed to measure student engagement • Bad Students: We deal with a different type of student than other schools • Bubba’s problem: The results do not apply to me and my colleagues in my department

20. A few months later . . .

21. National Survey of Student EngagementSenior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher • Used e-mail to communicate with an instructor or other students. (Sig. .018) • A&S (2.80), Bus (2.90) ,Ed (3.40), HS (3.40) • Made a class presentation. (Sig. .015) • A&S (2.87), Bus (2.69) ,Ed (3.42),HS (3.07) • Worked with other students on projects during class. (Sig. .001) • A&S (2.33), Bus (2.55) ,Ed (3.32),HS (2.60)

22. National Survey of Student EngagementSenior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher • Participated in a community-based project as part of a regular class course. (Sig. .000) • A&S (1.47), Bus (1.25) , Ed (2.21),HS (2.53) • Used an electronic medium to complete or discuss a project. (Sig. .044) • A&S (2.27), Bus (2.17) ,Ed (2.90), HS (2.53) • Discussed grades/assignment with instructor. (Sig. .014) • A&S (3.13),Bus (2.52) ,Ed (3.00),HS (2.73)

23. National Survey of Student EngagementSenior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher • Number of assigned textbooks, books, or book length packs of course reading. (Sig. .005) • A&S (3.24), Bus (2.90) , Ed (2.79),HS (3.80) • Number of papers over 20 pages. (Sig. .004) • A&S (1.50), Bus (1.50) , Ed (1.25),HS (2.00) • Number of papers under 20 pages. (Sig. .022) • A&S (2.72), Bus (2.62) ,Ed (3.40), HS (3.20)

24. National Survey of Student EngagementSenior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher • Course work emphasizes synthesizing and organizing ideas. (Sig. .005) • A&S (2.83), Bus (2.28) , Ed (2.70),HS (3.27) • Course work emphasizes making judgments about the value of information. (Sig. .027) • A&S (2.79), Bus (2.31) , Ed (2.55),HS (3.20) • Preparing for class (studying, reading, etc) (Sig. .017) • A&S (3.77),Bus (2.86) , Ed (2.95),HS (4.29)

25. National Survey of Student EngagementSenior students –by GCSU Academic SchoolsStatistically significant responses at .05 level or higher • Contributed to acquiring a work related skills. (Sig. .007) • A&S (3.13), Bus (2.83) ,Ed (3.45), HS (3.73) • Contributed to analyzing quantitative problems. (Sig.033) • A&S (2.69),Bus (3.00) ,Ed (2.40),HS (3.20) • Contributed to the welfare of your community. (Sig. .005) • A&S (2.07), Bus (2.21) , Ed (2.65),HS (3.07)

26. A few more months later . . .

27. NSSE survey Level of Academic Challenge Active and Collaborative Learning Student Interaction with Faculty Enriching Educational Experience Supportive Campus Environment

28. Evaluating Academic Challenge Beyond the NSSE • Stephen L. Payne, Karynne L. M. Kleine, Jim Purcell, and Ginger Rudeseal Carter • http://www.springeronline.com/sgw/cda/frontpage/0,11855,4-40406-70-35612147-0,00.html Kuh (2001) strongly encouraged institutional leaders to adopt multi-faceted, multi-year, data-collection strategies that capitalize on what contrasting assessment approaches do best.

29. Academic Challenge Eleven items are incorporated in the academic challenge dimension of the NSSE. Many items relate strongly to objectives on Bloom’s and other learning taxonomies and others involve the amount of time or effort that students must have on the course.

30. Focus Group Questions Do students at particular institutions discount the importance of aspects of academic challenge that the NSSE tries to measure? Is the NSSE missing aspects of academic challenge that faculty and students at particular institutions believe would be helpful for improving academic challenge there? How might an institution with a specific mission, academic culture, and set of characteristics examine its degree of academic challenge and move toward more comprehensive assessment and improvement of this academic challenge?

31. Focus Groups • Twenty-nine faculty • Fifty-four students • Main and branch campus students were interviewed. • Transcriber • Asked questions concerning: • conceptions of academic challenge, • the appropriateness of the academic challenge items on the NSSE, • and what might be done to improve academic challenge at this institution.

32. Academic Challenge Questions

33. What is Academic challenge? Students • Initial reactions to the concept of academic challenge were frequently negative – • Emphasized the extra time and effort as well as the initial uncertainty or confusion associated with such courses • Students later mentioned rewards of academically challenging courses. • Synthesis and, application of learning content • Faculty teaching style and student interactions were a concern

34. Off-campus students • Stressed stronger concern for career entry and success • Reacted less favorably than other students to academic challenge in courses, particular in core courses of those connected to their majors

35. What is Academic challenge? Faculty • Significant minority of faculty emphasized lack of student preparation to enable student engagement. • “Basics” limiting student opportunities for higher-order learning and academic challenge • Need for strategies to challenge students. • Need improved resources for faculty development

36. What is Academic challenge? Faculty • Academic challenge requires a recognition of different student learning styles and offering multiple paths • Learning outcomes become more important • Need more and improved assessment using the NSSE and other indicators

37. Proposal for expanded assessment • Accepted by the Academic Governance Committee of the University Senate of GC&SU (2004) • Ask to devote time to developing this internal assessment instrument • Used NSSE measures for benchmarks for quality enhancement plan

38. Ideas for local research related to the NSSE discussions • Critical thinking, reading, and writing are at the core of academic challenge, but these important concerns are not handled the existing NSSE items • Several academic challenge items stress student effort • Items are not linked to learning outcomes

39. Ideas for local research related to the NSSE discussions • Study relationships between student perceptions of learning and actual outcome measures of affective, creative, aesthetic, and leadership development particular programs • Development of items that permit distinction between experiences of courses • Distinguish between academic challenge and experiences that are merely highly competitive, demanding, or even intimidating.

40. What other interesting tidbits from this study? • Students have less interest and motivation for learning in core courses they perceive as having less direct career relevance. • What roles of responsibilities might faculty and students assume for improved learning outcomes? • Establishing both high academic rigor in their courses as well as a high level of faculty-student relationships out of class seems to be a contradiction to many faculty.

41. What other interesting tidbits from this study? • Students placed greater responsibility on faculty in their learning experience. • Faculty feel that assignments can result in more meaningful learning outcomes than students might realize at this point in their lives

42. Closing Remarks • We should design educational experiences that are empowering • Measure quality of encounters • Ensure encounters are transformational • We should desire positive restlessness