Assessing Student Attainment in the Undergrad. Major:

1. Assessing Student Attainment in the Undergrad. Major: • Examples of Discipline-Specific Strategies • Senior Thesis or Writing Project • Senior Research Project • Capstone Course with Assessment Embedded in the Course • Comprehensive Examination or Proficiency Test • Student Portfolio, Performance, or Exhibit • -- with expert or peer assessment • Senior Essay followed by Focus Group Discussions with Faculty • Student/Faculty Retreat for Collective Assessment • Internship or Field Work • Analysis of Historical, Archival, or Transcript Data • Alumni and Employer Surveys and Interviews Volkwein- Penn State

2. Should I purchase an instrument or create my own?? If you develop your own instrument: • can be less expensive (at least in dollars) • can address institutionally-specific issues • can be very time consuming • must attend to formatting, ease of completion for respondents, printing, on-site scoring • may require pilot testing • can require extensive commitments for reliability, validity checks or can jeopardize credibility • Has less external credibility, can increase chances for criticism Volkwein- Penn State

3. Development of Local Instruments Literature Review and Collection of Survey Items Item Reduction & Revisions by Research Team Pilot Test to Determine Psychometric Properties & Revise Final Survey Items Seniors Vet with Faculty, Employers, Alumni, and Administrators Alumni Faculty Program Chairs Employers Volkwein- Penn State

4. Tips for Improving Response Rates • Explain the Importance of the Survey • Explain How the Results will be Used • Assure Confidentiality • Thank Participants • Make the Instrument Interesting • Offer Incentives, if possible • Offer to Share the Results • Be sure the Survey is easy to understand and complete • Offer multiple modes of response (web, paper, telephone) Volkwein- Penn State

5. Should I purchase an instrument or create my own?? If you purchase an instrument: • Requires less time for design, printing, etc. • Likely has established reliability and validity • Often includes norms groups for inter- institutional comparisons • May include scoring and reporting services • Likely to have external credibility, Increases face validity • May not fit your curriculum & goals • Can be expensive in dollars Volkwein- Penn State

6. Advantages of Standardized Tests • Validity Established through • content specification • expert panel reviews for fairness • pilot testing • Minimal Staff Time Involved • Comparative Data and User Norms Available • Short Turn-around Time for Results • Access to Professional Expertise Volkwein- Penn State

7. Why not use the GRE for assessment? • GRE Exams have the following deficiencies. • GRE scores are relational and only answer the comparison question. You don’t know what a 600 means in terms of student achievement because you don’t know how many questions were answered correctly and incorrectly. • The GRE uses the wrong comparison group. Instead of comparing each GRE score against all college graduates, student GRE scores are in relation to a graduate school-bound population. • There are no GRE subfield scores within each major field, so you cannot tell if student performance is congruent with the curriculum. • The ETS Major Field Exam was constructed in response to these weaknesses in the GRE. The Major Field Exam scores not only are normed on populations of graduating seniors, and not only indicate the number answered correct (non relational), but also report scores by subfield, thus providing useful information for analyzing the curriculum. I recommend that the faculty send off for a specimen set, take the exam, discuss and decide.

8. Validity of Self-Reported Data Self-reports vs. actual scores/tests in various studies shown to correlate (r = .50 to .90) with: • SAT & GRE Verbal and Quantitative scores • National Teacher Examination scores • Criterion-referenced achievement tests in English, math, science, and social studies • College and High School GPA • Course material tests • Behaviors/activities consistent with reported gains Volkwein- Penn State

9. When Self-Reports are Reasonable Proxies for Objective Measures • The information requested is known to the respondent • Questions are phrased clearly and unambiguously • A moderate-to-high degree of overlap in content exists between the measures • The instruments measure the same constructs, and questions refer to recent activities • Respondent believes questions merit a serious, thoughtful response • Answering won’t threaten, embarrass, or violate the respondent’s privacy, nor encourage socially desirable responses • Self-reported gains/proficiencies are for groups (averaged), rather than for individuals (Based on Kuh, 2005 and Pike, 1995) Volkwein- Penn State

10. Strengths of Surveys • Great Flexibility and Universal Applicability • Ease of Construction • Application to all Students • Results do not Require Expert Interpretation • Relatively Inexpensive Volkwein- Penn State

11. Weaknesses of Surveys • Better for Measuring Group Responses than for Individual Responses • Best if Supplemented by other information from Interviews & Focus Groups. • To Construct Good Surveys • avoid vague items • Construct multi-item scales • Link to constructs in your model or to educational goals • Effective visual layout Volkwein- Penn State

12. Value of Alumni Studies(Volkwein, 1990) • Alumni provide a ‘Janusian’ perspective - Internal and external experiences. • Alumni and employers have legitimacy with both internal and external stakeholders. • Internally, Alumni Studies can assess important outcomes and provide info for enhancing curriculum, programs, and policy. • Externally, Alumni studies can support accreditation, accountability, recruitment, and fund raising. • Such studies provide an opportunity for faculty and administrative collaboration. • Centralized collection, decentralized uses Volkwein- Penn State

13. Outcomes Approach to Alumni Assessment Assessment of alumni rests on the premise that institutional quality and effectiveness can be measured by examining: • Educational & Job satisfaction • Income & Socio-economic Status • Occupational Attainment • Engagement in civic and political activities • Values like Tolerance for diversity • Collegiate experiences • General Satisfaction with Institution • Quality of Instruction received • Preparedness for employment & Grad School • Willingness to enroll again & Donate $ (Volkwein, 1990, 1998; Dellow & Romano, 2002) Volkwein- Penn State

14. Selected Alumni Surveys • ACT Alumni Survey (2-year & 4-year) • SUNY Alumni Outcomes Survey • Appalachian Region Alumni Outcomes • SUNY-Albany Alumni Survey • HEDS Alumni Survey • HERI Alumni Survey • NCHEMS Alumni Assessment Survey • Clemson University Alumni Survey • Georgia Tech Alumni Survey • Penn State Engineering Alumni Survey Volkwein- Penn State

15. ACT Alumni Outcomes Survey • Demographics/Background/Career (15) • Employment History and Experiences (18) • Educational Outcomes (22) • Critical Thinking & Problem Solving (4) • Life-long Learning (2) • Social/Moral Values/Ethics (3) • Multi-Cultural/Diversity (4) • Citizenship/Global Issues (2) • Team Work (2) • Communication (2) • Overall (3) • Educational Experiences/Satisfaction (31) • Activities and Organizations (11) • Additional Questions (30 spaces) • Space for Comments/Suggestions Volkwein- Penn State

16. Penn State - CSHE Engineering Alumni Survey • Demographics/Background/Career (13) • Undergraduate Engineering Outcomes/Experiences (72) • Technical Skills and Abilities (15) • Professional Skills (23) • Analytical/Thinking Skills (10) • Nature of Engineering Courses (15) • Diversity & Tolerance (9) • Additional Information (9) • Satisfaction (1) • Test performance (1) • Degrees, field, and minor (4) • Planned and Actual employment (3) Surveys available at http://www.ed.psu.edu/cshe/abet/instruments.html Volkwein- Penn State

17. Percent of Alumni That Would "Attend All Over Again" and Select Same Major and Same Career (N=1743) Mean Response on a 5 point Scale 12 66 3.9 Attend All Over Again 11 67 3.9 63 14 3.8 26 54 3.5 20 Select Same Major 61 3.7* * = These mean responses are significantly higher than those in careers not related to the major.(p=<.01) 34 41 3.1 19 58 3.6 16 62 Select Same Career 3.7* 21 52 3.5 100 80 60 40 20 0 0 20 40 60 80 100 Probably & Definitely No Probably & Definitely Yes All Alumni Career Not Related to Major Career Related to Major

18. Perceived contribution of the College Experience To Alumni Development as Undergraduates Mean Response (Scale of 1 to 5) 68 3.8 4 Intellectual Growth 4.2* 70 4 3.7 63 9 * Means are significantly different from pre-2000 (p=<.01). Personal Growth 3.9* 71 12 3.6 58 11 Pre-2000 3.9* Social Growth 70 10 Post-2000 3.7 62 Effective Preparation 9 51 for Graduate School 3.5* 13 3.6 Effective Preparation 56 12 for Employment 42 3.3* 19 0 20 40 60 80 100 80 60 40 20 0 Percent responding 1 or 2 on a 5 point scale Percent responding 4 or 5 on a 5 point scale

19. Abilities and Skills Rated by Engineering Employers as Highly Important for Success Communicate effectively* Problem solving* Apply math, science, and engineering Use modern engineering tools Teamwork* Understand professional and ethical responsibilities Design a system to meet needs Life-long learning* Design and conduct experiments Knowledge of contemporary issues Engineering In global and social contexts 99% 99% 97% 97% 95% 94% 92% 90% 85% 73% 70% Volkwein- Penn State

20. Function independently, without supervision Exercise personal responsibility Listen effectively* Exercise self-discipline Exercise problem-solving skills* Maintain openness to new ideas* Speak effectively* Evaluate and choose between alternative actions* Think analytically and logically Acquire new skill and knowledge on your own* Possess clear goals Cope with conflict Understand myself Write effectively* Lead and supervise tasks and groups of people Function effectively as a member of a team* Learn how to learn* 96% 96% 93% 91% 91% 90% 90% 89% 89% 87% 84% 82% 80% 80% 78% 75% 75% Abilities and Skills Rated by Non-Engineering Alumni as Highly Important for Success

21. Consensus Liberal Education Outcomes • Over the past ten years, the AAC&U appears to have forged a national consensus among business, government, and accreditation leaders about the liberal education outcomes that all undergraduates should possess, summarized in Our Students’ Best Work: A Framework for Accountability Worthy of Our Mission (2004): • Strong analytical, communication, quantitative, and information skills. • Deep understanding or hands-on experience with the inquiry practices of disciplines that explore the natural, social, and cultural realms. • intercultural knowledge and collaborative problem-solving skills. • A proactive sense of responsibility for individual, civic, and social choices. • Habits of mind that foster integrative thinking and the ability to transfer skills and knowledge from one setting to another.

22. The Value of a Model • Encourages Clarity of Purpose • Stands as a Road Map or Guide for Developing • Research Questions • Assessment Design • Data Collection • Hypothesis Testing • Statistical Analysis • Serves as a Logic Chart (Otherwise Causality is not Clear) • Assists Workload Conservation and Fog Dissipation (concentrates energy and attention, streamlines research design, reduces the amount of data collection, data storage, analysis, and reporting) Volkwein- Penn State

23. Volkwein Effectiveness Model 1. Janusian Duality 2. Five Questions: 1. Are you meeting your goals?2. Are you improving? 3. Do you meet the standard?4. How do you compare?5. Are your efforts cost-effective? Inspirational Purposes-for Internal Improvement • 5. • CommunicateResults • Take Academic & • Administrative Action • Improve & Strengthen • Programs 4. Collect & Analyze Evaluation Evidence Pragmatic Purposes- for External Accountability 3. Select Methods & Measures 3. Select Model, Methods, Measures For Academic Effectiveness: Student Learning, Research & Scholarship For Administrative Effectiveness:Planning & Resource Management For Administrative Effectiveness:Strategic Planning & Resource Mgmt For Academic Effectiveness: Student Learning, Research & Scholarship For Each Level --Institution --Program --Individual

24. Today’s Effectiveness Approach • Undergraduate Focus, • Goal driven, • Evidence based, • Improvement oriented Volkwein- Penn State

25. Assessment Resources • Patrick Terenzini’s article, “Assessment with Open Eyes” (JHE, 1989). • Middle States 2003 Handbook, Student learning assessment: • Options and resources. • Linda Suskie’s 2004 book, Assessing Student Learning: A Common Sense Guide • Several books by Trudy Banta and her associates constitute helpful • resources for campus assessment efforts, Assessment in practice: • putting principles to work on college campuses(Banta, et al. 1996), • Assessment essentials: planning, implementing, and improving • assessment in higher education (Palomba & Banta 1999), and • Building a scholarship of assessment (Banta et al., 2002). • Additionally, there have been several constructive national attempts to • develop guidelines and standards for good assessment practices. • The most significant of these are the AAHE Nine Principles of Good • Practice for Assessing Student Learning, and the recommendations of • the AAC&U (from Our Students’ Best Work, 2004). Volkwein- Penn State