Using Performance Reviews in Capstone Design Courses for Professional Skills Development

1. USING PERFORMANCE REVIEWS IN CAPSTONE DESIGN COURSES FOR DEVELOPMENT AND ASSESSMENT OF PROFESSIONAL SKILLS(Paper 1041, Session 3525)Greg Kremer Associate Professor and Chair, Mechanical Engineering (kremer@ohio.edu) David BurnetteMSME, 2007Ohio University, Athens, OH

2. OR----------------------------------Expanding ME program outcomes from"How-2-Do" engineeringto"How-2-Be" a good engineer

3. HOLD PARAMOUNT…

4. 1ST DO NO HARM HOLD PARAMOUNT…

5. ENGINEERS MAKE A WORLD OF DIFFERENCE WILL IT BE FOR GOOD OR BAD? SOCIAL CONTEXT

6. CASTL 2005-2006 Integrative Learning Project Overall Goal My true passion is to craft a capstone design experience that is truly an “education of the whole person” in order to encourage engineering students to appreciate the importance of integrity and wholeness in life. I want our graduates to see the profession of engineering as an opportunity to serve the world and support social justice, and I want them as individuals and as groups to “care as well as know”. My integrative learning project deals with integrating the non-technical but critically important "professional skills" into student conceptions of what it means to be a good engineer.

7. CASTL 2005-2006 Integrative Learning Project Initial research tasks • 1. Develop a taxonomy of professional skills outcomes for engineers that are understandable to undergraduate engineering students and that are linked to the ABET professional skills outcomes. • 2. Develop an authentic way to get undergraduate engineering students to value and develop professional skills. • 3. Begin initial assessments of the effectiveness of purposeful identity development activities (integrated throughout the Mechanical Engineering program and organized around a team-based capstone design project) on student achievement of an integral engineering identity.

8. OU ME Program Educational Objectives Prepared for Engineering Careers and Advanced Education Technical Skills Skills to perform in the work environment (Professional Skills) Informed & aware of contemporary issues and the impact of engineering on society INDUSTRIALADVISORYBOARD

9. Skills to perform in the work environment (Professional Skills) Safety and health Technical communication: Oral and written Teamwork Project management: planning, scheduling, budgeting Self evaluation, leading to improvement Awareness of international standards, quality standards, and systems of units Appreciation of engineering integration with business (market awareness, customer satisfaction, quality, continuous improvement, profit, mission / vision / core values, ...) Ethical and effective decision making, involving environmental health and safety, economics, time, quality, performance / operability, and reliability / life. ABET PROFESSIONAL SKILLS

10. OU ME CAPSTONE DESIGN Professional behavior (responding to open-ended problems, dealing with uncertainty, making “good” decisions based on informed engineering judgment) Student behavior (responding to what the instructor asks)

11. Professional Skills – Prior Work Lewis and Bonollo (2002) Shuman et al. (2005) Claim that “portfolios, along with performance appraisals and behavioral observations, offer the most comprehensive information for measuring many outcomes and are conducive to evaluating professional skills.” Identified 5 professional skills that professional design clinic supervisors valued highly in students: 1. Negotiation with clients 2. Problem solving 3. Acceptance of responsibility for outcomes (professional behavior), 4. Interpersonal skills5. Project management

12. Professional skills must be presented and developed in an authentic way ---------------- Add-on requirement vs Integral to engineering identity DECISION: ACTIVITY WITHIN CAPSTONE PROJECT VS ACTIVITY CONSTRUCTED FOR ASSESSMENT

13. OU ME CAPSTONE DESIGN ~ENERGY EFFICIENT CAMPUS TRANSPORTATION, NISH Need statement based on contemporary issues ~50 STUDENTS/YEAR Identify acustomer & market Large (6-8) and diverse teams Develop targetspecs based on customer needs Year-long Capstone Design Project Develop aproduction plan Create and maintain a project schedule Build and test a prototype Develop feasibleconcepts Select a conceptand refine the design

14. OU ME CAPSTONE DESIGN Large (6-8) and diverse teams Need statement based on contemporary issues Professional Skills Identify acustomer & market Develop targetspecs based on customer needs Year-long Capstone Design Project Create and maintain a project schedule Develop aproduction plan Build and test a prototype Develop feasibleconcepts Select a conceptand refine the design

15. Example PR from Industry LearningNeeds

16. OU ME CAPSTONE DESIGN Create list of Professional Skills Important to their team with explanations and examples and use it for peer ratings Need statement based on contemporary issues Large (6-8) and diverse teams Identify acustomer & market Performance Reviews with self and peer rating of professional skills Develop targetspecs based on customer needs Participate in face-to-face performance reviews with “supervisor” to review accomplishments and set project and personal development goals Year-long Capstone Design Project Develop aproduction plan Create and maintain a project schedule Build and test a prototype Develop feasibleconcepts Select a conceptand refine the design

17. DECISION: ASSESSMENT RUBRIC VS TAXONOMY DECISION: STUDENT DEVELOPED VS FACULTY DEVELOPED DECISION: DEVELOPMENT LEVEL VS EXPECTED BEHAVIOR I always bring positive energy to the group. An example is when we don't do as well as we expected on one of our presentations or reports, I am the one that will make a joke to lighten up the tension in the room. I usually make fun of Phil or someone.

18. PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006-2007

19. PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006-2007

20. PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006-2007

21. PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006-2007

22. PROFESSIONAL SKILLS RATING SHEET Developed by students in 2006-2007

23. Professional Skills Performance Review: PURPOSE IS DEVELOPMENT AND ASSESSMENT

24. Assessment Results from 2005-2006 Class (Developmental Taxonomy, Development Project Defined in Winter and Assessed in Spring) * Students gave good examples but tended to over-rate their performance - but truth emerges in face-to-face dialogue as part of the performance review **Senioritis prevented some from documenting the results from their development plans

25. Assessment Results from 2006-2007 Class (Expected behavior Taxonomy, Development Project Defined in Fall, 1st assessment in Winter, Follow-up Assessment in Spring) Started the process earlier and was more clear in describing why we were doing performance reviews and how the would impact grades Performance level approach was more effective than development level Being part of Carnegie Scholar project made it seem important to students * Treated as mastery outcome with recycling

26. Observations and Lessons Learned Context Matters: >95% self report awareness of social impact of engineering, also high for the importance of professional skills • • When asked to discuss and describe professional skills that are important for being a good engineer, students come up with impressive lists and examples • “They know professionalism when they see it”. • Student involvement in creating the list is essential • Although students can identify areas for personal improvement and come up with improvement plans, most need reminders and assignments or else the plan will get “lost in the shuffle.” • Most students are not self-motivated for “self evaluation, leading to improvement”

27. Observations and Lessons Learned • • Although it takes a lot of time, face-to-face performance review meetings are very effective and allow a more authentic and professional discussion of goals, accomplishments, behaviors, etc. • Increases student engagement and accountability • • There are some advantages to using team consensus rather than class consensus or consensus of the profession for establishing the professional skills list • Universal consensus is difficult (especially with language and terms that are understandable by all), and most behavioral expectations and norms are set and evaluated locally • Most behavioral expectations can be integrated into the “team contract” to model “effective teamwork” practices

28. Thanks! Greg Kremer (kremer@ohio.edu)

29. ABET and Professional Skills Outcomes • A sense of professional and ethical responsibility • An appreciation for the impact of engineering solutions in a global and societal context • A knowledge of contemporary issues • An awareness of the integral nature of learning and engineering practice (lifelong learning) • An ability to function effectively on multi-disciplinary teams (including an understanding of the positive role of diversity and specialization) • An ability to communicate effectively