Faculty-Industry Collaborations for Assessing Student Learning

1. Faculty-Industry Collaborations for Assessing Student Learning Flora McMartin University of California - Berkeley Jack McGourty Columbia University

2. Why Involve Industry? • Insures what is assessed is valued • Strengthens accountability • Addresses accreditation mandate

3. Campus-IndustryRelationship ACE Campus Trends Survey - 1996

4. Synthesis Coalition Cornell Iowa State Berkeley Hampton Stanford Northern Arizona Cal Poly Tuskegee Southern • Synthesis Coalition Goals • Institutionalize reforms • Assess & disseminate reforms • Develop NEEDS • Institutionalize K-12 outreach

5. Gateway Coalition - Drexel - Cooper - USC - Ohio State - Academic Associates - Polytechnic - NJIT - Columbia • Major Gateway Goals • Curriculum • Assessment • Underrepresented Populations • Instructional Technology • Professional Development • Linking & Sharing

6. Assessment Components • Stakeholder participation (faculty, students, & industry) • Faculty assessment training program • Campus (local) control of assessment process • Flexible, valid, and reliable tools

7. Assessment Goals A. Involve stakeholders to measure effects of coalition program reforms B. Develop/promote valid & reliable performance assessment tools C. Promote the institutionalization of assessment at participating campuses D. Satisfy needs of: individual faculty, colleges, & coalition program evaluation E. Provide a foundation for ABET accreditation

8. ABET 2000 Learning Outcomes 1. math, science & engineering knowledge 2. experiment, analyze & interpret data 3. design a system, component or process 4. function on multi-disciplinary teams 5. identify, formulate & solve engineering problems 6. understand professional ethics 7. communicate effectively 8. understand global/societal impact of engineering 9. engage in life-long learning 10. knowledge of contemporary issues 11. use modern engineering techniques, tools & skills

9. Synthesis Case Study

10. Student Learning Outcomes • Open Ended problem solving • Teamwork • Multi-disciplinary design • Hands-on facility with hardware • Communication skills - (oral, written, & interpersonal)

11. Goals for Involving Industry • Determine common learning outcomes valued by industry & engineering education • Articulate student learning outcomes in a common language • Bridge industry & engineering educational needs • Energize the assessment process through active participation of stakeholders

12. Mechatronics Industrial Board Berkeley Process Control Motorola Maytag Rockwell International Xerox PARC Ford Raychem Hewlett-Packard

13. The MIB, continued Membership: • primarily project engineers • observed needs vs. self reports Role: • define what industry values in educational outcomes • try out & review assessment tools • try out & review assessment rubrics & measures • evaluate assessment results

14. Planning Process: Track 1 ALL stakeholders define & articulate Student learning outcomes (SLOs) & classroom activities related to Synthessis goals Identifying Learning Outcomes SLOs reduced & refined SLOs reviewed by ALL stakeholders - draft of Synthesis Assessment Framework (SAF) Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF

15. Track 1: Learning Outcomes Engineering Ed. Industry Most valued student learning outcomes = Synthesis Assessment Framework

16. Planning Process: Track 2 MIB writes scenarios Validating the Outcomes Scenarios edited, reviewed by MIB for accuracy Scenarios analyzed to ID activities related to SLOs Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF

17. Track 2: Validating with Scenarios Scenario • a description of common or critical situations faced by an engineer • actual past, present, or future situation experienced by the MIB member • includes the context and actions of the engineer

18. Validation • approximately 80% overlap • Differences • ability to estimate resources (costs, time) • ability to supervise others • ability to take risks • ability to deal with ambiguity • ability to decide if project is worth pursuing

19. Assessment Planning Process Track 1: Identification Track 2: Validation ALL stakeholders define & articulate Student learning outcomes (SLOs) & classroom activities related to Synthesis goals MIB writes scenarios Scenarios edited, reviewed by MIB for accuracy SLOs reduced & refined Scenarios analyzed to ID activities related to SLOs SLOs reviewed by ALL stakeholders - draft of Synthesis Assessment Framework (SAF) Draft SAF compared to scenario activities Overlap between SLOs & activities = MOST VALUED OUTCOMES = Final version of SAF

20. 3) Exposure to Synthesis over time 1) Institutional effect 2) Individual course & faculty pedagogy effect on learning outcomes • Course portfolio • Course • questionnaire • Degree map • Syllabi • Track learning • outcomes • assessments • Student • portfolios • Alumni • interviews • Self/peer • assessment • Design project report • Scenario • assignments • Audio/video • observations Assessment Tools

21. Scenario-Based Assessment • Assignment developed from MIB scenarios • Scoring rubric based on SAF • Assignment & rubric tested by MIB • MIB responses established baseline expert score • MIB analysis of rubric used to refine measurement criteria

22. Faculty select from 1 of 4 scenarios Scenario Assessment Process Faculty administer pre & post test • Syn. HQ assess • holistic • analytic • Faculty evaluate • diagnostic • course grade faculty revise course dept/col - revise curriculum Syn. - longitudinal study

23. Synthesis - Lessons Learned • Collaboration created a bridge between industry & faculty regarding engineering education • Clarified what is/is not possible to teach & learn • validated measurable outcomes • Identifying & articulating learning outcomes valued by ALL stakeholders built commitment to the assessment process & results. • Vitalized assessment process for all stakeholders

24. Gateway Case Study

25. NJIT/NCE Assessment Statement Institute an assessment plan and process that moves from the current anecdotally based assessment to a formal, rigorous, valid, and useful assessment and continuous improvement process.

26. NJIT/ NCE Industrial Advisory Board Lucent Technologies PSE&G Foster Wheeler PrimeMedia SIAC Raytheon US Army Curtiss Wright CompUSA Becton Dickinson

27. Industrial Sub-Committees NJIT/NCE Advisory Board Strategic Planning Curriculum Review Assessment Gateway National Visiting Committee

28. Industrial Assessment Committee Charter To support the development of a comprehensive qualitative and quantitative curriculum assessment process that incorporates inputs and feedback from the employer and takes into account the fast pace of technological change.

29. Industrial Assessment Committee • Short-term objectives • Identify/prioritize the core competencies required for entry-level engineers • Work with faculty to identify educational requirements based on core competencies • Assess current curricula as to its effectiveness in providing students with the required core competencies • Establish employer-level metrics to measure graduate’s skill proficiency and provide a baseline for continuous improvement

30. Industry Partner Role • Participating member of faculty assessment team • Bring in assessment best practices from industry such as planning, TQM processes, metrics, multi-source feedback, assessment centers, etc • Form relationships between school and engineering/human resources organizations • Act as liaison with Gateway Coalition National Visiting Committee

31. Assessment Design Steps Step #1 Define Objectives, Strategies, & Outcomes Step #2 Identify Assessment Methods Step #5 Apply Results Continuous Improvement Step #3 Develop/Pilot Assessment Processes Step #4 Implement/Expand Assessment Processes

32. Focus Group Objectives • Identify outcomes, objectives & performance criteria for each course and program represented • Discuss existing assessment tools • Review potential use for continuous improvement Design Step #1

33. Curriculum Reform FE/FED - Freshman course in design & manufacturing General University Requirements Learning Tools Computer-aided learning Industrial Interaction Intern program Advanced Manufacturing Laboratory (AML) Seminars Series - “What Do Engineers Do?” Advance Design Engineering in Product Teams (ADEPT) Participating Programs

34. People Involved • 35-45 faculty members involved • Multidisciplinary - 3 of 4 colleges involved • Across all levels with enthusiastic involvement from tenured faculty • Covers courses & programs in all four years • Deans and department chairs • Members of NJIT/NCE Advisory Board

35. Learning Outcomes What critical knowledge, skills, and behaviors must students acquire in this course? • Question: Student will demonstrate an ability to actively participate, listen and collaborate with other team members while working on a technical design project. Example:

36. Analytical Thinking Communication Skills Creative Problem Solving Project Management Research Skills Self-Learning Systems Thinking Teamwork Technical Competence Results: 9 Core Learning Outcomes

37. Learning Outcome Definitions Analytical Skills • Applies logic in solving problems and analyzes problems from different points of views. Translates academic theory into practical applications and recognizes interrelationships among problems and issues.

38. Learning Outcome Survey NCE Learning Outcomes Instructor/Coordinator The purpose of this survey is to gather your ratings on the importance of each of the nine broad learning outcomes and their specific objectives as identified by you during the focus group sessions conducted recently by the NCE Assessment Team. Please review each statement carefully and rate it using the scale provided. In addition, you are provided with extra space to add and rate objectives within each broad learning outcome. Technical Competence Not at all Minor Moderate Considerable Critical How important are the following student important importance importance importance importance learning objectives to the satisfactory completion of your course or program: Demonstrates a basic knowledge of 1 2 3 4 5 fundamental engineering principles in the specific disciplines focused on in this course/program Integrates basic knowledge of other 1 2 3 4 5 engineering disciplines within the scope of the course’s project

39. Analytical Thinking (3.83, 3.52) Communication Skills (3.99, 4.62) Problem Solving (3.62, 4.20) Project Management (4.10, 4.44) Research Skills (3.82, 3.54) Systems Thinking (3.84, 4.65) Self-Learning (3.82, 3.22) Teamwork (4.23, 4.67) Technical Competence (3.61, 3.54) Ratings by Faculty and Industry Partners Other analyses show that learning outcome emphasis changes as one moves to upper division curricula.

40. Advisory Board - Post Review • Discuss/answer the following questions: • Think about an engineer that has recently entered your organization and is regarded as highly effective (up to speed quickly, little retraining, etc.) • Does he or she demonstrate the selected competencies? Give examples of how they demonstrate each selected competency. Would you add specific statements to the definition of a selected competency? • What performance criteria would you use to evaluate successful demonstration of the selected competencies in your organization? • Would you add any other competencies that new engineers should have based on your experience with the successful engineer above?

41. Other Outcomes • Creation of several competency-based assessment processes • Revised portfolio process for freshman design experience • Team Developer version for engineering learning outcomes • Senior Exit, Alumni, and Employer surveys • Development of new assessment planning process diffused to all Coalition Schools and beyond

42. Gateway - Lessons Learned • Industrial partners add value to assessment process, especially in providing a real-word perspective on educational objectives and learning outcomes • Process fostering comparison helps to promote new thinking • Important for sub-group of industrial partners to work closely with faculty throughout assessment cycle • Must keep all participants actively involved in planning and decisions

43. INPUT Fresh perspectives & real world insights to the assessment enterprise. New ways of thinking about education Commitment to improving education OUTCOME Vitalizes planning process & boosts faculty commitment/involvement to assessment Increased campus support for innovative teaching & assessment practices Industry partnerships focused on education in both sectors Conclusions

44. The Future of Collaboration Current research shows: • In most cases, industry kept to the periphery of assessment • Faculty & campus administrators fear industry partners will make unreasonable demands because they don’t understand the challenges facing higher education • Colleges do not know how to capitalize on the expertise of industrial partners • Assessment is still in fledgling stages The conclusions from case studies confirmed by Colleges who integrated industry into their assessment processes