ASSESSMENTS AND CULMINATING EXPERIENCES

1. INSTITUTIONAL WORKSHOP ASSESSMENTS AND CULMINATING EXPERIENCES Mr. Bill Kohlruss-Dr. Mark Sherman CO-CHAIRS, GUIDANCE COMMITTEE FEBRUARY 21, 2013

2. ASSESSMENTS ANDCULMINATING EXPERIENCES Where Have We Gone Astray? Based on Feedback From Accreditation Visits…

3. ASSESSMENTS ANDCULMINATING EXPERIENCES COMPILATION OF RECOMMENDATIONS February 2009 to July 2012 Recommendations related to: • Program Educational Objectives – 48 • Culminating Experience – 23 • Assessment – 19 • Outcomes – 14 • Safety – 11 • Industry Advisory Boards/Committees – 10 • Curriculum – 8 • Faculty – 6 • Facilities, Equipment & Services – 4 Thanks Ceci & Vic

4. ASSESSMENTS ANDCULMINATING EXPERIENCES Number ONE on the list (48 Hits) is Program Educational Objectives ! Program Education Objectives was addressed in depth at the Workshop last February and we are addressing that weak area through a change in wording in the Criteria… Now called “Program Mission and Educational Goals”. New terminology, perhaps easier to understand ?

5. ASSESSMENTS ANDCULMINATING EXPERIENCES So… Number 2 and 3 are on the agenda for today. #2 Culminating Experiences. #3 Assessments.

6. ASSESSMENTS ANDCULMINATING EXPERIENCES To put a POSITIVE SPIN on today’s workshop… We have asked Dr. Beth Bjerke and Dr. Mark Dusenbury to do a presentation on “How They Assessed the Programs at UND”. This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

7. Beth Bjerke and Mark Dusenbury

8. Assessment “Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” (Palomba and Banta, 1999, pg. 4)

9. Assessment Process • Establishing your Program Mission • Defining your Program Educational Goals • Defining your Student Learning Outcomes (AABI General, Core and Program) • Identifying your Techniques and Target Groups • Data Collection • ‘Closing the Loop’ process • Implementing changes to improve student learning • Assessing the Assessment Process!

10. Techniques and Target Groups • Techniques • Direct Assessment of Student Learning • Written Exams • Performance Evaluations • Assignments • Target Group – Students • Indirect Assessment of Student Learning and Program Quality • Surveys • Target Group - Students, Alumni, Industry • Interviews/Focus Groups • Target Group - Students, Alumni, Industry

11. Survey Data

12. Designing Surveys • Online Survey Tools • Easy to use and abuse • Set up a timeline and stick to it • Have a purpose for the survey • Share results with faculty and students • Use to improve the learning experience

13. Gather Indirect Perception Data(AABI a-k from Students)

14. Gather Indirect Perception Data(AABI a-k from Alumni)

15. AABI General Outcomes

16. Closing the Loop with Survey Data • Take the time to have the conversation with faculty • Make recommendations and action items • Document the conversation!

17. Interviews and Focus Groups • Students • Have a lot to say, and many great ideas • Focus Group Discussions • Graduating Senior Interviews

18. Interviews and Focus Groups • Industry/Alumni • Use of Advisory Board • Other Sources • Campus Visitors • Document!

19. Closing the Loop with Interviews/Focus Groups • Take the time to have the conversation with faculty • Make recommendations and action items • Document the conversation!

20. Direct Learning • Course Prime Reports • Stage Check Analysis • Capstone

21. Course Mapping

22. Course Prime Reports

23. Stage Check Analysis

24. Aviation 222 – Instrument Aviation 415 - CFII

25. Capstone • Writing analysis between majors in capstone courses • Infant stages – small sample size – will get better with time • Food for thought at this time ATC-90% AVIT-81% UAS-92%

26. Techniques Timeline

27. Faculty Outcome Discussions

28. Most important Aspect to Assessment… Using what is learned through the process to improve the student learning experience in your program!

29. ASSESSMENTS ANDCULMINATING EXPERIENCES To address our second topic we have Ms. Andrea Georgiou and Mr. Paul Carlson doing a presentation on “Our Capstone/ Culmination Experience” at MTSU. This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.

30. CULMINATING EXPERIENCES Middle Tennessee State University Andrea M. Georgiou Paul R. Carlson FOCUS Lab Project

31. Project Background CULMINATING EXPERIENCES • FOCUS concept first proposed in 2009 by Dr. Paul A. Craig • Received NASA grant in 2010 to study team decision-making • Project would address best practices for communication, disruption management, teamwork, and group situational-awareness • Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate students • Students began with tabletop discussions and now has evolved to include: • Complex and real-world scenarios • Immediate feedback and ramifications to decisions • Real-time performance analysis • In-depth after action reviews

32. Project Concept CULMINATING EXPERIENCES • All concentrations take aerospace core classes Seminar + Simulation • MTSU AEROSPACE program comprised of 5 concentrations: • PROFESSIONAL PILOT • MAINTENANCE MANAGEMENT • AEROSPACE ADMINISTRATION • FLIGHT DISPATCH • TECHNOLOGY Technology Flight Dispatch Maintenance Management Pro Pilot Admin Aerospace Core General Education Core • After specializing in their chosen concentrations, students return to take a capstone course • Prior to FOCUS lab, capstone course only addressed industry topics, résumé, and interview preparation

33. Project Concept CULMINATING EXPERIENCES • Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand their role and impact within an organization. “On average, it usually takes our employees 10 years to fully understand the big picture; how the decisions they make impact others in the company.” • Research revealed that new aviation professionals lacked: • operational understanding of others’ roles in the organization • basic understanding of how their decisions and job performance impacts others • the ability to communicate and coordinate effectively with others to meet organizational objectives • the use of foresight, planning, and time management in making decisions and performing tasks —Chief Ramp Tower Coordinator, FedEx

34. Project Concept CULMINATING EXPERIENCES • The FOCUS project is specifically designed to address teamwork deficiencies of new employees in the aviation industry • Students are placed in a realistic simulated airline flight operations control center • Here, students from all aviation disciplines are placed in teams to practice working with others

35. Project Concept CULMINATING EXPERIENCES • Teams work to meet organizational goals like safety, on-time performance, customer satisfaction, and disruption management • Teams are given various scenarios (or triggers) that require the students to work together to resolve issues quickly and effectively • Teams are monitored and evaluated by graduate student and faculty observers

36. WHAT IS FOCUS? CULMINATING EXPERIENCES • FOCUS is a high-fidelity replication of a part 121 regional airline operations control center • Students participate in a 3 hour simulated work shift working for a virtual airline we call “Universal E-Lines” • Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout the southeastern United States. • Utilizing a hub and spoke system, Nashville and Jacksonville act as the airline’s hubs • Students manage up to 80 flight events in a 3 hour simulation session F O C U S light perations enter nified imulation

37. Project Concept CULMINATING EXPERIENCES

38. CULMINATING EXPERIENCES

39. Photos CULMINATING EXPERIENCES

40. Project Concept CULMINATING EXPERIENCES • There are 10 distinct positions in the FOCUS Lab. They are: • Flight Ops. Coordinator • Weather & Forecasting • Crew Scheduling • Fuel/Cargo Management* • Flight Tracking & Schedule Management* • Maintenance Planning • Maintenance Control • Ramp Tower • Pseudo Pilot • CRJ Simulator Crew Flight Operations Coordinator MX Control WX & Forecasting *Flight Ops Data MX Planning & Scheduling Crew Scheduling CRJ Pilot Crew Pseudo Pilot Crew Ramp Tower

41. Photos CULMINATING EXPERIENCES

42. Photos CULMINATING EXPERIENCES The MTSU and its industry partners pioneered the ability to electronically link to a CRJ-200 simulator located at the flight school. Pilots in this simulator can communicate with anyone in the operations center. The CRJ simulator even appears on flight tracking monitors and is displayed in the ramp tower as a photo realistic aircraft moving about the airport environment.

43. Photos CULMINATING EXPERIENCES

44. Photos CULMINATING EXPERIENCES

45. Scenario Design & Control CULMINATING EXPERIENCES • Scenario Design & Control: • Graduate Students • Monitoring team performance and status of the operation • Provide real-world feedback and consequences • Issue disruptive ‘triggers’ • Library • Taxonomy • Disruptive potential scale • Monitor response to triggers • Evaluate effectiveness

46. After-Action Review CULMINATING EXPERIENCES • The Aerospace Department collaborates with the Psychology Department on the FOCUS project. Studies include: • Occupational stereotypes & attitudes • Group dynamics • Individual & team cognition • After- Action Review: • Engages students to discuss events of simulation • Feedback from team members and observers • Discussion of unfavorable outcomes • How to prevent in the future? • Anticipating problems • Communication & time management • Reinforces positive behaviors and solutions • Teams establish goals and new strategies

47. Advantages CULMINATING EXPERIENCES • Active learning experience • Students provide performance feedback to peers • Has the potential to reveal holes in curriculum • Encourages faculty to incorporate a greater ‘real-world’ perspective in the classroom • Allows students to use knowledge gained from their coursework to creatively solve problems • High motivation to succeed and learn from mistakes • Students learn to consider multiple solutions to a problem, weighing likelihood, impact, and other downstream implications • Gives students a sense of time sensitivity while problem solving • Internship & job opportunities • Students enjoy it!

48. QUESTIONS CULMINATING EXPERIENCES t h a n k y o u Dr. Paul A. Craig Ms. Andrea Georgiou Mr. Paul R. Carlson with Dr. Glenn Littlepage Dr. Mike Hein Dr. Richard Moffett Mr. Gerald Hill paul.craig@mtsu.edu a.georgiou@mtsu.edu prc2h@mtmail.mtsu.edu glenn.littlepage@mtsu.edu michael.hein@mtsu.edu rick.moffet@mtsu.edu gerald.hill@mtsu.edu

49. ASSESSMENTS ANDCULMINATING EXPERIENCES • Thank You UND & MTSU • Culminating Experience • Assessment • Guidance Workshop July 2013 Kansas City