Peggy Brouse, PhD. Ariela Sofer , Ph.D. Systems Engineering and Operations Research Department

1. Measuring Outcomes and Objectives for ABET Accreditation in a Systems Engineering Undergraduate Program Peggy Brouse, PhD. ArielaSofer, Ph.D. Systems Engineering and Operations Research Department Vogenau School of Information Technology and Engineering George Mason University INCOSE 2007

2. 0utline Background Descriptions of the processes used to assess the outcomes of our program Measurement results of assessing our outcomes Descriptions of the processes used to assess the objectives of our program Measurement results of assessing our objectives A summary of noteworthy feedback gained through the assessment processes A list of improvements made as a result of the feedback INCOSE 2007

3. Background The Accreditation Board for Engineering and Technology, ABET, is an accreditation organization for college and university programs in applied science, computing, engineering, and technology ABET provides a sets of goals and objectives for accreditation and evaluates and certifies engineering related education The purpose of this INCOSE paper is to present the process used by the Systems Engineering and Operations Research Department at George Mason University to accredit our Systems Engineering undergraduate program with special emphasis on measurement of outcomes and objectives INCOSE 2007

4. Outcomes Processes [1] • Achievement of Program Outcomes is determined by the assessment of appropriate evidence supporting the department’s outcomes • samples of student work • surveys and focus groups of students • surveys of alumni and employers • informal interaction between faculty, students, alumni and industry representatives INCOSE 2007

5. Outcomes Processes [2] • Bulk of our assessment activity comes through the evaluation of students’ performance in courses • Samples of student work, in the form of exams, homework, project and laboratory reports, presentation and project evaluation, are collected and archived in Course Notebooks • Material from Course Notebooks is extracted and compiled into Outcome Notebooks for each of our 12 Program Outcomes • Each Outcome Notebook is reviewed and evaluated by a faculty member and a member of the SEOR Industry Advisory Committee • Recommendations are compiled, evaluated by the SEOR Undergraduate Curriculum Committee, and used as input for changes to the curriculum INCOSE 2007

6. Outcomes Processes [3] • Additional mechanisms by which Program Outcomes are assessed • Senior surveys are done at the School and University level in conjunction with the graduation application process. Results are subsequently provided to the program • A focus group interview is conducted at the end of the spring semester with students completing the senior design sequence • The University conducts course evaluations at the end of each semester and the results are provided to the department midway through the following semester • Informal feedback is obtained through the advising process, through ongoing informal contacts between faculty and students, and through informal contacts with outside constituencies in industry, government and academia INCOSE 2007

7. Outcomes Processes [4] The Systems Engineering Undergraduate Curriculum Committee (SUCC) reviews the evidence for accomplishment of the outcomes and the instruments used for collecting evidence Recommendations for changes in the BSSE Program are developed by the SUCC and considered for vote by the faculty as a whole Proposed changes are reviewed by the SEOR Department Industry Advisory Committee INCOSE 2007

8. Outcomes Processes [5] Mason SEOR Outcomes Process Table INCOSE 2007

9. Outcomes Measures • The measures used by the department to determine achievement of program outcomes are: • Course exams and homework exercises • Project and laboratory reports • Oral presentations given in class and evaluated using Presentation Evaluation Form • Results of group projects; Systems Engineering undergraduates have won several major national awards for their group projects in their Senior Design SYST490/495 class • Results from surveys of students and alumni • Responses from industry who serve as clients for our senior design project. Our SYST490/495 projects are sponsored by industry groups; they attend both the final SYST490 and SYST495 presentations • Responses from our advisory board; Advisory Board meetings have been conducted on a regular basis, feedback for the BS/SE program has been very positive INCOSE 2007

10. Outcomes Results [1] Average 2004/2005 Program Outcomes – Results (scale 1-7) 5.20 - Knowledge of mathematics through differential and integral calculus and advanced topics in differential equations, linear algebra, statistical analysis, and complex variables 6.60 - Knowledge of core systems engineering theory and methods 5.40 - An ability to use modern engineering techniques, skills and tools, including computer based tools for analysis and design 5.20 An ability to apply knowledge of mathematics, science and engineering to the analysis of systems engineering problems 5.80 - An ability to design scientific and engineering experiments 5.60 - An ability to conduct scientific and engineering experiments 6.40 - An ability to analyze and interpret data INCOSE 2007

11. Outcomes Results [2] Average 2006 Program Outcomes – Results (scale 1-7) 6.20 - An ability to identify, formulate, and solve engineering problems, including the planning, specification, design, implementation and operation of systems, components and/or processes that meet performance, cost, time, safety and quality requirements 6.40 - An ability to function on multi-disciplinary teams 6.40 - An understanding of professional and ethical responsibility 6.60 - An ability to convey technical material through oral presentation and interaction with an audience 6.00 - An ability to convey technical material through formal written papers/reports that satisfy accepted standards for writing style. 6.20 - A recognition of the need for and an ability to engage in life-long learning. 6.20 - The broad education and knowledge of contemporary issues necessary to understand the impact of systems engineering solutions in a global and societal context INCOSE 2007

12. Objectives Processes [1] • Program Educational Objectives are evaluated through a process that includes: • Collating and evaluating evidence from our alumni who have graduated three to six years from date of survey • Surveying the employers of this group of students • Evaluation of the objectives themselves and the degree to which they are being met by our educational program involves two components: • A global evaluation aimed directly at the objectives • A detailed examination using the mapping of program outcomes to the program educational objectives INCOSE 2007

13. Objectives Processes [2] The Program Educational Objectives are presented for comment and review to the students of the introductory freshman systems engineering course, SYST 101, and also to the students in the senior seminar, SYST 489 The Systems Engineering Undergraduate Curriculum Committee (SUCC) reviews the evidence for accomplishment of the objectives and the instruments used for collecting evidence Recommendations for changes in the BSSE Program are developed by the SUCC and considered for vote by the faculty as a whole Proposed changes are reviewed by the SEOR Department Industry Advisory Committee INCOSE 2007

14. Objectives Processes [3] Mason SEOR Objectives Process Table INCOSE 2007

15. Objectives Measures • The measures used by the department to determine achievement of program objectives are: • Feedback from alumni through information collected via questionnaires in conjunction with the Mason Alumni Office • Feedback from the employers of our alumni through information collected via questionnaires • Responses from our advisory board; Advisory Board meetings have been conducted on a regular basis, feedback for the BS/SE program has been very positive INCOSE 2007

16. Objectives Results [1] Results for SEOR graduates graduating from Mason in the past 3 to 5 years are summarized below •  How do you rate your current competency in: scale 5 choices (very unsatisfied to very satisfied) • a. Applying fundamental concepts of mathematics to contemporary and future systems Satisfied • b. Applying fundamental concepts of science to contemporary and future systems Very satisfied • c.  Applying fundamental concepts of information technology to contemporary and future systems Very satisfied • d.  Applying fundamental concepts of engineering to contemporary and future systems Very satisfied • e. Contributing to the development of systems using systems engineering methods and processed Very satisfied • f. Contributing to the development of systems using systems engineering models and tools Very satisfied INCOSE 2007

17. Objectives Results [2] • g. Working effectively as a member of multi-disciplinary teams Very satisfied • h. Knowing issues related to ethical behavior and professional responsibility Very satisfied • i. Possessing skills and establishing a foundation for life-long learning systems engineering and related areas Very satisfied • j. Communicating effectively with team members and decision makers orally Very satisfied • k. Communicating effectively with team members and decision makers in writing Very satisfied • l.  Overall ability as a Systems Engineer Very satisfied • m. Contribution of your Mason SEOR education to your overall ability as a Systems Engineer Very satisfied • n. Overall ability to be successful in your professional career Very satisfied • o.Contribution of your Mason SEOR education to your overall success Very satisfied INCOSE 2007

18. Assessment Feedback and Actions Taken [1] • SYST489, Senior Seminar, students are required to conduct a curriculum review. Some recommendations from the students and actions taken are: • Instruction regarding Configuration Management • Topic is covered in SYST489 but should be covered in more detail early in the curriculum • Faculty is considering where this might be included. • Increased communication between faculty and students • A concerted effort has been undertaken to increase communication via email contact, and more comprehensive advising INCOSE 2007

19. Assessment Feedback and Actions Taken [2] • Some recommendations from the students and actions taken are: • Review of the SYST101 course • An extensive review of SYST101 was undertaken and significant changes have been made. The evaluation of the new course content has been very favorable • Teach students how Systems Engineering is applied in the real world. • Significant progress has been made in this area both in the use of sponsored real-world problems in the SYST490/SYST495 Senior Design and in the use of guest lecturers from commercial firms in various classes including SYST489 INCOSE 2007

20. Assessment Feedback and Actions Taken [3] • Some recommendations from the students and actions taken are: • Allow for more and varied concentration tracks • Since this recommendation, new tracks have been added • It is also possible for students to design their own track with advisement from their academic advisor • Increased availability of career advisors for the students in the program. • This is an ongoing process • We have sponsored design projects wherein the students are in contact with commercial representatives • Students receive advice throughout their senior year INCOSE 2007

21. Assessment Feedback and Actions Taken [4] • Improvements and actions taken as a result of feedback from the Advisory Board: • Board recommended SEOR refer students to their companies • Undergrads with a 3.5 or more GPA will be considered for eligibility • Students will get tuition reimbursement and stipend • This is a very worthy recommendation. We have tried to facilitate the interface of students and local companies • Recruit our BS/SE students into the MS/SE degree • We have a joint BS/MS program that is recommended to qualifying students INCOSE 2007

22. Assessment Feedback and Actions Taken [5] • Improvements and actions taken as a result of feedback from the Advisory Board: • Increase number of required credits from 120 to 132 to include more domain information and preparation for the FE exam • The SEOR faculty is limited to 120 hours by school requirements • A review of capabilities necessary to pass the FE exam is being taught by faculty to students on Saturdays for weeks before the exam INCOSE 2007

23. Questions? INCOSE 2007