700 likes | 867 Views
Outcomes Assessment 2 Program Assessment. Joseph A. Shaeiwitz West Virginia University joseph.shaeiwitz@mail.wvu.edu. Daina M. Briedis Michigan State University briedis@egr.msu.edu. Outline. ABET and engineering criteria Program objectives Program outcomes Assessment
E N D
Outcomes Assessment 2Program Assessment Joseph A. Shaeiwitz West Virginia University joseph.shaeiwitz@mail.wvu.edu Daina M. Briedis Michigan State University briedis@egr.msu.edu
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.
Initial Quiz • What do you know about ABET? • Describe the engineering criteria? • According to ABET’s definitions, what is the difference between outcomes and objectives? • Identify four assessment methods. Classify each as direct or indirect.
ABET and Engineering Criteria • ABET = Accreditation Board for Engineering and Technology • Engineering criteria changed to assessment basis (TQI) around 2000 • Must prove that students are achieving objectives and outcomes • measure output, feedback model • previously, feed forward model
Feed Forward Model curriculum input output assumed
Feedback Model education process one class one course graduate alumnus one class entering college one course
Determine educational objectives Determine Outcomes Required to Achieve Objectives Determine How Outcomes will be Achieved Evaluate/Assess Determine How Outcomes will be Assessed Input from Constituencies Formal Instruction Student Activities Establish Indicators that Objectives are Being Achieved The Two Loops of the Engineering Criteria
ABET and Engineering Criteria • 8 criteria • students • program educational objectives • program outcomes and assessment • professional component • faculty • facilities • institutional support and financial resources • program criteria
ABET and Engineering Criteria • Focus of this workshop • program educational objectives • definition • how to establish • how to assess • program outcomes and assessment • definition • how to establish • how to assess
Minute PaperClearest vs. Muddiest Point • What have you just learned about ABET and assessment? • What points are the clearest? • What points are the “muddiest?”
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Program Objectives* • “…broad statements that describe the career and professional accomplishments that the program is preparing the graduates to achieve.”* • “Our graduates will be successful …” • must define “successful” *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Program Objectives* • must be • detailed and published • include constituencies/periodically evaluated • educational program to achieve outcomes (defined later) and to prepare graduates for accomplishments that achieve objectives • ongoing evaluation to determine extent objectives attained, use results for program improvement *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Exercise • New program in nanobiomolecular engineering • Define two program objectives.
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Program Outcomes* • “…statements of what students are expected to know and be able to do by the time of their graduation.” • Outcomes must “…foster attainment of the program objectives…” • Process to produce outcomes • Assessment process, with documented results • demonstrating measurement • demonstrating degree of achievement • Evidence results used for program improvement *Criteria for Accrediting Engineering Programs, 2007-2008 Cycle, ABET, Inc., Baltimore, MD, http://www.abet.org
Program Outcomes • Minimum outcomes “a-k” plus program criteria • Opportunity to be unique, i.e., define unique outcomes, not just repeat “a-k” • Helpful to map program-defined outcomes into a-k • Helpful to map outcomes into classes • Necessary to map outcomes into objectives
Exercise • Define three unique program outcomes for the nanobiomolecular engineering program.
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Performance Criteria • What will students do to demonstrate achievement of outcome • Example: “An ability to communicate effectively.” (3g in ABET list) • What are attributes of effective communication?
Performance Criteria for Effective Communication • When making an oral presentation, students will • maintain eye contact • … • A written report will • follow a prescribed format • demonstrate proper grammar and punctuation • adhere to commonly accepted word usage
Performance Criteria Learning Outcome 1.Has knowledge of current technological issues related to XXX engineering and society 2. Is able to discuss major political and societal issues and their pertinence to XXX engineering Educational Objective Contempo-rary Issues Professionally responsible
Exercise • Define two or three performance criteria for one of the outcomes previously defined.
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review
Indirect vs. Direct Assessment • Indirect – based mostly on student self-evaluation • surveys • interviews • focus groups • Direct – by faculty or some other means of evaluation of student performance (advisory boards evaluate design projects)
Indirect vs. Direct Assessment • Indirect • necessary, but not sufficient provides uncalibrated snapshot • self-assessment not necessarily reliable • Terminology may be unfamiliar • Direct • necessary for quality assessment plan • not now for all, but new Program Evaluators are being trained to look for this feature • relies on faculty experience, expertise, and judgment
How do we know if the students have the requisite outcomes? • When thestudentsthink they do, based on student survey results • With direct evidence from student work • Evidence is the key to accreditation • Faculty evaluation of student work is the key to providing evidence
Assessment Measures • Primary assessment of student outcomes should be based on student work (direct) • e.g., student portfolios, student projects, assignments and exams, some employer surveys where skill is observed • Secondary evidence • Senior exit surveys, alumni surveys, employer surveys (qualitative evidence based on opinion), other • Combination of both methods – triangulation
Indirect Assessment Measures • Surveys • Interviews • Course satisfaction surveys
Direct Assessment Measures • End-of-Course Assessments • Targeted Assignments • Capstone Experiences • Capstone Exams • Portfolios
End-of-Course Assessments • Course should have objectives • perhaps set by department committee • Related to program outcomes • Assigned problems (assignments, exams, projects) each related to course objectives • Evaluation/reflection by instructor
End-of-Course Assessments • Advantages • quick and easy • assessment can be done in parallel with grading • Disadvantage • not comprehensive • no big picture • Opinion • a component of assessment plan
Targeted Assignments/Problems • Key assignments that relate to specific program outcomes • Multiple assignments per outcome recommended • Integrate through curriculum • can demonstrate progress toward achievement of program outcome
Targeted Assignments/Problems • Advantages • quick and easy • assessment can be done in parallel with grading • Disadvantages • none really • need consistent evaluation method with reliable inter-rater reliability
Capstone Experiences • Can be design, laboratory, research • All programs have them • Where students are supposed to demonstrate and synthesize what learned • Usually includes teamwork, communication
Capstone Experiences • Advantages • already required in program • assessment can be done in parallel with grading • Disadvantages • none really • need consistent evaluation method with reliable inter-rater reliability
Capstone Exams • FE Exam • detailed, subject-related results available • Department-generated
Capstone Exams • Advantages • FE is standardized • the most direct measure possible • Disadvantages • FE may not set bar as high as some want • students may not take department-generated exam seriously if results do not impact grades or graduation
Portfolios • Collection of student work demonstrating outcomes • Must also evaluate the portfolio • Can also have students reflect on work in portfolio
Portfolios • Advantage • comprehensive • Disadvantages • portfolio evaluation is additional work • need consistent evaluation method with reliable inter-rater reliability
Other Direct Assessment Methods • Journals • Concept maps • Oral presentations with follow-up questions (like M.S./Ph.D. defense)
Exercise • Select an assessment method to be used for direct assessment of program outcomes previously defined.
Outline • ABET and engineering criteria • Program objectives • Program outcomes • Assessment • performance criteria • assessment measures – direct and indirect • rubrics • Review