1 / 37

University of XYZ

University of XYZ. ?? Programs to be evaluated ?? member team ?? observers, ?? co-chairs, ?? PEV’s We’ll need to stay focused!. Important Team Expectations. Evaluators represent the EAC of ABET We are accrediting programs, certifying that they satisfy the criteria

taariq
Download Presentation

University of XYZ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. University of XYZ • ?? Programs to be evaluated • ?? member team • ?? observers, ?? co-chairs, ?? PEV’s • We’ll need to stay focused!

  2. Important Team Expectations • Evaluators represent the EAC of ABET • We are accrediting programs, certifying that they satisfy the criteria • Team effort—team decisions • Confidentiality • Conflict of interest—every visitor should have signed a conflict-of-interest statement • Observers • no evaluative statements • exit interview—thank-you only

  3. Confidentiality Do not discuss conclusions with faculty, students, and others Keep all materials until the July 2010 EAC meeting. At conclusion of accreditation process (July 2010) materials are to be destroyed Information specific to the institution is to remain confidential without time limit Institutional data are confidential except with written authorization of institution ABET materials only released by ABET staff

  4. Communication Maintain open line of communication with the department head Identify deficiencies as soon as possible Discuss all issues with the department head at the debriefing Do not discuss the recommended accreditation action with anyone except team members

  5. General Visit Outline • Today – • Visit campus, review curriculum materials • Tonight share what we’ve learned, plan • Tomorrow – • AM: Dean’s overview, PEV’s in departments • Lunch with institutional officials • PM: Support areas, PEV’s in departments • Evening: Share what we’ve learned, tentative conclusions; draft exit statements • Tuesday – • Tie up loose ends • Draft exit statements by late morning • Debrief department heads before lunch • Exit meeting at 2PM

  6. Support Program Assignments

  7. Visit details • Important forms – done by Tues AM • Transcript & curriculum analyses • Program Audit Form + Explanation of Shortcoming (two hard copies to Team Chair—one will be left with Dean after Summary Meeting with Administration—one soft copy to Team Chair) • Exit Statement—soft copy to Team Chair • Program Evaluator Worksheet—soft copy to Team Chair • Form A2a - Short form—recommended actions—prepared by Team Chair on basis of information from evaluators • Program Evaluator Report—soft copy to Team Chair

  8. Terminology • Deficiency – criterion, policy, or procedure is NOTsatisfied. • Weakness – criterion, policy, or procedure is satisfied, but lacks strength of compliance to assure the quality of the program will not be compromised prior to next general review. • Concern – criterion policy, or procedure is satisfied, but potential exists for non-satisfaction in the near future.

  9. Working Definition of Key Terms • Deficiency: assigned to any criterion, policy, or procedure that is totally or largely unmet • Weakness: criterion, policy, or procedure is met to some meaningful extent, but compliance is insufficient to fully satisfy requirements • Concern: criterion, policy, or procedure is fully met, but there is potential for non-compliance in the near future • Observation: general commentary possibly, but not necessarily, related to criteria

  10. Limit Use of Key Terms Use Key Term only in reference to overall evaluation of each criterion The Key Term (defined in the previous slides) is the overall assessment for the criterion as a whole, not the worst finding among the sub-areas on the worksheet. Do not, for example, give a deficiency to a program that lacks only a measurement for outcome 3(e)

  11. Consistency Issues for the Team • Depth and completeness of the evaluation from program to program • Consistency across all programs in an institution • Assignment of appropriate key terms (deficiency, weakness, concern) to describe shortcomings • For weaknesses, consistency on interim recommendations—IR vs. IV

  12. Applicable Criteria • Use Criteria for 2009-2010 Accreditation Cycle. These are the same as the renumbered criteria used in 2008-2009 • If this is an interim visit, the criteria in effect at the time the prior shortcomings were identified should be used, unless it is to the program’s benefit to use the currently applicable ones, and it is the program’s choice.

  13. Application & Interpretation of Criteria In 2008-09, EAC editors observed that: • Issues arose related to the linkage between Criterion 2, Criterion 3, and Criterion 4 • Criterion 2 and new programs • Criterion 4 and new programs

  14. What Does Criterion 2 Say? The program must have in place: • Published PEO’s consistent with mission and these Criteria • Process that periodically documents and demonstrates that the PEO’s are based on the needs of the program’s constituencies • An assessment and evaluation process that periodically documents and demonstrates the extent to which these objectives are attained

  15. Criterion 2 Highlights • The process needs to document and demonstrate that the PEO’s are based on constituent needs – NOT “a process based on the constituents needs in which PEO’s are determined and evaluated” (the old language) • Requirement for assessment and evaluation is the same, but there is no longer language in Criterion 2 that requires that the results of the assessment and evaluation process be used for program improvement. (Program improvement is now in Criterion 4.)

  16. Consistency Issues Criterion 2: • Do the published PEO’s meet the definition? • Does the program convince the team that the PEO’s are based on constituent needs? • Notice that there is no language that insists on constituent approval or involvement! • The program does need to convince the team that it has a way of determining what the needs of its constituencies are. • Is there an assessment and evaluation process in place that gives info about the extent to which PEO’s are attained by grads

  17. Criterion 2 Problem in 08-09 • Several instances in which programs were being required to show how results of C2 or C3 assessment and evaluation processes were being used to improve the program. • This reflects application of old criteria. • That's not in C2 or C3 any more. C4 refers to results of C2 and C3 processes as possible sources of information upon which continuous improvement could be based.

  18. Criterion 2 FAQ’s • What if the PEO’s really sound like outcomes (instead of objectives? • If PEO’s are not PEO’s, there is a C2 shortcoming. • What if PEO’s are ambiguous or reflect outcomes retooled to apply after graduation? • Team judgment – do they meet the intent of the Criterion? • Is an assessment process for PEO’s that considers predominately data based on accomplishments of current undergraduates adequate? • Probably not

  19. What Does Criterion 3 Say? • The program must demonstrate that (a) – (k) are attained • Program outcomes are defined as (a) – (k) plus any additional ones articulated by the program • Program outcomes must foster attainment of the PEO’s • There must be an assessment and evaluation process that periodically documents and demonstrates the degree to which outcomes are attained

  20. Important – for Criterion 3 • The definition of program outcomes as being (a) – (k) plus locally articulated ones • The program may not have its outcomes expressed as (a) – (k) plus others. It may have just identified a set of outcomes. As long as the program has demonstrated attainment of (a) – (k) and its own outcomes, this element of the criterion is met. • Requirement for assessment and evaluation is the same, but there is no language in Criterion 3 that results of the assessment process be applied to further development of the program. (Program improvement is now in Criterion 4.)

  21. Criterion 3 Consistency Issues • Be sure to apply this criterion in a holistic sense • The process of assessment and evaluation needs to demonstrate the degree to which outcomes are attained, but … • There is no language that says all outcomes must be attained to the same degree • There is no language that says anything about a numeric scale measuring degree of attainment • There is no language that says the outcomes must be measured • There is nothing in Criterion 3 that says anything about use of the assessment and evaluation information for program improvement

  22. Criterion 3 FAQ’s • What about assessment data? What is adequate data? • Does it all have to be objective/direct? (NO) • Can it be subjective? (Some of it may be; nothing says it has to be) • Is the observation or conclusion of course instructor adequate? (What was his or her basis for the observation?) • Does evidence for each outcome have to be in the form of work the student has produced? (No, but the team needs to be convinced that outcome attainment has been demonstrated.)

  23. What Does Criterion 4 Say? • “Each program must show evidence of actions to improve the program. These actions should be based on available information, such as results from Criterion 2 and Criterion 3 processes.” • The improvements can be based on any available information!

  24. Consistency Issues • The language of Criterion 4 simply insists on evidence of action to improve the program. • Such actions could be stimulated by results of the C2 and C3 assessment and evaluation processes • But they could also be stimulated by other information • The language of this Criterion does not require that the C2 and C3 information be used as the basis for program improvement. It suggests use of the results of C2 and C3 processes as sources of information for program improvement.

  25. Deliverables • From each PEV, the team chair must have: • A recommended action relative to the program under review that is consistent with the team’s conclusions (remember, it is a TEAM recommendation) • A PAF that accurately reflects the team findings • An Exit Statement that will: • Be read verbatim in the Exit Meeting • Be used as the basis for construction of the draft statement to the institution

  26. Program Audit Form (for GR or New)(We provide a copy to the institution at the Exit Meeting) If heading doesn’t have the current visit year, you are using an old form! Do not use old forms!!

  27. Program Audit Form (for Interim Report Evaluations)We provide a copy to the institution at the Exit Meeting for IV only If the heading does not say INTERIM REPORT do not use it for an IR evaluation

  28. Program Audit Form (for Interim Visit Evaluations)We provide a copy to the institution at the Exit Meeting for IV only If the heading does not say INTERIM VISIT do not use it for an IV evaluation

  29. Exit Statement Format INTRODUCTION—USEFUL PROGRAM STATISTICS PROGRAM ISSUES Strengths (special, unique or particularly conspicuous strengths standing above the norm) 1. 2. Deficiencies (In order, only for those criteria where deficiencies exist) 1. XXX 2. etc. Weaknesses(In order, only for those criteria where weaknesses exist) 1. YYY 2. etc. Concerns(In order, where concerns exist) 1. ZZZ 2. etc. Observations (do not have to relate to criteria) 1. etc.

  30. Writing the Exit Statement For each shortcoming – use the following structure for your statement of finding: What is required: Describe what criterion or policy is applicable Describe what that criterion or policy requires What was observed: Describe what was observed Describe how your finding violates criterion or policy Negative Impact Describe the negative impact it has on the program (for D or W) or the current or potential future impact on the program (C)

  31. What does your TC need in the exit statement you write? • The language must be clear: • “There is inadequate assessment of outcome 3(j).” • How does the program know why its assessment of student knowledge of contemporary issues is inadequate? • “There is incomplete evidence that students attain outcome 3(i).” • What is it that is incomplete about the evidence that students have a recognition of the need for and the ability to engage in life-long learning. • “It appears that outcome 3(h) is not fully assessed.” • Why did the team find that the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context is not fully assessed? What does “full assessment” mean? • The program has to understand what the shortcoming really is in order to fix it, the TC has to be able to explain the team recommendation, and later teams have to understand the issues

  32. What Would Make it Clearer? • Instead of“There is inadequate assessment of outcome 3(j).” • The only mechanism used for assessment of student knowledge of contemporary issues is through administration of a survey instrument asking students whether they have knowledge of contemporary issues. This mechanism does not provide adequate information for determining the degree to which this program outcome is attained. • Instead of“There is incomplete evidence that students attain outcome 3(i).” • Although student grades in the “Introduction to the Profession” course were claimed as documentation and demonstration that students have recognition of the need for and an ability to engage in life-long learning, no clear links between the student grades and this outcome were established. • Instead of“It appears that outcome 3(h) is not fully assessed.” • Even though some anecdotal assessment of whether students do have the requisite broad general education was described in the self-study report, evidence shows only preliminary plans for the development of a systematic process for assessment and evaluation of this program outcome.

  33. The language must support the team recommendation • If it is a D, the words have to clearly say that the criterion is totally or largely unmet • “there is no evidence that” • “there is no assessment and evaluation process” • “not all students are required to engage in a major design experience”

  34. The language must support the team recommendation • If it is a W, the words have to clearly say that the criterion is met but that strength of compliance is lacking, and how it is lacking needs to be clear • “Assessment of outcomes 3(d), 3(f), 3(g), 3(h), and 3(i) appears to be ad hoc. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.” [supporting a recommended citation of a Criterion 3 weakness relative to strength of assessment and evaluation process] • “While the program has demonstrated that graduates are able to engage in most of the activities required by the Civil Engineering program criteria, the program has not clearly demonstrated that graduates are able to apply knowledge of physics, chemistry, and at least one additional area of science.” [supporting a recommended citation of a Criterion 9 weakness for a Civil program]

  35. The language must support the team recommendation • If it is a C, the words have to clearly say that the criterion is met and indicate exactly what leads to the concern about the potential for future noncompliance • “It is clear that all of the criterion 3 outcomes are being assessed, but evaluation of some of the assessment data appears to be inconsistent. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 3 concern] • “Although all of the transcripts examined provided evidence that students meet all graduation requirements, the processes by which graduation requirements are audited prior to graduation appears to be ad hoc. This leads to concern that future compliance with this criterion may be jeopardized.” [supporting a recommended citation of a criterion 1 concern]

  36. The PEV Competency Modeland its Application • Your performance as a PEV will be evaluated against the PEV Competency Model. You are expected to be: • Technically current • Effective at communicating • Interpersonally skilled • Team-oriented • Professional • Organized • The evaluation forms are found on the ABET web site http://www.abet.org/pev.shtml

  37. Last Words • Remember – this team is “the face of ABET” • The 4 big C’s go a long way in promoting a productive evaluation: • Courtesy • Consistency • Clarity • Confidentiality

More Related