Outcome Based Assessment Methods

1. Outcome Based Assessment Methods Adopted from ABET IDEAL (Institute for the Development of Excellence in Assessment Leadership) Workshop material (August 2009) By Wajih Abu-Al-Saud 8 December 2009

2. Outline • Classification of Assessment Methods • Assessment Methods • Written Surveys and Questionnaires • Exit and other Interview • Commercial Norm-Referenced Standardized Examinations • Locally Developed Exams • Focus Groups • Portfolios • Simulations • Performance Appraisals • External Examiners • Archival Records • Oral Examinations • Behavioral Observations Dr. Wajih Abu-Al-Saud

3. Classification of Assessment Methods Dr. Wajih Abu-Al-Saud

4. 1. Written Surveys and Questionnaires Definition Asking individuals to share their perception about the target being studied (such as their own skills or knowledge, or attributes or qualities of a program or course) by filling survey/questionnaire forms Dr. Wajih Abu-Al-Saud

5. 1. Written Surveys and Questionnaires Advantages • Provide accessibility to individuals who otherwise would be difficult to include in assessment efforts (alumni, employers) • Can cover a broad range of attributes within a brief period of time • Results are usually very clear even to a lay person • Typically produce students’ and alumni’s perspective of the institution (which lead to changes beneficial to relationship with these groups) Dr. Wajih Abu-Al-Saud

6. 1. Written Surveys and Questionnaires Advantages (when third-parties [parents, employers] are completing surveys) • Enable recognition and contact with important but often under-valued constituents (relations may improve by just asking for their input) • Offer different “less-biased” opinion then students • Can increase validity of results (through triangulation) • Provide unique and valuable stakeholder input (how is a course/program/college serving their purposes) Dr. Wajih Abu-Al-Saud

7. 1. Written Surveys and Questionnaires Disadvantages • Results tend to be highly dependent on wording and organization of items (good surveys are difficult to create) • May be biased because they often rely on volunteer participants • Mail or Email surveys tend to have low response rates • Commercially prepared surveys may not be entirely relevant to individual institutions • Multiple-choice surveys may not provide opportunities for respondents to express their true opinions Dr. Wajih Abu-Al-Saud

8. 1. Written Surveys and Questionnaires Disadvantages (when third-parties [parents, employers] are completing surveys) • Very difficult to anticipate and detect bias of third-parties • Expensive because of logistics (identifying sample, making contact, getting useful responses) • Very low response rates because of low investment by third parties in assessment process • If information about specific individuals is requested, confidentiality becomes an issue Dr. Wajih Abu-Al-Saud

9. 1. Written Surveys and Questionnaires Ways to Reduce Disadvantages • Use only carefully construct surveys that have been reviewed by survey experts • Include open-ended items along with multiple choice items • Use pilot samples and request feedback from respondents on clarity, sensitivity, and format of items • Obtain response from the larges sample size possible and follow-up with non-respondents (in person or by phone) • If using commercially prepared surveys, add locally developed items of relevance to institution • If using locally prepared surveys, add externally-referenced items (e.g. from surveys for which national data is available) • Attain consent in cases where information about specific individuals is being requested • Coordinate contacts with other organizations contacting the same group to increase response rates Dr. Wajih Abu-Al-Saud

10. 2. Exit (and other) Interviews Definition Asking individuals to share their perception through face-to-face dialogue about the target being studied (such as their own skills or knowledge, or qualities and attributes of a program or course) Dr. Wajih Abu-Al-Saud

11. 2. Exit (and other) Interviews Advantages (similar to written surveys but require direct contact limiting accessibility to certain populations) • Sometimes yield benefits other than data collection stemming from interaction with students • Allow more individualized questions and follow-ups questions based on responses of interviewees • Provide immediate feedback to interviewer • Can include a greater variety of items than surveys Dr. Wajih Abu-Al-Saud

12. 2. Exit (and other) Interviews Advantages (when third-parties [parents, employers] are being interviewed) • Provide unique and valuable stakeholder input (how is course/program/college serving their purposes) • Offer different “less-biased” opinion then students • Enable recognition and contact with important but often under-valued constituents (relations may improve by just asking for their input) • Can increase validity of results (through triangulation) Dr. Wajih Abu-Al-Saud

13. 2. Exit (and other) Interviews Disadvantages • Require direct contact, which may be difficult to arrange • May be intimidating to interviewees, thus biasing results in the positive direction • Results tend to be highly dependent on wording of items and the manner in which interviews are conducted • Time consuming, especially if large number of persons are to be interviewed Dr. Wajih Abu-Al-Saud

14. 2. Exit (and other) Interviews Disadvantages (when third-parties [parents, employers] are being interviewed) • Very difficult to anticipate and detect bias of third-parties • Expensive because of logistics (identifying sample, making contact, getting useful responses) • If information about specific individuals is requested, confidentiality becomes an issue Dr. Wajih Abu-Al-Saud

15. 2. Exit (and other) Interviews Ways to Reduce Disadvantages • Plan interviews carefully with assistance from experts • Provide training for interviewers that include guidance in putting interviewees at ease and related interview skills • Coordinate contacts with other organizations contacting same group to increase rate of acceptance for interviews • Conduct pilot testing of interview and request feedback from interviewee to improve interview process • Interview purposeful samples of students when it is not possible to interview all • Conduct phone interviews when face-to-face interviews are not possible • Utilize focus groups when individual interviewing is not possible or is too costly Dr. Wajih Abu-Al-Saud

16. 3. Commercial Norm-Referenced Standardized Exams Definition Group administered exams, mostly or entirely multiple-choice, “Objective” tests in one or more areas. Scores are based on comparison with a reference or norm group. Typically must be purchased from a private vendor. Used primarily on students in individual programs or courses. Dr. Wajih Abu-Al-Saud

17. 3. Commercial Norm-Referenced Standardized Exams Advantages • Can be adopted and implemented quickly • Objective scoring • Reduce/eliminate faculty time demands in development and grading (relatively low frontload and backload) • Provide for externality validity of measurements (degree to which the conclusions in your study would hold for other persons in other places and at other times) • Very beneficial for benchmarking and cross-institutional comparison studies Dr. Wajih Abu-Al-Saud

18. 3. Commercial Norm-Referenced Standardized Exams Disadvantages • Eliminate process of learning and clarification of goals and objective associated with locally developed tools • Unlikely to completely measure or assess specific goals and objectives of a program, department, or institution • May limit what can be measured • Norm-referenced data depends on comparison group and method of selecting students to be tested (most norm-referenced tests in higher education do not utilize randomly selected sample) • Multiple-choice tests generally include a high degree of error resulting from guessing correct answers • Results unlikely have direct role on program improvement • Either students or program have to pay for exam • If used repeatedly, faculty may teach to the exam Dr. Wajih Abu-Al-Saud

19. 3. Commercial Norm-Referenced Standardized Exams Ways to Reduce Disadvantages • Choose test carefully, and only after reviewing all test components and finding a satisfactory degree of match with the learning outcomes of the program • Request and review technical data, especially reliability and validity data and information on normative sample from test publishers • Insure that such tests are only one aspect of multi-method approach in which no firm conclusions based on norm-referenced data are reached without validation from other sources (triangulation) Dr. Wajih Abu-Al-Saud

20. 4. Locally Developed Exams Definition Objective and/or subjective tests designed by faculty of the program or course sequence being evaluated Dr. Wajih Abu-Al-Saud

21. 4. Locally Developed Exams Advantages • Content and style can be geared to specific goals and objectives • Local grading by faculty can provide relatively rapid feedback • More direct implication of results for program improvements • Greater faculty/institutional control over interoperation and use of results • Process of development can lead to clarification/crystallization of what is important in the process of student learning Dr. Wajih Abu-Al-Saud

22. 4. Locally Developed Exams Disadvantages • Require considerable leadership/coordination, especially during the various phases of development • Costly in terms of time and effort (more “frontloaded” effort for objective exams; more “backloaded” effort for subjective exams) • Cannot be used for benchmarking, or cross-institutional comparisons • Demand expertise in measurement to assure validity, reliability, and utility Dr. Wajih Abu-Al-Saud

23. 4. Locally Developed Exams Ways to Reduce Disadvantages • Join with other programs, departments, or institutions with similar goals and objectives as means of reducing costs of developing exams and adding externality at the same time • Utilize on-campus measurement experts whenever possible for test construction and validation • Incorporate outside experts and community leaders into development and grading process • Embed in program requirements for maximum relevance with minimum disruption (insert into capstone course), • Validate results through use of multi-method approach (triangulation) Dr. Wajih Abu-Al-Saud

24. 5. Focus Groups Definition Typically conducted with 6-12 individuals who share certain characteristics that are related to a particular topic linked to a research or evaluation question. Group discussions are conducted by a trained moderator with participants having the task of identifying trends/patterns. Moderator’s purpose is to provide direction and set the tone for the group discussion, encourage active participation from all group members, and manage time. Moderator must not allow own biases to influence discussion. Dr. Wajih Abu-Al-Saud

25. 5. Focus Groups Advantages • Useful to gather ideas, details, new insights and to improve question designs of tests • Helpful in the design of surveys and interpreting results of and getting more in-depth information on issues identified by a survey • Interaction among focus group participants often leads to new insights • Allows the moderator to probe and explore unanticipated issues Dr. Wajih Abu-Al-Saud

26. 5. Focus Groups Disadvantages • Groups can be difficult to assemble • Not suited for generalization about population being studied • Conducting effective focus groups requires training of moderators • Moderator has less control than in individual interviews • Differences in the responses between/among groups can be problematic • Data provided by focus groups can itself be difficult to analyze Dr. Wajih Abu-Al-Saud

27. 5. Focus Groups Ways to Reduce Disadvantages • Offer an incentive for participants if possible to increase participation • Train moderators to use open-ended questions, probes, and learn when and how to move into new topic areas Dr. Wajih Abu-Al-Saud

28. 6. Portfolios Definition Collection of multiple student work samples usually compiled over time and rated using rubrics. The design of a portfolio is dependent upon how the scoring results are going to be used Dr. Wajih Abu-Al-Saud

29. 6. Portfolios Advantages • Can be used to view learning and development longitudinally (samples of student work can be collected over time), which adds a very useful perspective • Multiple components of a curriculum can be measured at the same time • Economical in terms of student time and effort, since no separate “assessment administration” time is needed • Samples in portfolios generally reflect student ability more than test results when pre-planning • Avoids or minimizes “test anxiety” and other “one shot” measurement problems • Increases student participation in the assessment process Dr. Wajih Abu-Al-Saud

30. 6. Portfolios Disadvantages • Can be costly in terms of evaluator time and effort (management of the collection, and scoring process, including establishment of reliable and valid scoring rubric may be challenging) • If samples to be included have been previously submitted for course grades, faculty may be concerned that a hidden agenda of the process is to validate their grading • May not provide for externality • Security concerns may arise as to whether submitted samples are the students’ own work Dr. Wajih Abu-Al-Saud

31. 6. Portfolios Ways to Reduce Disadvantages • Consider having portfolios submitted as part of the requirements of a specific course requirements • Have more than one rater for each portfolio; establish inter-rater reliability through piloting designed to fine-tune rating criteria • Investigate the use of electronic portfolios as means to increase process efficiency • Utilize portfolios from representative samples of students rather than having all students participate • Provide training for raters • If portfolios are selected by students, understand that this is probably their best work Dr. Wajih Abu-Al-Saud

32. 7. Simulations Definition A competency-based measure where a person’s abilities are measured in a situation that approximates a “real world” setting. Simulation is primarily used when it is impractical to observe a person performing a task in a real world situation (e.g. on the job) Dr. Wajih Abu-Al-Saud

33. 7. Simulations Advantages • Better means of evaluating depth and breadth of student skill development than test or other performance-based measures • More flexible; some degree of simulation can be arranged for virtually any student target skill • For many skills, can be group administered, thus providing an excellent combination of quality and economy Dr. Wajih Abu-Al-Saud

34. 7. Simulations Disadvantages • For difficult skills, the higher the quality of simulation the more subjective the rating of student will be compared to standardized tests • Usually require considerable frontloading effort for planning and preparation • More expensive than traditional testing options in the short run Dr. Wajih Abu-Al-Saud

35. 7. Simulations Ways to Reduce Disadvantages • Limit complexity degree of simulation situations to those that can be rated more objectively • Administer group simulations as much as possible to reduce cost and increase number of skills being rated Dr. Wajih Abu-Al-Saud

36. 8. Performance Appraisals Definition A competency-based method whereby abilities are measured in most direct, real-world approach Dr. Wajih Abu-Al-Saud

37. 8. Performance Appraisals Advantages • Provide a more direct measure of what has been learned (presumably in the program) • Go beyond paper-and-pencil test and most other assessment methods in measuring skills • Preferable to most other methods in measuring the application of learning to specific setting and situations • Particularly relevant to the goals and objectives of professional training programs and disciplines with well defined skill development Dr. Wajih Abu-Al-Saud

38. 8. Performance Appraisals Disadvantages • Requires considerable time and effort (especially frontloading effort), thus being costly • Samples of behavior observed or performance appraised may not be typical, especially because of the presence of observer • Rating of student performance is typically more subjective than standardized tests Dr. Wajih Abu-Al-Saud

39. 8. Performance Appraisals Ways to Reduce Disadvantages • Conduct observations/appraisals in the least intrusive manner possible (e.g. using one-way observational mirrors, videotaping) • Conduct pilot-testing in which rate of agreement (inter-rater reliability) between observers/appraisers is determined • Provide training for observers/appraisers • Develop measurable criteria for observing and appraising performance • Cross-validate results with other measures Dr. Wajih Abu-Al-Saud

40. 9. External Examiners Definition Using an expert in the field from outside your program, usually from a similar program at another institution to conduct, evaluate, or supplement assessment of your students. Information can be obtained from external evaluators using many methods including surveys, interviews, … etc. Dr. Wajih Abu-Al-Saud

41. 9. External Examiners Advantages • Increases impartiality by having third party objectivity • Outsiders can see attributes to which insiders have grown accustomed • Students may disclose to an outsider what they might not otherwise share • Feedback useful for both student and program evaluation and provides valuable program consultation • May serve to stimulate other collaborative efforts between department/institutions • Evaluators may have skills, knowledge, or resources not otherwise available Dr. Wajih Abu-Al-Saud

42. 9. External Examiners Disadvantages • Risk of misfit between examiner’s expertise and expectations and program outcomes • Can be very costly and time consuming for individualized evaluations in large programs Dr. Wajih Abu-Al-Saud

43. 9. External Examiners Ways to Reduce Disadvantages • Share program philosophy and objectives with the external examiner and agree on assessment criteria before the assessment • Form reciprocal external examiner “consortium” among similar programs to minimize costs, swapping external evaluations back and forth Dr. Wajih Abu-Al-Saud

44. 10. Archival Records Definition Biographical, academic, or other filed data available from the college or other agencies and institutions Dr. Wajih Abu-Al-Saud

45. 10. Archival Records Advantages • Tend to be accessible, thus requiring minimal effort • Constitute non-intrusive measurement not requiring additional time or effort from students or other groups • Can be cost efficient if required data is readily retrievable in desired format • Build upon data collection efforts that have already occurred • Very useful for longitudinal studies • Good way to establish a baseline for before and after comparisons Dr. Wajih Abu-Al-Saud

46. 10. Archival Records Disadvantages • Especially in large institutions, may require considerable effort and coordination to determine exactly what data are available campus-wide and to then get that information in desired format (may require a lot of data mining) • To be most helpful, data from different sources may need to be combined. This may require the ability of designing good databases • Availability of data may discourage the development of other more appropriate measure or data source • May encourage attempts to find ways to use data rather than assessment related to specific goals and objectives Dr. Wajih Abu-Al-Saud

47. 10. Archival Records Ways to Reduce Disadvantages • Early-on in the development of an assessment program, conduct a comprehensive review of data typically being collected through the institution and its units • Only use archival records that are relevant to specific goals and objectives of learning and development to reduce demand on database design Dr. Wajih Abu-Al-Saud

48. 11. Oral Examinations Definition An assessment of student knowledge levels through a face-to-face dialogue between the student and an examiner (usually a faculty) Dr. Wajih Abu-Al-Saud

49. 11. Oral Examinations Advantages • Content and style can be geared to specific goals, objectives, and student characteristics of the program • Provide process evaluation of student thinking and speaking skills along with knowledge content • Dialogue format decreases miscommunications and misunderstandings, in both questions and answers • Allow measurement of student achievement in considerably greater depth and breadth through follow-up questions, probes, encouragement of detailed clarifications • Greater faculty/institutional control over interpretation and use of results • More direct implication of results for program improvements • Non-verbal reactions of students aid in interpretation of student responses • Local grading by faculty can provide immediate feedback Dr. Wajih Abu-Al-Saud

50. 11. Oral Examinations Disadvantages • Requires considerable leadership/coordination, especially during the various phases of development • Can be difficult to document by note-taking and providing student feedback with a grade • Costly in terms of time and effort (more “frontloaded” effort for objective exams; more “backloaded” effort for subjective exams) • May not provide for externality (objectivity associated with comparisons external to the program or institution) • Requires considerably more faculty time since oral exams must be conducted one-to-one or with a small group of students at most • Can be inhibiting on student responsiveness due to intimidation, face-to-face pressures, Oral versus written mode. May have similar effect on faculty Dr. Wajih Abu-Al-Saud