Assessment Exploration Days: Assessing “Difficult” Outcomes: Ethics, Communication, Life-long Learning

1. Assessment Exploration Days:Assessing “Difficult” Outcomes:Ethics, Communication, Life-long Learning Julia M. Williams, Ph.D. Associate Professor of English and Coordinator of Technical Communication Julia M. Williams, RHIT

2. Workshop Agenda • Provide a framework for understanding strategies for assessing student learning outcomes • Use group processes to involve participants in writing learning objectives, performance criteria, and evaluation rubrics • Provide models of evaluation rubrics Julia M. Williams, RHIT

3. Your Goals for the Workshop • Warm up for work • 5 minute writing on two questions: • What is the most difficult outcome to assess and why? • What strategies have you tried in order to assess this outcome? • Name, department, email Julia M. Williams, RHIT

4. Assessment Terminology Sharing Vocabulary and Conceptual Frameworks Julia M. Williams, RHIT

5. Julia M. Williams, RHIT

6. Julia M. Williams, RHIT

7. Julia M. Williams, RHIT

8. Context: Subject matter Faculty member Pedagogy Student Facility Physical processes Thermodynamic Energy storage methods First law calculation Pressure-volume Phase changes Heat capacity Definition Processes Heating Curve Adiabatic Bomb Terminology Internal energy Work Heat Enthalpy Calorimetry Thermochemistry Assessment Focus: Evaluate individual student performance (grades) Evaluate teaching/learning Classroom Assessment Concepts Topics Subject Julia M. Williams, RHIT

9. Institutional Context Coursework & Curricular Patterns Classes chosen; major Out-of-classExperiences Student Precollege Traits Educational Outcomes Co-curricular; co-ops; internships; support services Classroom Experience Pedagogy; Facilities; Climate; Faculty & Student Characteristics ProgramAssessment Adapted from Terenzini, et.al. 1994,1995 Julia M. Williams, RHIT

10. How do classroom and program assessment differ? • Degree of complexity • Interpretation of results • Making improvements • Assessment tools available • Span of time • Level of specificity of the measure • Two sticks and a chain or a micrometer • Accountability for assessment process • Level of faculty involvement and buy-in • Cost Julia M. Williams, RHIT

11. Goals, Objectives, Performance Criteria: Institutional and Program Measuring Student Learning Julia M. Williams, RHIT

12. SDSMT Mission The mission of the South Dakota School of Mines and Technology is: • To prepare men and women for an enhanced quality of life by providing a broad educational environment which fosters a quality educational experience leading to baccalaureate and post-baccalaureate degrees emphasizing science and engineering. • To contribute to the expansion of knowledge through programs of basic and applied research, scholarship, and other creative endeavors. • To utilize the special capabilities and expertise on the campus to address regional, national, and international needs. Julia M. Williams, RHIT

13. SDSMT Objectives The principal objectives in support of this mission are: • To make the South Dakota School of Mines and Technology an outstanding undergraduate educational institution, enhanced by quality graduate education. • To enhance our national recognition as an educational institution with emphasis in science and engineering.• To continue to develop centers of excellence in research and graduate education using faculty expertise, and to further develop interdisciplinary research that involves faculty from several departments.• To create and continually ensure an environment which nurtures growth of the intellect, character, and spirit of students, faculty, and staff. • To build mutually beneficial partnerships with the broader community. • To increase significantly the resources available to the institution. This statement of mission and objectives serves as a framework for the continued growth of excellence at the South Dakota School of Mines and Technology. Julia M. Williams, RHIT

14. Civil Engineering The Department of Civil and Environmental Engineering at the South Dakota School of Mines and Technology has established the following goals for the Civil Engineering program: 1. Provide a quality undergraduate educational program that prepares the graduate for the practice of Civil Engineering; 2. Provide a progression of course work that prepares the student for entry into any graduate school for advanced training in the discipline of the student’s choosing; 3. Develop the student’s ability to maintain professional competency through continued self-study and advanced professional training; 4. Develop a professional attitude by encouraging participation in student activities of ASCE and stressing obtaining professional registration by emphasizing the need to take the Fundamentals of Engineering examination; and 5. Develop the student’s sensitivity to social and economic aspects of technical problems and of problem solutions that confront Civil Engineers. Julia M. Williams, RHIT

15. Electrical/Computer Engineering The undergraduate curriculum is designed to provide Electrical Engineering students with an education that is broadly based in the fundamentals of the profession so that they will be able to maintain a high degree of adaptability throughout their professional career. It is also intended that the student will develop a dedication to the profession and an ability to maintain professional competency through a program of lifetime learning. Design experience: team projects and often the team projects are multidisciplinary . . . provides students with a broad base of understanding that allows them to apply their knowledge of scientific and engineering principles to the practical and innovative solutions of existing and future problems . . . written and oral communication skills . . . to work well as a member of a team . . . social and ethical awareness so they understand their responsibility to protect both the occupational and public health and safety and to implement these factors in their professional activities. Julia M. Williams, RHIT

16. Environmental Engineering Graduates of this program are expected to be able to: 1. Apply mathematical, scientific, and engineering principles in conjunction with humanities and social sciences in definition and solution of existing or potential environmental problems. 2. Think critically in the iterative decision-making processes associated with engineering design. 3. Work and learn, on a lifelong basis, both independently and cooperatively with peers. 4. Communicate their work and ideas effectively, both orally and in written form, to their peers and at all societal levels. Julia M. Williams, RHIT

17. Mechanical Engineering OBJECTIVESWe realize that to build upon traditions of excellence, requires continual development of active partnerships among the faculty, the students, and our constituents. In keeping with these objectives, the mechanical engineering program produces graduates who are able to perform at a level that meets or exceeds industry expectations. Our students will be able to achieve the objectives listed below within a few years of graduation through attainment of the outcomes listed below at the time of graduation. Objective (1) Work effectively in an evolving engineering environment by: Outcomes • Possessing a solid foundation in engineering science and mathematics • Adapting to changing needs of management and society • Effectively managing multi-task assignments and working on multi-disciplinary teams Objective (2) Understand, learn, and apply evolving technology by: Outcomes • Applying modern engineering software and computational tools • Applying modern communication software • Applying modern data acquisition software and hardware Julia M. Williams, RHIT

18. Mechanical Engineering Objective (3) Communicate effectively in inter-disciplinary environments by: Outcomes • Applying effective written and oral communication skills • Understanding the dynamics of multi-disciplinary groups • Being aware of societal norms and engineering ethics Objective (4) Assume leadership roles by: Outcomes • Appreciating the importance of the business environment • Understanding and applying issues in planning and time management. • Understanding and applying the elements of engineering design • Recognizing the relationship between research, design, development, and manufacturing Objective (5) Practice life long learning by: Outcomes • Recognizing the importance of remaining current within your profession • Recognizing the relationship between the individual and his/her professional organizations • Understanding citizenship and the need to be involved in community outreach activities Julia M. Williams, RHIT

19. Mining Engineering Design experience built into the curriculum and enhanced by the use of sophisticated design software emphasizes the development and improvement of the following educational aspects and outcomes: • Creativity. • Problem-solving skills with the use of technology. • Writing skills. • Communication skills. • Leadership and team work. Julia M. Williams, RHIT

20. Julia M. Williams, RHIT

21. Easy? a.an ability to apply knowledge of mathematics, science, and engineering b. an ability to design and conduct experiments, as well as to analyze and interpret data c. an ability to design a system, component, or process to meet desired needs d. an ability to function on multi-disciplinary teams e. an ability to identify, formulate, and solve engineering problems Julia M. Williams, RHIT

22. f. an understanding of professional and ethical responsibility g. an ability to communicate effectively h. the broad education necessary to understand the impact of engineering solutions in a global and societal context i. a recognition of the need for, and an ability to engage in life-long learning j. a knowledge of contemporary issues k. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice Hard? Julia M. Williams, RHIT

23. Julia M. Williams, RHIT

24. Educational Goals/Objectives • Broadly stated • Desired outcome from student collegiate experience • As stated, is not a “measurable” objective • Need to create performance criteria that define student activities and are measurable: • Teams • Communication • Ethics Julia M. Williams, RHIT

25. Performance Criteria • Identify performance criteria for each objective • Performance criteria give further definition to the performance required • Answering the question: how will we know it when we see it? When working on a team, the student: • initiates and maintains task-oriented dialog • initiates and participates in group maintenance • strives for meaningful group consensus • works for constructive conflict resolution • supports other team members Julia M. Williams, RHIT

26. Developing performance criteria • Two essential parts • Action verb • Direct students to a specific performance • Content reference • Subject content for treatment in instruction Julia M. Williams, RHIT

27. Educational Practices/Strategies • Determine practices/strategies to achieve the objectives • Objectives should guide practices • during ‘orientation’ class, students are provided with team training; • ad hoc teams encouraged for in-class assignments; • class projects assigned to multi-disciplinary teams of students; • faculty receive development in use of teaming to enhance learning; • faculty work together to develop multi-disciplinary projects Julia M. Williams, RHIT

28. Environmental Engineering Graduates of this program are expected to be able to: 4. Communicate their work and ideas effectively, both orally and in written form, to their peers and at all societal levels. Julia M. Williams, RHIT

29. Performance Criteria:Acceptable standard of performance Effective oral communication When given the opportunity to present an oral report, the student: • Organizes the appropriate content concisely and logically • Presents a professional demeanor appropriate to the audience and situation • Speaks clearly and loudly as appropriate to conditions • Achieves rapport with the audience • Varies vocal tone and pattern • Effectively responds to questions and comments Julia M. Williams, RHIT

30. Assessment: Collection of Evidence • Specify assessment method(s) • Multiple strategies should be used when possible • --Peer evaluation forms, faculty evaluation forms • Determine timetable for data collection Julia M. Williams, RHIT

31. Evaluation: Interpretation of Evidence • Determine who is responsible for the collection of evidence • Determine who is responsible for the evaluation of the evidence • How is evidence going to used? • By whom is evidence going to be used? Julia M. Williams, RHIT

32. Feedback for Continuous Improvement • Determine feedback channels • Make feedback timely • Determine the audiences for feedback: students, faculty, administration Julia M. Williams, RHIT

33. Group Work: Class Assessment Developing Objectives, Measurable Performance Criteria, and Strategies Julia M. Williams, RHIT

34. Performance Criteria • What characteristics, skills, knowledge, attitudes, and/or values will the student exhibit so you will know he/she has achieved the desired outcome? • Multiple criteria for each objective • What does an accomplished student do when he/she performs this task effectively? Julia M. Williams, RHIT

35. Educational Strategies and Practices • What is done to provide students the opportunity to achieve the performance criteria? • Be specific: lab experiences, course activities, lectures • More than merely stating that a student does an oral report in your class • Presentation, feedback from instructor and peers, class lecture on presentation skills, models of effective presentations Julia M. Williams, RHIT

36. Team Formation(10 min) Purpose - To work together to develop performance criteria for a student learning objective Process - Form teams per instructions Julia M. Williams, RHIT

37. Team Assignment: Part 1 • Choose a learning objective that the team will focus on for plan development • Once the objective is chosen, develop the performance specifications using silent brainstorming and affinity process • (these processes are described on the following two slides) • After you have completed the affinity process, identify the themes of the groupings on the header cards Julia M. Williams, RHIT

38. Brainstorming Purpose - To generate a number of ideas in an analytical manner; to maximize individual involvement and commitment; to document team ideas in short period of time Process - Using “post-its,” write only one criterion per note (SEVEN WORDS or LESS); generate as many criteria as you can think of in the time allotted -- display all Post-its on wall near the team table Julia M. Williams, RHIT

39. Affinity Process Purpose - To organize a large set of items into a smaller set of related items Process - Team members silently move their Post-it cards around to form closely-related idea groups If disagreement exists when grouping, make copies of the contested card and place in more than one group Label each group with a “header card” which clearly identifies and reflects the theme of the cards If there are single idea cards that don’t fit well with the other ideas, have the team decide if they should be kept (they may be excellent ideas thought of only by one person) Julia M. Williams, RHIT

40. Results of Group Work • The objective • Multiple performance criteria • Refining performance criteria • Generating strategies • Linking strategies to multiple criteria Julia M. Williams, RHIT

41. Letting Students In On It! • Course objectives, performance criteria stated in syllabus • Educational strategies/practices linked to objectives during class activities • Discussion in class: what skills does each assignment contribute to, what will a student get out of it, and why Julia M. Williams, RHIT

42. Group Work Rubrics and Ratings: Assessing Student Learning with Portfolios Julia M. Williams, RHIT

43. Group Work: Rubrics • Reform groups • Select one performance criteria you developed earlier in the workshop • What evidence might a student select to make a case for this criterion? • Discuss features of performance associated with the criterion • What will a good one look like? • What kind of evidence is not acceptable? Example • Record on overhead Julia M. Williams, RHIT

44. Assessment Exploration Days:Assessing “Difficult” Outcomes:Ethics, Communication, Life-long Learning Julia M. Williams, Ph.D. Associate Professor of English and Coordinator of Technical Communication Julia M. Williams, RHIT