“ICON”oclastic Assessment:

1. “ICON”oclastic Assessment: Enhancing Group Learning Through Real-Time, Web-Based Feedback A. Rosenfeld, W. Ham, C. Dede, R. Leffert, J. Quattrochi ICON Program for Multidisciplinary Learning Harvard University, Cambridge MA 02138 Supported by the Harvard Provost Innovations in Instructional Technology Award

2. What is the Interactive Case-Based Online Network (ICON)? • A platform for case-based learning that allows experts in a field to interact with learners via a web-based simulation of real-life events. • In the current project, ICON was used to teach a neuroscience curriculum to small groups (n=8) of Harvard medical students. Guided by a faculty tutor, students evaluated a “virtual” patient with the help of expert clinicians and scientists who could be consulted via ICON.

3. ICON Interface Left panel: Communication Center For each case, the students select a team leader who is “on call.” The team leader is assigned a beeper for urgent patient updates. “Go Conference” is an instant messaging feature that allows for live discussions. Case characters (patient, consultants), faculty mentor, and students are listed. The green button indicates ICON users who are currently online. (Student names and photos are blurred to protect confidentiality.) A mailbox feature provides for internal administrative communications. Center panel: Case and Discussion Modules Right panel: Library Using the drop down menu on top (“Module”), the ICON user can access the current case and pertinent discussions. ICON is organized into several modules: “Case” contains introductory information (patient background, history, initial clinical examinations) that is presented to the students at the beginning of a new case. “Virtual Contact” is where students interact with case characters. “Brainstorm” is where students and the faculty mentor discuss new information, integrate their thinking, advance hypotheses, and develop problem-solving strategies. “Cornerstones” is where students present syntheses and analyses at the end of the case. Resources are updated in real-time. At the start of each case, the Library is empty. Faculty mentors add resources as the case progresses, helping to shape the students’ framework of thinking.

4. Assets of ICON • Facilitation of active learning: the case goes on 24/7 and students become more engaged with the realistic case. • Students ask and answer questions to and from the patient and their family (Virtual Contact module). • Students call consults to clinicians and researchers, from neurologists to radiologists to geneticists to healthcare policy experts, to assist them in caring for the patient (Virtual Contact & Brainstorm modules). • Students present relevant research and create conceptual maps to augment one another’s learning experience (Brainstorm, Library, & Cornerstones modules).

5. More assets of ICON… • Enhanced faculty-student partnerships in navigating clinical aspects of the case and bridging basic and clinical neuroscience concepts. • A continuous, collaborative learning environment that connects learning experiences in and out of the traditional classroom and allows real-time evaluation and reshaping of the learning environment by the faculty tutor using ICONfb (feedback).

6. What is ICONfb (feedback)? • In order to maximize the learning benefits of the ICON environment, faculty tutors require an instrument to analyze students’ continuous online interactions in real-time: • To assess strengths and weaknesses in students’ understanding of key concepts in a way that facilitates the adjustment of tutorial structure in real-time. • To allow course directors to modify case design and learning objectives to better convey key concepts.

7. ICONfb is an online system that allows faculty to rate and quantify students’ online contributions to the tutorial as the case progresses. Online discussions by the students for a case are presented in the Brainstorm and Virtual Contact modules of ICON. Each student discussion is classified into a specific sub-category: epidemiology, basic science, clinical science, patient safety, patient care, and teamwork.

8. 3. The faculty tutor then uses a simple rating tool to evaluate each posting for content related to a priori learning objectives and for the quality of such content.

9. RESULTS: • The initial evaluation of ICONfb demonstrates that students’ online discussions of hypothesis-driven, differential diagnoses and scientific concepts can be categorized and scored to accomplish two initial objectives:

10. 12 10 8 6 4 2 0 Assessment of tutorial coverage of pre-specified learning objectives (L1-L4) using a quantitative method with fair overall inter-rater reliability (>0.5) L3 L2 Cumulative Score L1 L4 Mon Tues Wed Thur Fri Sat

11. Assessment of students’ focus on different areas of the medical sciences throughout a case

12. Going Beyond Assessment • The goal of ICONfb is not simply assessment, but more importantly, student-centered enhancement of the learning environment based on assessment results. • Post-hoc analysis suggests that by actively altering case content, faculty can shift students’ attention to focus on specific topics and themes.

13. 12 10 8 L3 L2 6 4 Cumulative Score L1 2 L4 0 Mon Tues Wed Thur Fri Sat In the example below, students’ ICONfb scores for 4 learning objectives (L1-L4) progress at different rates during the week. Equipped with these data, the tutor can see on Wednesday that L1 & L4 were neglected in student discussions. This allows the tutor to redirect students toward these themes during the case. Awareness of divergence of student scores on learning objectives affords tutor the opportunity to redirect student discussion to address the discrepancy.

14. Instructional Feedback • The data gathered with ICONfb not only demonstrate student weaknesses, but also make apparent specific learning objectives not met across tutorials. • In this way, ICONfb prompted modification of the a priori case learning objectives and pointed to gaps in student understanding of key concepts that could be remedied via lectures, lab work, or other pedagogical modalities.

15. Limitations • Validity/reliability of assessment instrument not yet established. • Inter-rater reliability: data suggest inter-rater agreement can be high, but further evaluation of the ICONfb scoring metric is required. • Comparative assessment with control tutorial groups (groups not using ICONfb) are planned to determine the validity and effectiveness of ICONfb. • ICONfb creates large amounts of data. • Data collection and rating process to be streamlined and automated where possible.

16. Conclusions • ICONfb allows faculty tutors to assess the content areas and learning objectives covered by students in a tutorial, as well as providing a formative measure of the quality of student participation and understanding. • Faculty can use this information to actively reshape the tutorial to ensure adequate coverage of important topics, a more balanced case discussion, and demonstration of sufficient understanding of core concepts.

17. Contact Information http://icon.fas.harvard.edu Wendy Ham (Program Manager) Email:wendyham@gmail.com