Reading the Tea Leaves: IDEA Results and Student Learning

1. Reading the Tea Leaves:IDEA Results and Student Learning

2. Goals: 1. To help faculty respond critically to student generated course evaluation; 2. To help faculty improve student learning. • Outcomes: Faculty participants will • Better understand the relationship between chosen objectives and the IDEA Diagnostic Form Report; • Recognize the connection between objectives and classroom teaching methods; • Have access to resources that will help them achieve greater student learning; • Understand additional evaluation tools. Goals and Objectives

3. [A]reflective analysis on the part of the instructor can be instrumental in promoting instructional improvement (Braskamp & Ory, 1994). In addition, the instructor is the only person who can supply certain kinds of information needed by those charged with making such evaluations, including information about course objectives; readings, assignments, and other learning activities; the creation of instructional materials or learning opportunities; procedures for appraising student achievement; results of, and course modifications based on, classroom research and other faculty efforts directed to improving instructional skills. All information in this presentation is credited to the IDEA Center

4. You will choose your IDEA Objectives after you have established your Course Learning Goals. Your course learning goals must be tied to your Major, College, and Degree goals. After those goals are set, you move to IDEA Objectives. The IDEA form measures student perception of your teaching specific to the goals identified for the course. Learning Goals

5. Most instructors will be unable to address more than 3-5 objectives • Select with these three questions: • Is this a significant part of the course? • Do I do something specific to help the students accomplish this objective? • Does the student’s progress on this objective affect his or her grade? Choosing Objectives

6. Essential/Important and the Score you get

7. 1. Gaining factual knowledge (terminology, classifications, methods, trends) 2. Learning fundamental principles, generalizations, or theories These objectives are mainly cognitive in nature as distinguished from affective objectives, which focus on feelings or attitudes. They are closely related. Both are concerned with the acquisition of information or knowledge. Stresses learning at its most basic level. What do the objectives mean?Idea’s note on selecting objectives

8. 3. Learning to apply course materials (to improve rational thinking, problem solving and decisions) 4. Developing specific skills, competencies and points of view needed by professionals in the field most closely related to this course The first of these emphasizes applications of principles, theories, and concepts to solve a problem or arrive at a decision. Course materials are employed to develop this general intellectual skill. It is appropriate in courses where applications are intended to develop problem-solving skills--analysis, synthesis, and evaluation. Only in courses where a primary goal is to develop a generalized skill in applying course materials to concerns or problems should Objective 3 be selected as "Essential" or "Important." Applying what has been learned to solve problems, make decisions, or perform specialized functions:

9. 6. Developing creative capacities (writing, inventing, designing, performing in art, music drama, etc.) 8. Developing skills in expressing oneself orally or in writing Creativity connotes originality, imagination, and expressiveness. While it is often associated with the fine arts and literature, it is often relevant to aspects of science, engineering, and other fields where design, research, and innovation are required. For this objective to be classified as “Essential” or “Important,” it should be a significant emphasis in the course and there should be specific instruction or assignments designed to promote its development. Expressiveness

10. 7. Gaining a broader understanding and appreciation of intellectual-cultural activity (music, science, literature, etc.) 10. Developing a clearer understanding of, and commitment to, personal values. 11. Learning to analyze and critically evaluate ideas, arguments, and points of view. This objective should be selected as important or essential if the focus of the course is on broadening intellectual background and increasing the breadth of students’ interests and appreciations. Such courses are usually directed to non-majors, although they may serve as introductory courses for majors as well. Intellectual Development

11. 9. Learning how to find and use resources for answering questions or solving problems. 12. Acquiring an interest in learning more by asking questions and seeking answers. This objective is especially appropriate for courses in which students have already acquired sufficient background information and skill to permit them to identify the trends or unresolved problems that are likely to form the basis for future advances in the field. Lifelong Learning

12. 5. Acquiring skills in working with others as a member of a team. Instructors who emphasize the development of team skills will note the overlap with several other objectives included on the IDEA list, including Objective 8 Communication Skills, Objective 3 Applications for Problem Solving, and Objective 6 Creative Capacities. If the expectation is that all of these skills will be developed, together with skills involving effective interpersonal relationships, then this objective should be identified as “Important” or “Essential.” Team Skills

13. Let’s Choose Some Objectives

14. Post Objectives to your syllabus; Add objectives on each assignment and exam; Emphasize the connection between the class discussion and the objectives and learning outcomes; During the Semester

15. Part 2: The Results

16. An overall index of teaching effectiveness (PRO=Progress on Relevant Objectives) combines ratings of progress on the objectives identified by the instructor as Important (weighted “1”) or Essential (weighted “2”) The IDEA Center regards this as its single best estimate of teaching effectiveness. Two additional overall measures of teaching effectiveness are shown on the report. These are the average ratings of two items using a 5-point scale 1. Overall, I rate this instructor an excellent teacher. 2. Overall, I rate this course as excellent. As an index of teaching effectiveness, the average of these two ratings is commonly regarded as about equal in value to the “Progress on Relevant Objectives” index described above. An Interpretive GuideClick for IDEA notes

17. Your report compares your average ratings to results for three different groups of classes. The first comparison group is with all classes in the standard IDEA database, and is always reported. The other two are reported only if enough classes were available to provide a stable basis for comparison. These consist of (1) all classes in the same discipline as the class in question and (2) all classes at your institution. Pay attention to the 5 point scale numbers. The converted numbers are useful only insofar as they measure your results against other faculty. They are not necessarily a measure of progress on relevant objectives. IDEA Diagnostic Form Report:Page 1

18. English 2325: Fall 2009

19. The first portion of Page 2 lists the 12 objectives included on the IDEA form and summarizes student ratings on those you selected as either Important or Essential. The main purpose is to help you focus your improvement efforts. Note the % does not include student’s who respond with a 3. Students described the class by comparing it to other classes they have taken in terms of (1) amount of reading, (2) amount of work in non-reading assignments, and (3) difficulty. Average ratings are compared with “All classes” in the IDEA database. Students described their motivation by making self-ratings on the three items listed at the bottom of Page 2. These characteristics have been found to impact student ratings of progress. Diagnostic Report--Page 2

20. Diagnostic Report—Page 2

21. The main purpose of instruction is to facilitate progress on objectives that the instructor selects as Important or Essential. The IDEA Center has conducted many studies that relate ratings on each of these “methods” to ratings of student progress on the 12 learning objectives. Through these studies, 7-10 methods that are most closely related to progress on each of the 12 objectives for classes of different sizes have been identified. Although there is some overlap, there are distinct differences in the methods that facilitate progress on the 12 objectives; there are also some differences among class sizes. The objectives for which a given method is especially helpful in promoting learning are identified in the column titled ”Relevant to Objectives.” The last column proposes an action for you to take, depending on the relevance of the item and how students rated it. If the rating for a relevant item was well above the IDEA average, it is described as a “Strength to retain”; if the rating was well below average, you are advised to “Consider increasing use”; and if it was in the average range, it is suggested that you “Retain current use or consider increasing.” Diagnostic Report--Page 3

22. Diagnostic Report--Page 3

23. Retain current use or consider increasing. Why? Only 59% of the students marked that I “inspired” them to set and achieve goals. If I focus on improving that element, I will improve their progress on all selected relevant objectives. IDEA recommends: IDEA Item #15 Connect Page 3 to Page 2

24. To improve their progress on Objective 27 and 31, I need to focus some attention on asking students to share ideas. IDEA Recommends: IDEA Item #16 What I find, based on IDEA data, is the two issues are linked. Example 2

25. IDEA tells us that page 4 is for “those who want to confirm scores reported on pages 1-3. We can also use page 4 to determine if our Objectives correlate to the student experience. I’m surprised my students focused on Developing creative capacities. It’s not marked as an Objective, but it is worth my time to examine the course to see why they saw it as important. Diagnostic Report—Page 4

26. Diagnostic Report--Page 4

27. How to Improve?

28. IDEA Notes on Learning(click to link to Idea notes)

29. IDEA Notes on Instruction(click to link to idea notes)

30. Mid-semester evaluations (coming soon) • Find out in March so you prevent it by May; • Short Answer Response Questions • Ask your students specific questions related to course organization, content, texts, etc. • Small group analysis • Colleague or CITR Director can visit a class to discuss progress on objectives and learning outcomes. Additional Evaluation Tools

31. The goal is to respond critically to the numbers, not use the numbers to be critical