1 / 44

Faculty Use of Primary Trait Analysis to Assess Creative Thinking

Faculty Use of Primary Trait Analysis to Assess Creative Thinking. Sarah Murnen, Kenyon College Barbara Andereck, Ohio Wesleyan University (Nancy Grace, College of Wooster). Funded by the Teagle Foundation. Grant to the “Ohio-5” Colleges administered by the College of Wooster 2006-2009

brosh
Download Presentation

Faculty Use of Primary Trait Analysis to Assess Creative Thinking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Faculty Use of Primary Trait Analysis to Assess Creative Thinking Sarah Murnen, Kenyon College Barbara Andereck, Ohio Wesleyan University (Nancy Grace, College of Wooster)

  2. Funded by the Teagle Foundation • Grant to the “Ohio-5” Colleges administered by the College of Wooster 2006-2009 • “Creativity and Critical Thinking: Assessing the Foundations of a Liberal Arts Education” • Organized by Lori Bettison-Varga, former Associate Provost at the College of Wooster, and then Nancy Grace, Professor of English at the College of Wooster • http://www.wooster.edu/teagle

  3. How Do You Define Creativity?

  4. Starting Our Project • Year 1: • Assembled groups of faculty across four colleges in the Ohio-5 consortium of colleges to talk about evaluating creative and critical thinking among students • Developed survey for students and faculty, asked about traits defining critical and creative thinking

  5. Creative Thinking Critical Thinking Openness to Novelty Evaluation & assessment Idea Generation Elaboration Categorization & classification Curiosity Synthesis, integration & combination Invention Reasoning through logic Imagination Complexity Identification Abstraction & simplification Reasoning by metaphor & analogy Interpretation Awareness of Environment Divergent thinking Analysis Description Playfulness

  6. How Can We Best Assess Creative Thinking in Students?

  7. Primary Trait Analysis • Identify the “primary traits” – essential or central components of the discipline (both knowledge and skills) to be learned by the student • Build a scale for scoring the student’s performance on the trait • Evaluate the student’s performance against those criteria

  8. Primary Trait Analysis Resources: Presentation by Dr. Douglas Eder on “Primary Trait Analysis” Presentation by Barbara Walvoord on the use of rubrics for effective grading Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. Jossey Bass: San Francisco, CA.

  9. Stages in Constructing a Rubric From: Stevens & Levi (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing: Sterling, Virginia. • Reflecting – • What do we want for our students and from our students? (Learning objectives) • Why did we create the assignment? • What happened the last time we used the assignment? • What are our expectations for the assignment?

  10. Stages in Constructing a Rubric • Listing – Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment) • Grouping and Labeling the goals together - Organize the results of our reflections in first two stages, grouping similar expectations together in what will probably become the dimensions of the rubric • Application – Transfer groupings to a rubric grid; define assessment categories

  11. Group Discussion Rubrichttp://www.mashell.com/~parr5/techno/group.html Measures Emerging (1), Competent (2), and Exemplary (3) Performance on Each Trait LISTENING 1. Recognizes and responds to others speaking. 2. Uses and practices listening processes regularly. 3. Habitually uses listening processes. NON-VERBAL COMMUNICATION Eye contact, gestures, posture, facial expression, voice. 1. Comprehends some information from non-verbal cues. 2. Draws accurate conclusions from body language and facial expressions. 3. Able to recognize and use subtle non-verbal communication cues. CO-OPERATION 1. Sometimes shows ability to wait to give appropriate verbal / non-verbal responses. 2. Usually shows ability to wait to give appropriate verbal / non-verbal responses. 3. Habitually shows ability to wait with openness and awareness to give appropriate verbal / non-verbal responses. PARTICIPATION Tells thoughts, feelings, ideas so others understand. 1. Rarely talks during the discussion or talk is off the subject. Offers few ideas to the discussion. 2. Shares freely and explains with details. Makes connections to what others say. 3. Talk inspires others. Supports and leads others in discussion.

  12. Developing Rubrics through Primary Trait Analysis • Year 1: • Faculty developed rubrics to measure creative and critical thinking for their courses or departments • Year 2: • Taught other faculty to develop and use rubrics, and examined the rubrics they created along with student performance data from the rubrics • Developed a “generic” rubric using PTA designed to measure creative and critical thinking

  13. “GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING TRAITS 1. Elements of Argumentation a. Explanation b. Analysis c. Evaluation d. Interpretation e. Logic 2. Domain and Disciplinary Knowledge 3. Synthesis and Connections 4. Abstract Thinking 5. Complexity of Thought 6. Ideas Generated a. Fluency b. Flexibility

  14. “GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING TRAITS 7. Completeness/Coherence 8. Elegance 9. Divergent Thinking 10. Novelty or Uniqueness a. Germinal b. Original c. Transformational 11. Engagement 12. Risk Taking

  15. “GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING RATING SCALE DESCRIPTIONS 6 = Trait is evident to an exceptionally high degree 5 = Trait is evident to a high degree 4 = Trait is evident to an intermediate degree 3 = Trait is somewhat evident 2 = Trait is barely evident 1 = Trait is not evident

  16. What Assignments/ Pedagogies Promote Creative Thinking?

  17. Faculty Use Generic Rubric • Year 3: • Faculty across many disciplines measured student performance using the generic rubric in 31 classes (N = 444 students) across three of the participating colleges (College of Wooster, COW; Kenyon College, KC; and Ohio Wesleyan University, OWU)

  18. YEAR 3: KEY FINDINGS • Creative and critical thinking can be assessed in students, and we found improvement of performance on many creative and critical thinking traits among the students we studied (N = 444 students across 31 courses). • The traits measured most frequently across all disciplines were completeand originalmeasured in 90% of students, followed by knowledge measured in 85% of students. The traits least likely to be measured included abstract, logic, and novel. There were some differences across disciplines in the traits that were emphasized; for example, faculty teaching in interdisciplinary courses were more likely to examine complexity, and they shared with fine arts faculty a greater concern for engagement and risk.

  19. YEAR 3: KEY FINDINGS • In the longitudinal data measuring change in student performance across a semester, the traits where the most change was evident included complete, elegant, knowledge, and engaged; and the most change occurred among students in interdisciplinary courses. • In the cross-sectional data comparing less expert students to more expert, the most reliable differences were on the traits of analyze, logic, knowledge, and explain. There was no significant difference between the groups’ performance on abstract, fluent, flexible, elegant, divergent, novel, and risk. • Participating faculty who filled out an on-line survey indicated that they had discussed the use of the rubric with colleagues, and that the rubric helped them learn more about creative and critical thinking among their students

  20. Traits Measured in Classes by Discipline – All Data ALL FA HUM NSCI SOSC INT Explain 0.63 0.64 0.890.871.00 0.12 Analyze 0.54 0.44 0.890.87 0.72 0.12 Evaluate 0.44 0.00 0.54 0.73 1.00 0.12 Integrate 0.43 0.75 0.74 0.56 0.28 0.12 Logic0.40 0.44 0.12 0.71 0.72 0.12 Knowledge0.851.00 0.60 0.84 0.72 0.99 Synthesis 0.53 0.79 0.48 0.58 1.00 0.12 Abstract 0.38 0.00 0.12 0.71 1.00 0.12 Complex 0.62 0.00 0.63 0.58 0.55 1.00 Fluent0.63 0.45 0.74 0.58 0.72 0.65 Flexible 0.45 0.21 0.89 0.28 0.45 0.47 Complete0.900.89 0.63 0.861.001.00 Elegant 0.58 0.89 0.12 0.28 0.56 0.81 Divergent 0.52 0.21 0.48 0.14 0.72 0.81 Novel0.40 0.25 0.74 0.69 0.45 0.12 Original0.90 0.79 0.89 0.73 1.001.00 Transform 0.74 0.68 0.88 0.56 0.72 0.81 Engaged 0.72 0.89 0.12 0.56 0.72 1.00 Risk 0.59 0.75 0.48 0.14 0.28 1.00 _____________________________________________________________________ N Students 444 73 65 85 82 139 N Classes 31 5 6 6 8 6 _____________________________________________________________________ Note: High responses are noted in bold, low in italics. Number indicates proportion of students by discipline tested on the particular trait.

  21. Average Performance on Traits Across Time – Longitudinal Data *** indicates t value is significant at p<.001 revealing change in mean across time Note: High responses are noted in bold, low in italics. Trait M time 1 M time 2 tvalue Explain 3.85 4.52 7.26*** Analyze 3.764.73 7.84*** Evaluate 3.28 4.27 8.86*** Integrate 3.67 4.51 5.86*** Logic 3.764.78 6.85*** Knowledge 3.40 4.35 10.63*** Synthesis 3.774.63 8.50*** Abstract 3.53 4.58 9.76*** Complex 3.27 4.14 9.60*** Fluent 3.56 4.39 9.16*** Flexible 2.853.82 8.56*** Complete 3.56 4.41 10.98*** Elegant 3.03 4.05 10.83*** Divergent2.533.61 9.00*** Novel 3.19 4.22 5.78*** Original 3.19 4.01 9.40*** Transform2.233.31 9.22*** Engaged 3.56 4.48 10.58*** Risk2.803.55 7.60*** _____________________________________________________________

  22. Cross-Sectional data Scores on the traits could range from 1 to 6, with a higher score indicating better performance. Differences between groups were analyzed with independent sample t-tests. Trait M Less Exp M More Exp Difference? Explain 3.99 4.71 4.35*** Analyze 3.64 4.65 5.86*** Evaluate 4.18 4.70 2.51* Integrate 3.47 4.19 3.72*** Logic 3.87 4.80 5.28*** Knowledge 3.37 4.01 4.39*** Synthesis 3.67 4.61 4.32*** Abstract 4.26 4.56 ns Complex 3.15 3.71 3.04** Fluent 4.45 3.93 ns Flexible 3.92 3.59 ns Complete 3.60 4.16 3.52** Elegant 3.04 3.23 ns Divergent 2.87 3.21 ns Novel 3.38 3.67 ns Original 2.91 3.58 3.30*** Transformation 2.30 3.02 3.34*** Engaged 3.81 4.29 2.59** Risk 2.34 2.53 ns Note: ns indicates the t value is not significant; indicates significant at p<.05, **p<.01, ***p<.001

  23. Year 3 Interviews Participating Faculty • Twenty-two faculty who employed the “generic” rubric to measure creative and critical thinking among the students were interviewed about their experience • Interviewers asked questions about • the specific assignment used and whether it promoted creative thinking, • other assignments and pedagogies that might promote (or inhibit) creative thinking, • aspects of the discipline that might promote (or inhibit) creative thinking, • aspects of the college environment that might be important

  24. Year 3 Interviews Participating Faculty • Most of the faculty who participated chose an assignment they had used previously, and did not develop a new assignment for the grant. • Many of these assignments seemed designed to stimulate critical thinking more than creative thinking, but using a rubric that emphasized creative thinking increased the focus on these traits and likely increased the amount of creativity students showed. • In general, most people thought their assignments did promote creative thinking in students (and this was validated by the quantitative data gathered from the rubrics). However, the assignments people used for their data collection were not necessarily the most creative assignments the faculty had used or could imagine using.

  25. FACULTY INTERVIEWS YEAR 3: VALUE OF “ACTIVE LEARNING TECHNIQUES” • Take devil’s advocate position • Use problem solving • Students debate ideas • Use small group discussion • “Hands-on” science activities

  26. FACULTY INTERVIEWS YEAR 3: VALUE OF “ACTIVE LEARNING TECHNIQUES” One natural scientist spoke about student experience with creating their own experiments versus using “canned” labs. She said, “I didn’t know when I started doing this in my upper-level comparative class whether they would like it.... The experiments don’t usually work, because experiments usually don’t work, period. And, so, I would always say next year I am going to write labs. By the end of the class students come to my office to tell me emphatically, they write it on their evaluations, do not change the lab. This is what we want to do. This is what scientists do. I am always surprised by that… they always like it that way. Even if they have to put in more time. Evenings, weekends to feed their animals, take measurements…. the students like it better, it is better for them… thinking of something that has not been done but is doable is hard and you do have to be creative to do it.”

  27. FACULTY INTERVIEWS YEAR 3: VALUE OF “TAKING THE PERSPECTIVE OF OTHERS” • Similarly, another historian said, “I ask them a question that is surprising. It might be something like, ‘What does it smell like in 14th century Florence?’ A question we have not talked about at all, but they have done a great deal of reading from primary sources about markets in Florence, or about demography, or about domestic spaces and ask them to think about that evidence in some new sort of way and to extrapolate about the lived realities of 14th century people about things they know from records.”

  28. FACULTY INTERVIEWS YEAR 3: VALUE OF “TAKING THE PERSPECTIVE OF OTHERS” • A couple of other professors also mentioned types of “role playing.” One humanities professor said, “In my journalism class they do mock press conferences. They are given a topic – political topic - that is not a scandal. They have to draw on real-world knowledge, using realistic people who would appear at a press conference. Have to generate the story, present the information, and the rest of the class asks questions as reporters and they write the news story as journalists. Then the student presenting has to field the questions. They think outside, but within the box as well. They have to be active. I go back to the definition of [creativity as] ‘new but useful.’ “

  29. FACULTY INTERVIEWS YEAR 3: VALUE OF “STUDENT CHOICE” • One professor of an interdisciplinary course talked about the creation of a student-led senior colloquium. She said, “...second semester senior year giving seniors a term paper is just a death sentence. So I designed the senior colloquium as a way to get them to be more creative and to flow a little bit more with the material, rather than a research paper. The students design some of the colloquium itself, they get to pick the topic, they get to pick the readings and they get to determine the project that they will do collectively…. So for instance this year they did a magazine and I think they liked the opportunity to explore the genre of this sort of women’s magazine at the same time trying to make some sorts of statements about women and feminist issues that are important to them.”

  30. FACULTY INTERVIEWS YEAR 3: VALUE OF “PEER EVALUATION, INFLUENCE” • Many faculty talked about the importance of students engaging with their work, of “owning” their work. Faculty thought that to the extent that their peers would see their work, students would care more about it. The professor who had her students use “Moodle” to discuss their ideas publicly found that there was healthy competition among the students to present their unique voice which led to very creative expression.

  31. FACULTY INTERVIEWS YEAR 3:VALUE OF STUDENTS MAKING CONNECTIONS OUTSIDE THE CLASSROOM • A social science professor who has students do work outside the classroom said, “If they are out in the community, if they are working with a group of people, I think it gives them a way to think about an issue in a much bigger way… I mean you can look at the school violence issue, you can look at a bunch of material about school violence and have good book knowledge of it and the way that boys and girls interact in high school, but when you go to a football game and you watch it happen, you see these little kids pushing each other around and you see the bullies… it makes them come back with different questions, different ways to think about what they read.”

  32. FACULTY INTEVIEWS, YEAR 3: VALUE OF “INTERDISCIPLINARITY” • At Ohio Wesleyan the “National Colloquium” is a way to promote interdisciplinary thinking and discussion of issues outside of the classroom. One of the professors brought this up and said, • “There’s a different topic every year, a major national or international topic, and they bring in speakers and we try to bring tie in other events on campus and stage a play that focuses on that topic. Students will receive partial credit for attending the lectures or the performances or participating in activities which have something to do with the topic of the National Colloquium and one year it was about the arts. One of my colleagues raised the topic of having the National Colloquium for the arts and students could get other class credits for attending art openings or jazz band performance…. Making it part of the curriculum kind of forces the students to take advantage of things that are available to them… but that kind of thing is what I think educational institutions could do.” 

  33. Where does Creativity Happen on Campus?

  34. Summary Year 2 Survey Data Surveys were administered to 260 first-year students, 375 senior students, and 147 faculty across the four colleges during the 2007-2008 academic year to assess perceptions and experiences. Out of a wealth of information yielded by this research, a few interesting items are described below. 1.There seem to be more opportunities for critical than creative thinking in classes. 2. Class projects are evaluated by students as a means to engage in both critical and creative thinking. 3. Our campus environments were rated favorably with respect to the potential for creative thinking, although faculty rated “idea time” and “risk taking” as somewhat limited in their experience.

  35. Summary Year 2 Survey Data 4. Women students reported more experience with both creative and critical thinking and more positive attitudes about creativity compared to men students. 5. When asked to write about their most creative experiences, 61% of faculty mentioned their own research. Teaching and collaboration with students were each mentioned by 32% of the faculty. 6. When asked about barriers to creative thinking on campus, the most common response of senior students was that class pedagogy was a barrier (40% of students mentioned this); while faculty most often wrote about time as a barrier (18% of faculty). 7. A diverse campus environment (in terms of class, race, sex, sexual orientation, and national origin) is perceived to facilitate creative thinking.

  36. YEAR 2 SURVEY DATA:FACULTY AND STUDENT PERCEPTIONS OF CREATIVE ENVIRONMENT CHARACTERISTICS Participants rated the extent to which each characteristic existed on their campus: 1 = not at all, 7 = extensively Faculty 1st Yr Seniors Item 5.02 5.50 5.31* Challenge 5.08 5.50 4.94* Freedom 4.69 5.30 4.90* Idea Support 4.45 5.28 4.88* Trust/Openness 4.65 5.11 4.89* Dynamism/Liveliness 4.50 5.28 5.05* Playfulness/Humor 4.72 5.23 5.32* Debate 3.83 4.79 4.36* Risk Taking 3.73 4.80 4.45* Idea Time 4.14 4.28 4.56* Conflict 4.67 5.40 4.99* Supportive Environment 4.86 5.00 5.04 Working in groups 4.58 5.14 4.98* Active models of creative thinking 4.88 5.22 5.10 Assignments encourage independent problem- solving & risk-taking *indicates significant difference between groups

  37. YEAR 2 SURVEY: WHERE DOES CREATIVE THINKING HAPPEN ON CAMPUS, ACCORDING TO STUDENTS? Students rated how much each event involved creative thinking, rating 1 = low to 5 = high CREATIVE THINKING First Year Seniors Engaging in a hobby 4.18 4.13 Completing projects for class 3.74 3.78 Writing papers for classes 3.74 3.68 Extra-curricular or co-curricular activity 3.45 3.55 Interacting w/ students outside of class 3.51 3.52 Cultural events on campus 3.19 3.18 Interacting w/ faculty outside class 3.02 3.14 Interacting w/ students, faculty in class 3.02 3.04 Listening to speakers on campus 2.78 2.74 Communicating w/ peers internet 3.03 2.73* Reading material for class 2.83 2.61* “Surfing” the internet 2.63 2.59 *indicates significant difference between groups

  38. Student Ideas on the Encouragement of Creative Thinking, Focus Groups Year 2 • Professors allow students to give input (open-minded professors); discussion-based classes • Bringing in outside artists, bands, speakers, poets to “stir the pot” • Open-ended assignments in classes • Opportunities to connect work in different courses • Classes with real-world applications • Need more time • Student-themed housing • Classes where professors challenge students “to the edge” • Comfortable spaces where students can come in and “do their own thing” • Lunch tables with faculty • Diversification and diversity requirements encourage students to learn something new • Classes that require internships, service learning (first-year seminars with these components were praised) • Study abroad • Senior research project (projects in general); • Fewer required courses would mean more freedom for students; Any way for students to propose their own classes?

  39. Faculty Ideas on the Encouragement of Creative Thinking, Focus Groups Year 2 • General Comments. • Spaces for faculty to get together and discuss pedagogy • Fifty minute time periods are not long enough • Student-controlled spaces, events, organizations • Need to find time to live “the life of the mind” • Service learning that involves many members of the community • Senior seminars; Independent research project, summer science programs, senior projects; • Teaching abroad with students from your institution • Encourage diversity, appreciation for internationalism • Taking students to conferences, on field trips, etc. • Provide faculty with opportunities to do things differently – team-teach, time to do projects • Make interdisciplinary work easier, e.g., break down departmental autonomy • Allow faculty to remain “learners” as much as possible

  40. Faculty Ideas on the Encouragement of Creative Thinking, Focus Groups Year 2 • In the classroom. • Guest speakers • Use of Moodle and other forms of technology (that encourage engagement) • A final project with several options to the traditional paper (assignments encouraging active engagement) • Provide opportunities for students to interact with the “real world” • Setting up projects, assignments where outcome is unclear • Assignments that require “vulnerability” – students have to show something about themselves

  41. FACULTY INTERVIEWS, YEAR 3: VALUE OF ASSESSMENT “Doing the rubrics – I didn’t know how that would work, I had my doubts two years ago. I actually started using a rubric and I find it easier to grade consistently …. I really like it and use it now for all my labs, it helps my grading. If students would look at them they could realize exactly what they aren’t doing well. I give this back to students with written comments, and the ones who look at it could improve their labs which is helpful…. I found it helpful and I was totally shocked. Someone said you can’t do it with lab reports – I think you can – I showed that you could.”

  42. FACULTY INTERVIEW YEAR 3: QUOTE “The challenge is to encourage faculty members to be experimental in the ways they imagine student work, without abandoning key commitments to core educational goals having to do with having students express themselves clearly in argument to also say that the ways in which you might go about cultivating a sense of the discipline should go beyond, at least experiment with going beyond, writing 5 or 25 page papers. How you encourage faculty members to do that I am not sure. I suppose one way of doing that is giving them inspirational models. Make sure people within their departments and across disciplines talk to each other about whatever works, whatever seems to gain traction in this area.”

  43. FACULTY INTERVIEWS YEAR 3: Changing Classroom “Expectations of what a classroom is supposed to be like are changing. Using the classroom as a place to do presentations, as a place to do small group work, as a place to do debates… use the classroom as a theatre some days, making a place where we do lecture some days… making it a place where when they are getting ready to go to my class they are wondering what are we going to do today? Mixing it up, building some excitement. It can ‘t be like that everyday, but…. “

  44. AAC&U VALUE and Ohio 5 Teagle

More Related