A View of Evaluation: Perspectives of Faculty Assessors of Prior Learning Assessment

1. A View of Evaluation: Perspectives of Faculty Assessors of Prior Learning Assessment Presented at the Adult Higher Education Alliance 2010 Annual Conference Saratoga Springs, NY Cindy Stevens Amy D. Rose Northern Illinois University

2. Assessment Today • Assessment Discussions • Accountability • The Meaning of College Level Learning

3. Emergence of 2 forms of PLA within higher education • Testing • Portfolio Development

4. Portfolios • The Credit Exchange Model also called Course Match Model • The Developmental Model

5. Good Practice • CAEL guidelines • Literature on the mechanics of PLA • Institutional procedures • Actual discipline specific understanding of knowledge

6. Assessors • Faculty members or adjuncts • Specialists in subject matter or have experience in assessment • “Accuracy and fairness of evaluation including, validity, reliability, and predictiveness is largely dependent upon training” of assessors and on their “ability to remain objective” (Arnold, 1998, p.65).

7. Research Problem • Prior Learning Assessment (PLA) is a growing area • Involves assessment of learning and gauging of whether this learning is college-level • No clear idea of how different faculty members think about this process

8. Research Questions How do faculty assessors describe their PLA process? In what ways do faculty assessors conceptualize college-level learning?

9. Methodology Basic Qualitative Research Sample – interviewed 20 PLA assessors who also were teaching in higher education Semi-structured interviews

10. Participants • Average of 20.5 years teaching at the college level • 5 hold a masters degree, 1 ABD, and 14 hold a doctorate degree • 6 are adjunct, 4 non-tenured, 1 tenure tracked, 8 tenured, and 1 emeritus • PLA assessment in the areas of social science, communications, religious studies, science, music, dance, technology, business, art, and real estate.

11. Tentative Findings • There are many ways in which assessors learn PLA process and assessment. • Formal • Informal • Nonformal • Ambivalent feelings of competence • Participants used the following approaches to assessment: • External Outcome/Competency Based • Explicit Theory–Based • Implicit Guidelines

12. Formal “We have always had training and we are responsible for much of the training given at other institutions. Without this training, there would be even more confusion about what to do and how to do it.” “Meetings is an understatement , we have training, we do training, we sometimes do more of that than we d anything else.” A few participants did mention attending some kind of CAEL training or workshop. One mentioned ACE books.

13. Informal “I spent a lot of time shadowing my colleagues in the beginning.” “There were like 4 of us maybe so, we would look at them in the same room. We would ask questions of each other and it helped.” “We found each other as much as possible to reassure each other that we were doing this right. We helped each other.” “We would get together in a group, we’d all read the same essay and then we’d talk about it and we’d see how much we were totally on the same page.”

14. Nonformal Many assessors sought out informal training to help them. “My PLA training is basically I trained myself” “I wish we had training.” “Adjuncts never get training.” “No training per se.” “Minimal if any and nobody went.”

15. Feelings of Competence • “Who am I to evaluate anything?” • “I don’t know if I am ever right ever on any of my assessments – all of them.” • “I fumbled my way through and did so many of these. Maybe I did it right. I am not sure.” • “…you know there’s a little bit of playing God in this, in how flexible you are in granting students credit.”

16. Their Learning Process

17. Approaches to Assessment • External Outcome /Competency Based • Explicit Theory Based • Implicit

18. External Outcome/Competency Based “I look at it as if somebody took a course on the topic, you know this is their final exam, like this is an essay that they’re taking on this topic so they have demonstrated they have learned and because it’s experiential learning it’s not all that concrete like you have to address theory.” “Our internal guidelines are basically anything that follows our syllabus; like whatever they were supposed to be learning in the classroom environment, they should be showing me in the portfolio.” “instead of thinking of a 12 or 15 week syllabus with X number of topics on it and seeing whether the student meets those requirements, if you can think about things you wish a student would carry away from class and whether they wrote about it, but it is more does the person know what you would hope someone would know about this a year or so after taking the course, the kernel of knowledge that is critical that remains afterwards.” “I’m much more focused on learning outcomes like I think of it as taking a course.”

19. Explicit Theory Based “There is no guess work here. We look for what should be known. Much of what is taught in the classroom is theory and that is what I expect to see within the papers. My assessment theory is based on them knowing the theory and the people involved. If there are missing the point then they need to go back and get that info.” “I am looking for the degree of knowledge within the paper. The excellence in writing is not necessarily linked to that knowledge so you have to look. The information must be there or I will ask for more information.” “it’s all based on Kolb’s experiential learning model and I look for evidence of the learning and the circular form of Kolb’s model. They have to show me the learning in the narration of the essay, but the experience is the real thing that they have to have and learn from and then apply. If something is missing it is not what I give credit for.” In addition to Kolb’s Experiential Learning Cycle, Bloom’s Taxonomy was named by some participants.

20. Implicit/Tacit Knowledge • Many faculty have a difficult time expressing their assessment activities and with producing a working definition of college-level learning • Tacit knowledge is gained through experience but faculty have limited recall of this knowledge

21. Implicit “ I don’t know that I can explain it because it is sort of internally I can tell with the portfolio essay, it’s just this is what I do, I can tell.” “I am comfortable with it. I’ve been doing it forever. I know what I am doing and I have outlasted many faculty and directors. I can read between the lines and can tell when learning is there and when it is not. I consider myself knowledgeable about the field itself and how PLA fits into college plans.” “I can look at an essay and I can tell right away that they have what they need in the essay. I look for key words that trigger indications that learning has taken place. I don’t need everything there, I need my key words and then I know.”

22. Implicit continued • “When everything is said and done it only needs to be a C level paper to get credit.” • “They just need passing level.” • “You just need a C to get credit and no more and NO less either. I will pass you with average work because you would pass my class with average work.”

23. Conclusions • Difficult to define college level learning • The idea of portfolio development is predicated on the idea of individual institutional standards • Yet this boils down to individual standards • This research indicates that individuals are working from own internal frameworks • While some have external institutional frameworks, only a few have explicit philosophical frameworks.

24. Training occurs in several formats, but mostly informal and non-formal. • Individuals come out of this experience often feeling inadequate to the task. • This is not so different from faculty in traditional courses. • PLA has different meanings to different individuals, but the task of assessment may not be that different from others.

25. More to follow…..