Situated Evaluation

1. Situated Evaluation Chip Bruce March 9, 1999

2. Constructivism “Our perceptions do not come simply from the objects around us, but from out past experience as functioning, purposive organisms “ ... always wrong in any particular instance [book example]” --E. C. Kelley, 1947, p. 34

3. Hermeneutics Not occasionally only, but always, the meaning of a text goes beyond its author. That is why understanding is not merely a reproductive, but always a productive attitude as well. – H. Gadamer, Truth & Method

4. Intertextuality No member of a verbal community can ever find words in the language that are neutral, exempt from the aspirations and evaluations of the other, uninhabited by the other’s voice. On the contrary, he receives the word by the other’s voice and it remains filled with that voice. He intervenes in his own context from another context, already penetrated by the other’s intentions. His own intention finds a word already lived in. – M. Bakhtin, Problems of Dostoevsky’s Poetics

5. My history with IT evaluation 1969 generative CAI 1982-86 Quill => situated evaluation 1987-90 Statistics Workshop Cheche Konnen 1988-90 ENFI 1997 Reader Focused Writing 1997-98 Evaluation of IT course

6. Stereotype view • Quantitative only • Little attention to antecedent conditions or classroom transactions • Assumes fixed, knowable entities • Unable to address unanticipated effects • No model for diversity of realizations

7. Quill evaluation Differentially significant growth in writing ability across expository, persuasive, narrative modes, but... Not relevant to most stakeholders Ignored changes in classroom practices, reconceptions of curricula, etc. Obscured process of change

8. Electronic Networks for Interaction • New social dimensions in the classroom • Immersion in a writing community • Collaboration in writing • Writing for authentic purposes • Writing across the curriculum • Writing process made visible

9. Data sources • Classroom observations; videotapes • Interviews with students, teachers,... • Student writing • Survey data • Email: Students. teachers, experts • Curriculum guides • Teachers’ writing about their classrooms • Participant feedback

10. Changes mediated by the contexts of use • Students’ (parents’) beliefs and values • Teacher’s pedagogical approach • Teacher’s view of the educational potential • Classroom management issues • Support • Institutional realities

11. Text sharing Drama Socratic tutoring Scenarios Small group discussions Brainstorming Collaborative writing Devil’s Advocate Distance networking Twenty questions Cross-age tutoring Discussion of reading Discussion of issues Open discussion Realizations of ENFI

12. Alternate Realizations

13. Evaluation questions Summative: How well does it work? Formative: How can it be improved? Situated: What practices emerge as the innovation(s) are incorporated into different settings? => How well do they work? How can they be improved?

14. Aspects of situated evaluation • Analyze the innovation • Analyze existing practices • Observe changes over time as the innovation is incorporated into practice • Examine the functional relevance of differences in realizations • Reanalyze innovation => emergent properties

15. Abstraction vs. Generalization It is a mistake to equate “abstract” with “general”. Only the concrete permits a general understanding of systemic interconnectedness. –Y. Engestrøm, “Learning by Expanding”

16. Pasteur’s death-bed words “Bernard is right; the pathogen is nothing; the terrain is everything. – Oliver Sacks, Awakenings

17. Implications: Re-creation as central to change • Need to understand diverse realizations • Innovation begins with the teacher • Technology as tool for the re-creation

18. Technology as Social Practice • How do we relate classroom goals to IT characteristics? How does that analysis inform further development? • How do participants interpret IT resources? How do they construct their activities? • What elements persist across realizations? • What specific materials, activities, and collaborative environments facilitate the most successful projects?

19. Orientation • constructivism • particulars, e.g., Donald Graves • reader response theory

20. Types of evaluation Formative SummativeSituated Innovation Effects Practices Developer User Both Improve Judge Describe Minimize var. Control Use User feedback Experiment Ethnographic Development End Either List of change Comparisons Ethnography

21. Evaluation “the systematic collection and interpretation of evidence, leading, as part of the process, to a judgment of value with a view to action” --Beeby, 1977

22. Stakeholder positions • software developer • curriculum developer • teacher • teacher educator • parent • researcher

23. Countenance of educational evaluation “too little effort to spell out antecedent conditions and classroom transactions ... [and] to couple them with the various outcomes” --R. E. Stake, 1967, p. 524 [reprinted in Classics of Instructional Technology, Ely & Plomp, 1996]

24. Responsive evaluation “if it orients more directly to program activities than to program intents; responds to audience requirements for information; and if the different value perspectives present are referred to in reporting the success and failure of the program” --R. E. Stake, 1975, p. 14