Evaluation: Telling the Story of UIUC’s GK-12

1. Evaluation:Telling the Story of UIUC’s GK-12 Delwyn Harnisch, University of Nebraska, Lincoln Chip Bruce, University of Illinois, U-C Sharon Comstock, University of Illinois, U-C Jim Buell, University of Illinois, U-C

2. Why evaluate? To document for the National Science Foundation: • Collaboration between the academic and K-12 education communities • Enriched learning opportunities for K-12 students • Meaningful technology integration that has demonstrable impact

3. What is “evaluation?”* • Idea of “quality” • Differing subjective views of quality not invalidating • Unique meanings in unique settings; “situated” *from What is Evaluation? by Chip Bruce, et al. Inquiry Unit: http://www.inquiry.uiuc.edu/bin/update_unit.cgi?command=select&xmlfile=u10377.xml In particular, see the uploaded video of Prof. Bob Stake, Director of The Center for Instructional Research and Curriculum Evaluation at UIUC, and a critical scholar in qualitative case and evaluation studies. See also, Stake, R. E. (1995). The Art of Case Study Research. Thousand Oaks: Sage Publications.

4. What is “evaluation?” • The term’s meaning to educators: • “Evaluation” occurs constantly in the classroom. There is an intuitive sense of what is “working” moment to moment • “Evaluation” as a term conjures formal assessment with defined criteria (grading, testing, job performance, etc.)

5. What is “evaluation?” • The Evaluation Team’s approach: • Informal • Examines the individual and the complexity of a situation (situational, contextual) • Formal • Formalized and invites comparison across groups

6. Evaluation:Understanding Context • Learning logs (“participant observer”) • Observation • Unstructured interviews • Formal online surveys: • http://gk12uiuc.net • Intake (Fellows) • Mid-year (Fellows) • End-of-year (Fellows, teachers, mentors)

7. What is “evaluation?” • Evaluation as a new way of thinking about information technologies • As process, not product (balancing “result”) • Less about the tools, and more about the cultures in which these technologies are used

8. What is “evaluation?” • “Responsive Evaluation” of complexity • Progressive focusing, emergent themes • Triangulation for validation • Formative—informing project

9. What is “evaluation?” Evaluation is a movement from naïve, early perceptions toward subtle, sophisticated understanding.

10. Selected resources Bruce, B. C. (1999). Challenges for the evaluation of new information and communication technologies. JAAL, 42(6):450--455. (see online: http://www.readingonline.org/electronic/jaal/March_Column.html ) Educational Technology and Society-- special issue on Evaluation of Learning Technology. Frechtling, Joy, & Sharp, Laure (1997, August). User-friendly handbook for mixed method evaluations. Washington, DC: National Science Foundation. Quiñones, Sherri, & and Kirshstein, Rita (1998, December). An educator's guide to evaluating the use of technology in schools and classrooms. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.