Observing patterns of classroom connectivity: Protocol development and implementation

1. Observing patterns of classroom connectivity: Protocol development and implementation Stephen J. Pape, University of Florida,Sukru Kaya, Karen E. Irving, Vehbi Sanalan, Douglas T. Owens, The Ohio State University The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305K050045 to The Ohio State University.  The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.

2. Additional Authors L. Abrahamson, Better Education Foundation S. Granade,Wilkes Community College, & Sedat Ucar,Cukurova University, Adana, Turkey Additional Research Team Frank Demana, The Ohio State University; Christie Boscardin, Joan Herman, Hye Sook Shin UCLA, CRESST; Mike Kositzke, Project Administrative Assistant, OSU Ugur Baslanti, University of Florida TI Navigator slide adapted from a presentation by Eileen Shihadeh, Texas Instruments

3. Changing roles for teachers include • Thinking beyond skills-based conceptions • Setting norms for discourse • Fostering mathematical and scientific reasoning • Using problem solving and inquiry to support knowledge construction • Developing mathematical and scientific competence more broadly defined • Using formative as well as summative assessment • Differentiating instruction

4. Context -- Classroom Connectivity in Promoting Mathematics and Science Achievement • Interdisciplinary research project • National Sample of Algebra I and Physical Science teachers • Classroom connectivity technology • Professional Development—Summer Institute & T3 conference follow-up • Randomized assignment, cross-over design with control group provided treatment during year 2 • Mixed methodology

5. 10-Mar-14 The TI-Navigator™ Connected Classroom The TI-Navigator System allows the teacher to: • Create a collaborative learning environment • Engage in formative assessment by way of immediate feedback • Enhance classroom management of TI graphing technology • Quick Poll provides teacher understanding by receiving impromptu feedback 5

6. Prior Research on Connected Classrooms (Roschelle, Penuel, & Abrahamson, 2004) • Students: • Increased student engagement; student understanding; interactivity • Improved classroom discourse • Knowledge of classmates’ learning • Teachers: • Improved pre- and post- assessment of student learning • Increased awareness of student difficulties • Improved questioning

7. Owens , Demana , Abrahamson, Meagher, & Herman (2004)

8. The Potential of the Connected ClassroomIncludes • Multiple interconnected representations • Conceptual development supported through activity-based learning experiences • Immediate, anonymous formative assessment • Public displays of class knowledge • Teacher identified critical junctures

9. The Potential of the Connected ClassroomIncludes • Classroom discourse • Explanations and justifications • Focus on process • Strategic behavior as object of discourse • Changing classroom atmosphere making possible • Increased motivation/engagement • Positive dispositions toward mathematics and science

10. Theoretical Perspectives for Observation Protocol Include • Goal orientation of motivation & social cognitive view of learning (e.g., Ames, 1992; Ames & Archer, 1988; Blumenfeld, 1992; Meece, 1991) • Classroom context that promote involvement (Turner et al., 1998, 2002) • Classroom discourse practices (Cobb, Boufi, McClain, & Whitenack, 1997; Meyer & Turner, 2002; Nathan & Knuth, 2003; Turner, et al., 1998; Wood, 1999) • Contexts that support mathematical understanding (Fennema, Sowder, & Carpenter, 1999)

11. Theoretical Perspectives for Observation ProtocolInclude • Contexts that support SRL (DeCorte, Verschaffel, & Eynde, 2000; Pape, 2005; Perry & VandeKampe, 2000) • Formative Assessment (Bell & Cowie, 2001; Black & Wiliam, 1998) • Analytic & Social Scaffolding (Nathan, 2003) • Environmental scaffolding of strategic behavior (Pape, Bell, & Yetkin, 2003)

12. Purpose of Presentation • To describe the process of developing and piloting the CCMS: Classroom Observation Protocol-Algebra I • To discuss future implementation strategies

13. Observation Protocols Examined • Professional Development Observation Protocol (Horizon Research, 2004) • OPAL: Observing Patterns of Adaptive learning (Patrick, et al., 1997) • Oregon Mathematics Leadership Institute Classroom Observation Protocol (RMC Research Corporation, 2005) • Reformed Teaching Observation Protocol (RTOP) (Arizona Collaborative for Excellence in the Preparation of Teachers, 2000)

14. OPAL – Method of Protocol Development • Goal: “to investigate associations between teacher behaviors and practices and their students’ perceptions of the motivational climate of those classrooms” (p. 1) • Format – narrative running record • 9 categories: Task, Authority, Recognition, Grouping, Evaluation, Time, Social, Help-Seeking, & Messages

15. OPAL – Training • Project team familiar with literature • Observation protocol based on literature • Phase 1 • View videotapes & record running records • Team discussion of observations • Consensus building on important points of observation • Third person review & comment • Phase 2 • Pair observations of summer school & write running records • Compare field notes • Third person review & comment

16. CCMS Protocol Development • Extensive review of literature including: • Classroom discourse processes • Classroom contexts that promote self-regulated learning • Classroom contexts that increase student motivation • Formative assessment practices • Classroom observation protocols • Focus—list of instructional practices that support • Mathematical understanding • Self-regulation • Positive dispositions • Generate list of instructional strategies with descriptions

17. Three-day Training for Research Team—Day One • Reviewed constructs to establish understanding • Viewed videotape & scripted lesson • Pairs compared lesson scripts identifying similarities and differences • Initial ratings & discussion of ratings • Generated positive and negative evidence for constructs • Revised protocol based on discussion

18. Three-day Training for Research Team—Day Two & Three • Day Two – Classroom visit • Day Three – Viewed videotaped class • Groups observed, scripted, & rated lesson • Independently indicated ratings • Groups discuss ratings, discrepancies, and evidence for ratings • Revise observation protocol

19. Training (Phase 2) & Inter-rater Reliability • Initial focus – calibration and construct clarification • 3-4 observers visit class, script, & rate observation • Record individual ratings without discussion • Discussion toward consensus with rationale for ratings • Final revisions – deleted/clarified unreliable categories

20. Research Team Considerations and Data Collection Decisions • Given focus on communication – camera focused on teacher in relation to students • Evidence for ratings from observed behavior vs. inferences from self-report data • Three column format: time, behavior, analytical notes • Observers – consistency of ratings; reliability; number of observers

21. Inter-rater Reliability • Inconsistent constructs, in terms of large variability around the consensus score, included: • Multiple Answers • S-to-T & S-to-S discourse • Initiate-Respond-Evaluate (IRE) • Multiple Representations • Processes • Autonomy • Assessment, Knowledge, & Community Centeredness • In addition, one rater consistently higher than others; one rater consistently lower than others

22. Changes to Protocol during Development • Feedback during training incorporated to elaborate and clarify constructs • Low reliability constructs revised – clarification, elaboration, and modification • Constructs Deleted • Difficulty of providing evidence – e.g., modifying tasks to control challenge, strategy application, teacher knowledge • Relevance to research – e.g., conversational turns, wait time

23. CCMS Observation Protocol – Classroom Context Form • Background Information: Date of observation; Location (City, State); Observer; Observation duration • Classroom artifacts: Classroom rules and slogans; Classroom displays (including graded work); Instructional materials • Classroom layout: Provide a labeled map of the classroom. Physical arrangement of context “in terms of specific tasks.” • Scripting the Observation: Specific, objective descriptions; three column format: (1) time; (2) running log stated objectively; (3) Reflections, theoretical connections, observer notes.

24. CCMS Observation Protocol – Observation Rating Form • Classroom context summary • Tasks: manner in which teachers structure tasks and learning activities • Structural: Content, procedures (explicit & implicit), materials, participation structure, and products of task • Psychological: Student &/or teacher affect assoc with the task, level of challenge and press, instructional support

25. CCMS Classroom Observation Protocol (con’t) • Teacher Designed Questions • Lower-Order Questions – require recalling and stating known facts, carrying out a simple algorithm or procedure • Procedural, knowledge, comprehension questions • Higher-Order Questions – require manipulation of information, transform information, synthesize, generalize, explain, hypothesize • Justification, classification, comparison, exploration/extension, application, analysis, synthesis, evaluation questions

26. CCMS Classroom Observation Protocol (con’t) • Sociomathematical norms – ways in which teachers and students interact within classroom • Teacher press for student involvement – ways teacher ensures & increases involvement • Teacher press for elaboration, explanation, & justification – ways teacher pushes students to elaborate to make reasoning explicit; includes uptake of correct & incorrect responses • Soliciting multiple answers or solution methods • Scaffolding – teacher support such as modeling, questioning; analytic & social scaffolding • Learning strategies – explicit discussion of techniques for learning • Performance orientation – focus on the outcome/products & showing competence • Mastery orientation – focus on understanding, learning skills with meaning

27. CCMS Classroom Observation Protocol (con’t) • Classroom Discourse -- Relative proportion or focus on • Teacher-to-student – teacher-initiated interaction • Student-to-teacher – student-initiated interaction • Student-to-student interaction • Initiation-Response-Evaluation (IRE) • Classroom management procedures • Use of multiple representations • Collective argumentation – two or more students contribute to establish a claim • Student discussion of understanding

28. CCMS Classroom Observation Protocol (con’t) • Formative Assessment – teacher behaviors targeted toward gathering data from students and making instructional decision accordingly • Educational goals associated with instructional decisions • Rules and facts • Communication • Application • Evaluation • Mathematical processes

29. CCMS Classroom Observation Protocol (con’t) • Knowledge Construction • Individually vs. group constructed • Holder of knowledge – degree to which teacher or students are authority of knowledge construction • Depth of knowledge – superficial vs. central • Technology Use – overall use; quick poll; screen capture; learn check; student inquiry/data aggregation; activity center; TI presenter; graphing calculator; TI viewscreen; Overhead projector; Computer; Other

30. CCMS Classroom Observation Protocol (con’t) • Instructional Practices • Traditional/Teacher-centered • NCTM Standards-based, cognitively active environment, student-centered • Inquiry, problem-based learning, discovery learning • How People Learn • Learner-centeredness • Assessment-centeredness • Knowledge-centeredness • Community-centeredness

31. Update/Implementation/Future Work • To date, 2 years of observations have been conducted (approx. 15 per Algebra I cohort & 5 per Physical Science cohort) • Observations will be used to triangulate self-report data • Teacher Practices and Beliefs Survey • Student Perceptions of Instruction Survey • Implementation rating from Telephone Interviews • Planned Analyses • Differences between Algebra I and Physical Science teaching • Changes to protocol to analyze Physical Science instruction • Purposefully chosen cases examined and elaborated • Examine classroom instruction in relation to development of student dispositions and SRL • Conduct in-depth ethnographies of instruction with classroom connectivity technology during years 3 & 4