Jack B. Monpas-Huber, Ph.D. Director of Assessment and Program Evaluation Spokane Public Schools

1. High School Teachers’ Instructional Use of WASL Data: Exploring the Role of School Culture and Motivation Jack B. Monpas-Huber, Ph.D. Director of Assessment and Program Evaluation Spokane Public Schools

2. Who I Am Master of Science, Sociology, 1997 Ph.D., Educational Psychology Director of Assessment and Program Evaluation

3. Acknowledgments

4. Background of the Project Work experience: • Assessment department of large school district • Providing data to schools to support DBDM • Dealing with school cultures, politics, leadership Research interests: • Sociology of education / school organization • Motivation • Measurement, statistics & research design • Validating large-scale accountability systems

5. Organization of this Presentation • Framing the Problem • Teachers’ Use of State Assessment Data • Research Methods • Results of the Study • Discussion

6. Framing the Problem • Rise of state accountability programs • High stakes attached to student performance • Data fed back to schools for “data-based decisionmaking” • “Theory of Action” research (Fuhrman, 2004) • Two functions: • Accountability • Instructional/feedback • How do the two forces shape teachers’ instructional use of data?

7. Research Questions • Considering how much data the state provides to educators, how much are high school teachers using state assessment data as a resource to improve instruction? How useful do they find it? • Considering the mounting policy pressures to improve performance on the state assessment, what motivates teachers to use state assessment data? What is the influence of policy pressure specifically, and aspects of school context in general?

8. Limits in Focus • State Assessment data • Instructional decisions • Certificated teachers • High schools

9. Organization of this Presentation • Framing the Problem • Teachers’ Use of State Assessment Data • Research Methods • Results of the Study • Discussion

10. Teachers’ Use of State Assessment DataRelevant Literatures • Accountability Systems • Data-based Decision-making • Teacher Motivation • Accountability and High Schools

11. Teachers’ Use of State Assessment DataCapacity for Teacher Data Use Technical Skills • Technical skills for working with data • Databases, software • Analysis and interpretation of systematically collected data • Capacity building efforts in Washington • Hypothesis: • Exposure to training in assessment or in WASL item development should be a strong predictor of use of WASL data

12. Teachers’ Use of State Assessment DataCapacity for Teacher Data Use Access to Data • Advances in computer technology • Assessment personnel • Teachers may vary in their perception of access to data • Hypothesis: • Teachers who perceive more access to data should be more likely to use such data than teachers who perceive less access to data • Access as necessary but not sufficient condition

13. Teachers’ Use of State Assessment DataTeacher Motivation and Data Use The Policy Perspective • Some instructional changes are difficult • Teachers need consequences • Behavioral perspective on motivation • Research on high stakes testing • Research issue: perceived pressure as both outcome and predictor • Hypothesis: • Teachers who perceive higher levels of pressure will be more likely to use assessment data than teachers who perceive lower levels of pressure

14. Teachers’ Use of State Assessment DataTeacherMotivation and Data Use Alternative Perspectives • Cognitive perspectives on motivation • Motivation stems from mind/thought/interpretation • Social context and cognition

15. Teachers’ Use of State Assessment DataTeacherMotivation and Data Use Expectancy • “Teacher’s perceived probability that the teacher’s effort will result in the attainment of the goals” (Kelley, Heneman, & Milanowski, 2002, p. 378) • “Will do” of motivation; efforts will result in positive outcomes • Kentucky and North Carolina research • Hypothesis: • Teachers who report higher levels of expectancy (that working with assessment data will actually help them improve instruction for their students) will be more likely to use assessment data than those who expect less to result from it

16. Teachers’ Use of State Assessment DataTeacherMotivation and Data Use Efficacy • “Teacher beliefs about their span of influence and performance capacity” (Kelley & Finnigan, 2003, p. 604) • “Can do” of motivation • The role of performance feedback in motivation research • Hypothesis: • Teachers who feel more efficacious working with assessment data will be more likely to use assessment data than teachers who feel less efficacious

17. Teachers’ Use of State Assessment DataTeacher Motivation and Data Use Goals • Motivation as product of intentions or goals people have for engaging in a behavior • People pursue variety of goals • Goals may conflict with each other • Teachers and “perceived policy intentions” of accountability policies (Leithwood, Steinbach, & Jantzi, 2002) • Ingram, Louis, & Schroeder (2004) study • Hypothesis: • Teachers will be more likely to use state assessment data if they perceive its underlying purpose as consistent with their own goal of helping students learn

18. Teachers’ Use of State Assessment DataSummary of Motivation Research • Pressure, expectancy, efficacy, goals • Filtered through school context • Some aspects of context (collaboration, feedback data) influence motivations • These motivations vary: • Among teachers within one school • Possibly by groups of teachers between schools • Research issues • Motivations as predictors of data use • Motivational effects may be different in different schools

19. Teachers’ Use of State Assessment DataBuilding a Model of Teacher Data Use Quantity of Teacher Data Use = β0 (mean) + β1(state assessment training) + β2(perceived access to data) + β3(perceived pressure) + β4(expectancy) + β5(efficacy) + β6(goal alignment) + e (unmodeled variation)

20. Teachers’ Use of State Assessment DataContextual Influences on Teacher Motivation and Data Use • To the extent that motivations are shared by teachers in one school, what influences these motivations? • Are some schools more “motivating” than others, in this case, in regard to using and learning from state assessment data? • Sociological perspectives on school culture and other contextual influence

21. Teachers’ Use of State Assessment DataContextual Influences: Culture • Focus on shared attitudes and behavior is focus on culture • Two perspectives on culture (Swidler, 1995): • “Inside out” – internalized attitudes (motivations?) predict behavior • “Outside in” – shared practice, norms, codes regulate behavior irrespective of internal beliefs

22. Teachers’ Use of State Assessment DataCultural Perspectives The Loose Coupling Perspective (Weick, 1976; Firestone, 1985) • Educational organizations are multi-layered • Classrooms disconnected from administration • Because teaching and learning is not precise, schools do not evaluate technical quality of instruction • High schools especially loosely coupled • Challenges bureaucratic models of schools which emphasize centrality of leadership and formal rational procedures • Also helps explain why reform movements have historically failed to change instruction in schools • Loose coupling and assessment data • “Stick them in a drawer”

23. Teachers’ Use of State Assessment DataCultural Perspectives • Professional Accountability • Abelmann & Elmore (1999) • Strong and weak internal accountability systems • O’Day (2004) • Professional Collaboration • Student learning data as centerpiece of collaborative work • Recurrent predictor in past research

24. Teachers’ Use of State Assessment DataLeadership • Leaders filter and frame accountability policy (Spillane) • Transformational leadership has positive effects on teacher motivation • Trust, collaboration, shared accountability • Principals may vary in how they frame assessment results • School-level variable that influences motivations and assessment data

25. Teachers’ Use of State Assessment DataMethodological Observations • Lots of qualitative case studies • No quantitative studies of use as a criterion or dependent variable • Lack basic descriptive data about levels or frequencies of use

26. Teachers’ Use of State Assessment DataA Tentative Model

27. Organization of this Presentation • Framing the Problem • Teachers’ Use of State Assessment Data • Research Methods • Results of the Study • Discussion

28. Research MethodsDesign Issues • Teacher survey • Study population: certificated teachers in high schools in western Washington school districts that employ a full-time assessment director • Instrument: 4-page questionnaire • Matrix sampling • Three forms • Each contained common and unique items

29. Research MethodsSchool Sample Characteristics Final sample size for analysis: 376 “WASL teachers” (teach 10th grade AND math, English, science, special education, or ELL)

30. Research MethodsSample Characteristics – Teachers

31. Research MethodsScale Development • Classical Test Theory • Internal consistency reliability (Cronbach’s coefficient alpha (α)) • Item-total correlations • Exploratory factor analyses (EFA) • Item Response Theory • Rating Scale Model (Wright & Masters, 1982) • Item difficulty, fit statistics

32. Research MethodsOutcome Measures • Frequency of WASL Data Use • Utility of WASL Data Use

33. Research MethodsPredictor Measures • Perceived Access to Data • Training in State Assessment • Training in WASL Item Construction • Pressure to Increase WASL Performance • WASL Goal Alignment • Efficacy with WASL Data • Principal WASL Commitment • Principal Trust • Departmental Professional Collaboration • Professional Accountability

34. Organization of this Presentation • Framing the Problem • Teachers’ Use of State Assessment Data • Research Methods • Results of the Study • Discussion

35. ResultsHow much are teachers using data?

36. ResultsHow much are teachers using data?

37. ResultsHow much are teachers using data?

38. ResultsHow much are teachers using data?

39. ResultsHow much are teachers using data?

40. ResultsHow much are teachers using data?

41. ResultsHow much are teachers using WASL data? Range: 0-100 Mean: 37.5 SD = 13.9 What does this scale mean in practical terms?

42. ResultsHow much do teachers benefit from WASL data?

43. ResultsHow much do teachers benefit from WASL data?

44. ResultsHow much do teachers benefit from WASL data?

45. ResultsHow much do teachers benefit from WASL data?

46. ResultsHow much do teachers benefit from WASL data? WASL data are of use to me in my instruction

47. ResultsHow much do teachers benefit from WASL data?

48. ResultsHow much do teachers benefit from WASL data?

49. ResultsHow much do teachers benefit from WASL data? Range: 0-100 Mean: 46.4 SD = 15.1

50. ResultsWhat motivates teachers to use WASL data?