High School Teachers Instructional Use of WASL Data: Exploring the Role of School Culture and Motivation

1. High School Teachers� Instructional Use of WASL Data: Exploring the Role of School Culture and Motivation Jack B. Monpas-Huber, Ph.D. Director of Assessment and Program Evaluation Spokane Public Schools

2. 2 Who I Am

3. 3 Acknowledgments

4. 4 Background of the Project Work experience: Assessment department of large school district Providing data to schools to support DBDM Dealing with school cultures, politics, leadership Research interests: Sociology of education / school organization Motivation Measurement, statistics & research design Validating large-scale accountability systems

5. 5 Organization of this Presentation Framing the Problem Teachers� Use of State Assessment Data Research Methods Results of the Study Discussion

6. 6 Framing the Problem Rise of state accountability programs High stakes attached to student performance Data fed back to schools for �data-based decisionmaking� �Theory of Action� research (Fuhrman, 2004) Two functions: Accountability Instructional/feedback How do the two forces shape teachers� instructional use of data?

7. 7 Research Questions Considering how much data the state provides to educators, how much are high school teachers using state assessment data as a resource to improve instruction? How useful do they find it? Considering the mounting policy pressures to improve performance on the state assessment, what motivates teachers to use state assessment data? What is the influence of policy pressure specifically, and aspects of school context in general?

8. 8 Limits in Focus State Assessment data Instructional decisions Certificated teachers High schools

9. 9 Organization of this Presentation Framing the Problem Teachers� Use of State Assessment Data Research Methods Results of the Study Discussion

10. 10 Teachers� Use of State Assessment DataRelevant Literatures Accountability Systems Data-based Decision-making Teacher Motivation Accountability and High Schools

11. 11 Teachers� Use of State Assessment DataCapacity for Teacher Data Use Technical Skills Technical skills for working with data Databases, software Analysis and interpretation of systematically collected data Capacity building efforts in Washington Hypothesis: Exposure to training in assessment or in WASL item development should be a strong predictor of use of WASL data

12. 12 Teachers� Use of State Assessment DataCapacity for Teacher Data Use Access to Data Advances in computer technology Assessment personnel Teachers may vary in their perception of access to data Hypothesis: Teachers who perceive more access to data should be more likely to use such data than teachers who perceive less access to data Access as necessary but not sufficient condition

13. 13 Teachers� Use of State Assessment DataTeacher Motivation and Data Use The Policy Perspective Some instructional changes are difficult Teachers need consequences Behavioral perspective on motivation Research on high stakes testing Research issue: perceived pressure as both outcome and predictor Hypothesis: Teachers who perceive higher levels of pressure will be more likely to use assessment data than teachers who perceive lower levels of pressure

14. 14 Teachers� Use of State Assessment DataTeacher Motivation and Data Use Alternative Perspectives Cognitive perspectives on motivation Motivation stems from mind/thought/interpretation Social context and cognition

15. 15 Teachers� Use of State Assessment DataTeacher Motivation and Data Use Expectancy �Teacher�s perceived probability that the teacher�s effort will result in the attainment of the goals� (Kelley, Heneman, & Milanowski, 2002, p. 378) �Will do� of motivation; efforts will result in positive outcomes Kentucky and North Carolina research Hypothesis: Teachers who report higher levels of expectancy (that working with assessment data will actually help them improve instruction for their students) will be more likely to use assessment data than those who expect less to result from it

16. 16 Teachers� Use of State Assessment DataTeacher Motivation and Data Use Efficacy �Teacher beliefs about their span of influence and performance capacity� (Kelley & Finnigan, 2003, p. 604) �Can do� of motivation The role of performance feedback in motivation research Hypothesis: Teachers who feel more efficacious working with assessment data will be more likely to use assessment data than teachers who feel less efficacious

17. 17 Teachers� Use of State Assessment DataTeacher Motivation and Data Use Goals Motivation as product of intentions or goals people have for engaging in a behavior People pursue variety of goals Goals may conflict with each other Teachers and �perceived policy intentions� of accountability policies (Leithwood, Steinbach, & Jantzi, 2002) Ingram, Louis, & Schroeder (2004) study Hypothesis: Teachers will be more likely to use state assessment data if they perceive its underlying purpose as consistent with their own goal of helping students learn

18. 18 Teachers� Use of State Assessment DataSummary of Motivation Research Pressure, expectancy, efficacy, goals Filtered through school context Some aspects of context (collaboration, feedback data) influence motivations These motivations vary: Among teachers within one school Possibly by groups of teachers between schools Research issues Motivations as predictors of data use Motivational effects may be different in different schools

19. 19 Teachers� Use of State Assessment DataBuilding a Model of Teacher Data Use Quantity of Teacher Data Use = �0 (mean) + �1(state assessment training) + �2(perceived access to data) + �3(perceived pressure) + �4(expectancy) + �5(efficacy) + �6(goal alignment) + e (unmodeled variation)

20. 20 Teachers� Use of State Assessment DataContextual Influences on Teacher Motivation and Data Use To the extent that motivations are shared by teachers in one school, what influences these motivations? Are some schools more �motivating� than others, in this case, in regard to using and learning from state assessment data? Sociological perspectives on school culture and other contextual influence

21. 21 Teachers� Use of State Assessment DataContextual Influences: Culture Focus on shared attitudes and behavior is focus on culture Two perspectives on culture (Swidler, 1995): �Inside out� � internalized attitudes (motivations?) predict behavior �Outside in� � shared practice, norms, codes regulate behavior irrespective of internal beliefs

22. 22 Teachers� Use of State Assessment DataCultural Perspectives The Loose Coupling Perspective (Weick, 1976; Firestone, 1985) Educational organizations are multi-layered Classrooms disconnected from administration Because teaching and learning is not precise, schools do not evaluate technical quality of instruction High schools especially loosely coupled Challenges bureaucratic models of schools which emphasize centrality of leadership and formal rational procedures Also helps explain why reform movements have historically failed to change instruction in schools Loose coupling and assessment data �Stick them in a drawer�

23. 23 Teachers� Use of State Assessment DataCultural Perspectives Professional Accountability Abelmann & Elmore (1999) Strong and weak internal accountability systems O�Day (2004) Professional Collaboration Student learning data as centerpiece of collaborative work Recurrent predictor in past research

24. 24 Teachers� Use of State Assessment DataLeadership Leaders filter and frame accountability policy (Spillane) Transformational leadership has positive effects on teacher motivation Trust, collaboration, shared accountability Principals may vary in how they frame assessment results School-level variable that influences motivations and assessment data

25. 25 Teachers� Use of State Assessment DataMethodological Observations Lots of qualitative case studies No quantitative studies of use as a criterion or dependent variable Lack basic descriptive data about levels or frequencies of use

26. 26 Teachers� Use of State Assessment DataA Tentative Model

27. 27 Organization of this Presentation Framing the Problem Teachers� Use of State Assessment Data Research Methods Results of the Study Discussion

28. 28 Research MethodsDesign Issues Teacher survey Study population: certificated teachers in high schools in western Washington school districts that employ a full-time assessment director Instrument: 4-page questionnaire Matrix sampling Three forms Each contained common and unique items

29. 29 Research MethodsSchool Sample Characteristics

30. 30 Research MethodsSample Characteristics � Teachers

31. 31 Research MethodsScale Development Classical Test Theory Internal consistency reliability (Cronbach�s coefficient alpha (a)) Item-total correlations Exploratory factor analyses (EFA) Item Response Theory Rating Scale Model (Wright & Masters, 1982) Item difficulty, fit statistics

32. 32 Research MethodsOutcome Measures Frequency of WASL Data Use Utility of WASL Data Use

33. 33 Research MethodsPredictor Measures Perceived Access to Data Training in State Assessment Training in WASL Item Construction Pressure to Increase WASL Performance WASL Goal Alignment Efficacy with WASL Data Principal WASL Commitment Principal Trust Departmental Professional Collaboration Professional Accountability

34. 34 Organization of this Presentation Framing the Problem Teachers� Use of State Assessment Data Research Methods Results of the Study Discussion

35. 35 ResultsHow much are teachers using data?

36. 36 ResultsHow much are teachers using data?

37. 37 ResultsHow much are teachers using data?

38. 38 ResultsHow much are teachers using data?

39. 39 ResultsHow much are teachers using data?

40. 40 ResultsHow much are teachers using data?

41. 41 ResultsHow much are teachers using WASL data?

42. 42 ResultsHow much do teachers benefit from WASL data?

43. 43 ResultsHow much do teachers benefit from WASL data?

44. 44 ResultsHow much do teachers benefit from WASL data?

45. 45 ResultsHow much do teachers benefit from WASL data?

46. 46 ResultsHow much do teachers benefit from WASL data?

47. 47 ResultsHow much do teachers benefit from WASL data?

48. 48 ResultsHow much do teachers benefit from WASL data?

49. 49 ResultsHow much do teachers benefit from WASL data?

50. 50 ResultsWhat motivates teachers to use WASL data?

51. 51 ResultsVisualizing Hierarchical Linear Modeling, 1/3

52. 52 ResultsVisualizing Hierarchical Linear Modeling, 2/3

53. 53 ResultsVisualizing Hierarchical Linear Modeling, 3/3

54. 54 ResultsFrequency of WASL Data Use � HLM Results

55. 55 ResultsFinal Model of Frequency of Data Use � HLM Results Frequency of WASL Data Use = �0 (mean) + �1(utility of WASL data use) + �2(training in WASL item writing) + �3(perceived pressure to increase WASL scores) + �4(principal commitment to WASL improvement) + �5(efficacy with WASL data) + r (unmodeled variation) �0 = ?00 + ?01(school ethnic composition) + u0

56. 56 ResultsModeling Frequency of WASL Data Use

57. 57 ResultsUtility of WASL Data Use � HLM Results

58. 58 ResultsFinal Model of Utility of Data Use � HLM Results Utility of WASL Data Use = �0 (mean) + �1(perceived access to WASL data) + �2(frequency of WASL data use) + �3(departmental professional accountability) + �4(departmenal professional collaboration) + �5(WASL goal alignment) + �6(WASL efficacy) + r (unmodeled variation)

59. 59 Organization of this Presentation Framing the Problem Teachers� Use of State Assessment Data Research Methods Results of the Study Discussion

60. 60 Conclusions High School Teachers Use of Data Teachers are using data with moderate frequency and gaining some value from it This aspect of WASL program is working Motivation and School Context Multiple motivations are at work (pressure and efficacy) Principal leadership provides incentive to use Sensemaking of data is social / collaborative Data as feature of more tightly coupled schools

61. 61 Thank you! To contact me: Jack B. Monpas-Huber, PhD Director of Assessment and Program Evaluation Spokane Public Schools jackm@spokaneschools.org (509) 354-7396 Office (206) 947-9926 Cell