What Do the Student Survey Questions Tell Us on the CATS Assessment?

1. What Do the Student Survey Questions Tell Us on the CATS Assessment? By Natalie Bolton Stiglitz,PhD Office of Civic Education and Engagement University of Louisville September 23, 2008

2. School Characteristics (independent variables) Student Demographics (independent variables) Gender Teacher credentials Ethnicity KCCT S.S. Student Questionnaire Student Achievement in social studies on Grade 8 2006 KCCT (dependent variable) SES Duration S.S. Courses Disability Excellence and Equity in Social Studies Education

3. School Characteristics2003-2006 Kentucky School Report Cards Teacher Credentials school percentage of classes taught by teachers certified by subject and grade level school percentage of classes taught by teachers with a major, minor, or the equivalent in the subject being taught school percentage of teachers with a Masters degree or greater or the equivalent school average years of teaching experience

4. School Characteristics2006 Kentucky Performance Report S.S. Questionnaire Data and2003-2006 Kentucky LEAD Report Aggregated student S.S. KCCT questionnaire Eight items address social studies pedagogy (e.g., emphasis on usage of discussion, textbooks, worksheets, small groups, focus on current events, essential questions, scoring guides, open response questions) One item addresses the degree to which students feel the assessment is aligned with material covered in class Two items address students’ motivation for and evaluation of their performance School social studies average hours instruction 2003-2006, LEAD Report

5. Student Demographics2006 Kentucky Performance Report S.S. Disaggregated Data Gender Male= 0 and Female = 1 Ethnicity White= 0, African-American = 1, Hispanic = 2, Asian = 3, and Other = 4 Socio-economic status (SES) Free/reduced lunch approved= 0 and Free/reduced lunch not approved = 1 Disability No disability= 0 and Disability = 1

6. What is the greatest predictor on grade 8 s.s. KCCT?

7. What is the greatest predictor on grade 8 s.s. KCCT? Hierarchical Regression Results (N = 32,833): Model 1: Gender and ethnicity accounted for 5% of the variance in the total KCCT grade 8 social studies score Model 2: Disability status explained 10% of the variance in total social studies score over and above gender and ethnicity Model 3: SES status explained 5% of the variance in the total KCCT grade 8 social studies score over and above gender, ethnicity, and disability or Total model explained 19% of the variance in total KCCT grade 8 social studies score Student Achievement in social studies

8. Opportunity-to-Learn (OTL) “Whether or not…students have the opportunity to study a particular topic or learn how to solve an articulate type of problem presented by the test” (Husen, 1967) Purpose- maximize fairness and equity for students (Porter, 1993) Stevens and Grymes (1993)- OTL measures content coverage- how much of the curriculum is studied content exposure- the depth to which the curriculum is studied content emphasis- particular components emphasized in the curriculum quality of instructional delivery- the way the curriculum is taught teacher professional characteristics- as the development of content knowledge and knowledge and use of diverse types of assessment school support systems- affect student learning opportunities such as the amount of time allotted for teacher collaboration and time for teachers to become part of an activeprofessional community

9. What is the greatest predictor on grade 8 s.s. KCCT?

10. What is the greatest predictor on grade 8 s.s. KCCT? Hierarchical Regression Results (N = 251): • Model 1: Mean school demographics accounted for 21% of the variance in the total KCCT grade 8 school social studies score • Model 2: Teacher credentials explained 4% of the variance in total social studies school score over and above mean school demographics • Model 3: Mean school student KCCT s.s. questionnaireitems explained 20% of the variance in the total KCCT grade 8 school social studies score over and above mean school demographics and teacher credentials or Total model explained 44% of the variance in total KCCT grade 8 school social studies score

11. In addition to demographics and teacher credentials, which s.s. student questionnaire item is the greatest predictor of student achievement on grade 8 s.s. KCCT?

12. In addition to demographics and teacher credentials, which s.s. student questionnaire item is the greatest predictor of student achievement on grade 8 s.s. KCCT? Hierarchical Regression Results (N = 251): • Most influential individual predictors on the 2006 grade 8 KCCT included: + Student reports of using essential questions + Student reports of alignment between coursework and test items • Disability status • Being African American • Using textbooks • Discussing current events

13. If you know a school’s student demographics, to what degree can you predict student achievement on grade 8 s.s. KCCT between schools?

14. If you know a school’s student demographics, to what degree can you predict student achievement on grade 8 s.s. KCCT between schools? Hierarchical Linear Model: Random effects one-way analysis of variance model results: • 13% of the variance in s.s scores was between schools

15. If you know a school’s student demographics, to what degree can you predict student achievement on grade 8 s.s. KCCT within a school?

16. If you know a school’s student demographics, to what degree can you predict student achievement on grade 8 s.s. KCCT within a school? Hierarchical Linear Model: • Random coefficient model results: • 23% of the within school variance in s.s. scores were predicted from the four Level 1 predictors (gender, ethnicity, SES, and disability)

17. If you control student and school demographics, which two grade 8 s.s. questionnaire items predict almost 33% of student achievement on grade 8 s.s. KCCT?

18. If you control student and school demographics, which two grade 8 s.s. questionnaire items predict almost 33% of student achievement on grade 8 s.s. KCCT? HLM findings indicated that two OTL conditions (student reports of alignment between s.s. content taughtand s.s. test items and students’ perceptions of how well they did on the assessment) even after controlling for student and school demographics, significantly predict almost one-third of achievement in s.s. on the grade 8 KCCT.