210 likes | 370 Views
Key Education Indicators (KEI): Making Sense of NAEP Contextual Variables. Alan Ginsburg and Marshall S. Smith December 6, 2013. General goals. Improve utility and coherence of the NAEP background variables.
E N D
Key Education Indicators (KEI): Making Sense of NAEP Contextual Variables Alan Ginsburg and Marshall S. Smith December 6, 2013
General goals • Improve utility and coherence of the NAEP background variables. • Examine current structure and list of variables – suggest interim steps to improve list. • Suggest theoretically and empirically based framework for indicators. Purpose: create a stable, flexible approach to monitor and better understand trends in NAEP data. • Create indicators that combine policy and/or practice related combinations of variables that deserve tracking over time.
Why key education indicators (KEI’s)? • KEI’s are statistics that regularly measure the health of important conditions that may influence the outcomes of the education system over time. • KEI’s are part of a balance- scorecard approach focused on fairness and quality of education to complement NAEP assessment-based outcome indicators. • KEI’s make sense of dozens of NAEP contextual questions: bring together currently diverse context variables around research-based education performance criteria.
Why KEI’s for NAEP? We already have Condition of Education • NAEP is unique • Gets inside schools with combination of perspectives (students, teachers, principals). • Links schools with reports of out-of-school conditions affecting learning. • Questions embedded in a national assessment – respondents may take questions more seriously and readers better understand NAEP results. • Supports disaggregated information: • Student SES and race/ethnicity, classroom and school variation. • State and urban district breakouts. • Makes possible two or three views (4th, 8th, 12th) of students, teachers, principals. • Provides cross-sectional and possible cohort changes for indicators as well as achievement scores.
Current contextual variables • 1441 8th grade math variables. Vast majority not been used in past two years. See list of variables mentioning computers for 4th grade reading. (examples on following slides) • Now unused variables might be important to use (teacher perceptions of other teachers = measure of school mindset). • Need some core variables fixed or improved -- e.g. current measures of school SES are weak and compromised. Individual SES not clearly defined. • Full list cumbersome, discourages some researchers - suggest that researchers be able to select from a core list of currently live background variables to enable measuring differences over time. Full list of variables would be accessed separately. • KEI’s would provide a framework to help identify, organize, and consistently measure a set of important context variables.
Avoid data overload: Focus on currently available data on computers and instruction
From variables to indicators • NRC developing national indicators --very few for each sector. • NSF developing STEM indicators – focused on part of education sector. • NAGB unique opportunity to link indicators regularly over time with students, teachers, classrooms, schools, achievement results. • Propose, as an example, 4 evidence-based composite indicators formed from a total of 5 student, 6 teacher and 3 school variables. • Eventually expert teams would need to develop indicators – also SES team recommendation.
Example: evidence-based KEI’s: student factors • Student Factors: (basic variables: ethnicity, gender, ELL, disability) + (5 indicators) • SES (Have expert team create SES KEI’s as recommended by study committee) • Home Educational Climate (parent support, place to study, talks with, not to child etc.) • Preschool Experiences (one, two years of preschool, parent reads to child, sets boundaries) • Student motivation (effort, hard work more important than luck, engage with school, likes and goes to school) • After-school education opportunities (formal and informal)
Example of evidence based KEI’s: school quality factors • School Quality (basic variables: place, size, type, composition) + 6 indicators (KEI) • Teacher quality (student view of quality, education degree in field, experience, dispositions &mindset) • Teacher professionalism (Seeks help to improve, supports other teachers, seeks growth year after year, enjoys work, engaged in professional networks.) • School climate (excessive student absenteeism, school safety, teacher expectations, teachers support each other, principal trusted, mindset) • Effectively implement strong standards and curriculum. • School effectively uses technology to teach (access, use, quality) • Continuous improvement throughout (use formative assessment, pd focused on improving, admin processes [e.g. HR], etc.)
Examples of ways to create KEI’s • Indicators can be single or combinations of multiple variables. Some basic information such as ELL would probably be a single variable. Most indicators would require combining multiple variables. • Variables should be combined based on theory –the SES report, for example. • Possibly by using regression / IRT scaling to create weights for each of a set of variables or weights may be reached through theory or judgment. • Or by creating composite categories indicating, for example, highly qualified teacher, as measured by having achieved four of six measured attributes. • Or in other ways.
Examples of possible KEI’s • Teacher Quality • School Climate for Learning • Student Motivation • Technology • Socio-Economic Status (SES)
Three-component composite indicator: school climate for learning School Climate for Learning • Teacher Student Expectations • More favorable • Less favorable • Student • Misbehavior • More favorable • Less favorable • Student Excessive Absenteeism • More favorable • Less favorable Composite • Composite • All more favorable • Composite • Mixed favorable • Composite • All less favorable
School-climate composite: student attendance (sub-indicator 1)
School-climate composite: teacher expectations (sub-indicator 2)
School-climate composite: student misbehavior (sub-indicator 3)
School climate: two variable composite indicator of highly favorable and highly unfavorable school climate, 2003
School climate: three-variable composite indicator of very high school climate • Three-variable composite indicator • 39% of students in 2003 had 0-2 days absent the prior month, in schools with no more than minor discipline problems, and a math teacher with very positive expectations for students. Note: The three-variable composite indicator of teacher quality would ideally be displayed as above by school-poverty concentrations, but this is beyond the Data Explorer capacity.
Student motivation indicator: ex. science engagement variables
SES indicator: expert panel report • Key recommendations • 1. Family income, other indicators of home possessions and resources, parental education and occupational status components of SES measure. • 2. Neighborhood and school SES could be used to construct an expanded SES measure. • 3. Composite measures have many advantages. • 4. Recommendation: should form expert panel to develop an SES composite measure.
Summary: Proposed steps for NAEP • Make immediate improvements to NAEP Data Explorer: reduce number of variables dramatically. Recent, useful variables in a prominent file. Old, redundant, useless, in another file. • Develop indicator framework and initiate expert teams in several areas to develop individual and composite key education indicators and accompanying NAEP contextual items.
More detail on next steps for KEI’S • Review, revise, improve proposed KEI’s framework and indicators. • Create SES indicator – based on SES expert team. • Get expertise in each field to put variables and current and proposed questions into new KEI’s framework -- use psychometricians to build composite indicators. Start with a couple of areas. • Field test the new variables and indicators. • Build repository of articles, publications, etc. that used the NAEP variables and indicators for scholars to use.