1 / 56

Consequences of using Rasch models for educational assessment: Where are we today?

Consequences of using Rasch models for educational assessment: Where are we today?. Claus H. Carstensen , Institute for Science Education IPN Kiel, Germany IRDP Neuchâtel, January 14, 2008. Consequences of using Rasch models for educational assessment: Where are we today?. Outline

smcmullen
Download Presentation

Consequences of using Rasch models for educational assessment: Where are we today?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Consequences of using Rasch models for educational assessment: Where are we today? Claus H. Carstensen, Institute for Science Education IPN Kiel, Germany IRDP Neuchâtel, January 14, 2008

  2. Consequences of using Rasch models for educational assessment: Where are we today? Outline • Educational Assessment and the Rasch model • The Rasch model in brief • Assessing student competencies in PISA • A Rasch model for educational assessments • A Rasch model for longitudinal data • A Rasch model for competency profiles • Where are we today?

  3. Consequences of using Rasch models for educational assessment Educational Assessment and the Rasch model

  4. Educational Assessment • to diagnose a student‘s progress (to support teaching and learning) • to evaluate/mark a performance (in accountability systems) • to assess the quality of a system (to provide support to policy makers) compare educational outcomes • to the performance of others • or to a standard

  5. Educational Assessment Research questions in Educational Research • describe distributions of educational outcomes • analyze the relation of some outcome with conditions of teaching and learning • explain how outcomes like competence are related on teaching or context conditions

  6. How does the Rasch model help us with these issues? The Rasch model is a measurement model for educational outcomes • it assumes a latent trait that explains the probability of the item responses • it assumes unidimensionality or “Rasch homogeneity” of the items • this assumption can be tested empirically for any test and a given dataset

  7. How does the Rasch model help us with these issues? The “original” Rasch model (Rasch, 1960) addresses dichotomous responses, generalizations address more complex response data like • multi-categorical data (Rasch, 1961), • ordinal data (Andrich, 1978; Masters, 1982) • or even continuous rating scales (Müller, 1987)

  8. How does the Rasch model help us with these issues? Other generalizations model more heterogeneity in the response data, • Mixture Distribution Rasch Model (Rost, 1989; Yamamoto, 1987) • Multdimensional Rasch Models (Stegelmann, 1983; Andersen, 1985)

  9. How does the Rasch model help us with these issues? Further generalizations of the Rasch model combine measurement and data analysis into one model • the Linear Logistic Test Model LLTM (Fischer, 1972) models structures in a test, i.e. systematically constructed items. • Explanatory Item Response Models (De Boeck & Wilson, 2004) model structures in the test and in the population under investigation

  10. Outline again • Educational Assessment and the Rasch model • The Rasch model in brief • Assessing student competencies in PISA • A Rasch model for educational assessments • A Rasch model for longitudinal data • A Rasch model for competency profiles • Where are we today?

  11. Consequences of using Rasch models for educational assessment The Rasch model in brief

  12. The Rasch model in brief:Modeling item response probabilities Using the logit function as item characteristic curve ICC

  13. The Rasch model in brief:An assumption Assuming the response probability is only determined by • an person ability • and an item difficulty gives the model equation

  14. The Rasch model in brief:Modeling item response probabilities An item characteristic curve ICC

  15. The Rasch model in brief:Likelihood multiplying the model probability over items x subjects gives the likelihood • where the number of correct responses per subject and is the number of correct responses per items • is a sufficient statistic for a person’s ability

  16. The Rasch model in brief:model properties • the model is one dimensional • it assumes equally discriminating items • the order of items is the same for all persons • the order of persons is the same with all items (the graph shows ICCs of three items)

  17. The Rasch model in brief:parameter estimation • Parameters are estimated by maximizing the likelihood with respect to each parameter separately • A set of one estimation equation for each parameter has to be solved, which can be done with iterative maximization algorithms. • In a Newton Raphson algorithm, an estimation equation for a person parameter is:

  18. The Rasch model in brief: parameter estimation • JML Joint Maximum Likelihood) • jointcalibrationof item andpersonparameters, not unbiasedbecauseofincidentalparameters (oneparameter per person) • CML conditional ML • estimate item parametersonly,giventhesubjectabilitieswiththeirscores • MML marginal ML • estimate item parametersmaking an assumption (like a normal distribution) on thesubjectabilitydistribution

  19. Outline again • Educational Assessment and the Rasch model • The Rasch model in brief • Assessing student competencies in PISA • A Rasch model for educational assessments • A Rasch model for longitudinal data • A Rasch model for competency profiles • Where are we today?

  20. Assessing student competencies in PISA A Rasch model for educational assessments

  21. A Rasch model for educational assessmentsPISA study Now, looking at the PISA study as an example for Educational assessment Purpose of PISA • monitor the outcomes of educational systems • across participating countries • over time within participating countries

  22. A Rasch model for educational assessmentsSystem monitoring Multi matrixsamplingallowstousemoretaskthansinglestudentscouldwork on • only a fewitemsforeachstudentandassessmentdomain • moreinformation on aggregatedlevels • The item response model hastoequatethe different test form (booklets)

  23. A Rasch model for educational assessmentsPopulation level results • Results on competency distributions like means variances, percentiles or percentage above cutpoints are requested on the country level • Using individual level competency estimates leads to biased variance estimates for the population • Instead population parameters are estimated directly with latent regression models in MML • some simulation results (a few slides later) will illustrate this

  24. A Rasch model for educational assessmentsMultidimensional models • in PISA three domains are assessed, maths, reading & science • a multidimensional model assumes a multivariate normal distribution of the competencies • and estimates its parameters (variances & covariances)

  25. The response Model for longitudinal dataitem response model The Model presented here is a sub model of the MRCMLM (Adams, Wilson, Wang 1997, implemented in ConQuest) response model population model latent regression model

  26. A Rasch model for educational assessments Aggregated and individual level analysis • Population model results may (theoretically) be read from the covariance matrix and the regression coefficients • Multiple Imputations (Plausible values) are drawn to obtain complete data sets on the individual level for further analyses of the solution, using standard statistical software • given the appropriate conditioning model, hierarchical or structural equation models may be fitted using the plausible values

  27. Assessing student competencies in PISA A Rasch model for longitudinal data

  28. A Rasch model for longitudinal dataPISA-I-Plus Germany 2003 • 9th graders assessed in 387 classrooms, • 2 classrooms per school, • different types of school (Gymnasium, Realschule, Hauptschule) • Second assessment of the same students one year later:data from 297 10th grade classrooms in the analysis • Assessments in mathematics and science • Questionnaire information is available from schools, teachers, students and parents (first or both assessments)

  29. A Rasch model for longitudinal data PISA Germany 2003 General Research Questions • How do mathematical and scientific literacy grow/change from grade 9 to grade 10 • for the whole population and for particular subpopulations (gender, socioeconomic status, migration, type of school) • How are these changes related to context and treatment conditions? • Instruction in the last year and school level conditions • Student level variables, parental support, peers etc.

  30. A Rasch model for longitudinal data multidimensional modeling of time points • two dimensional response model assuming a bivariate normal distribution • Item parametersfixedacrosstime points, bothdimensionmeasurethe same construct • The latent correlationreflectstheconnectionbetweenobservedresponsesfromthe same personsat different time points Andersen (1985), CML estimated two dimensional model t0 t1 i21 i31 i22 i32 i11 i12

  31. A Rasch model for longitudinal data tamultidimensional modelling of timepoints reformulation of the dimensions as pretest proficiency and change(the change model) Embretson (1991), CML estimated difference model Fischer (2000, 2003), CML estimation and individual confidence intervals t0 t1-t0 i21 i31 i22 i32 i11 i12

  32. A Rasch model for longitudinal data the combined model Multidimensional and latent regression model: latent correlations between change and covariates are modeled • Submodel of MCML model (ConQuest ) z1 t0* z2 z3 t1-t0 z4

  33. A Rasch model for longitudinal data some results

  34. A Rasch model for longitudinal data Summary • A multidimensional MML estimated IRT model with latent regression to analyze true change was presented(which is a sub model of MRCMLM in ConQuest) • time point scores are modeled as multivariate normal distributed (latent correlation estimated) • change can be specified as latent dimension => latent correlation between change and background variables/ treatment assignments can be estimated

  35. A Rasch model for longitudinal data Simulation study Common practice: • In a first step, obtaining individual level point estimates, maybe from one dimensional IRT modeling (MLE, WLE) • In a second step, analyzing change (as differences or residuals, using simple or complex SEM/HLM models), In consequence, • the step two model will be based on point estimates with measurement error and not on a population model • Analysis results may be affected by “attenuation”

  36. A Rasch model for longitudinal data Simulation study Looking at • distributions of the change score (means and SD), • correlations between t0 and change, • in a sort of realistic setup for our PISA assessment How well do the different models, • the time points model • or the change model • reproduce the generating values? How well does the two step procedure work?

  37. A Rasch model for longitudinal data Simulation results: group means notes: r(t0;change) = 0 for this table, results very similar for other r(t0;change); t0 (t1) and t1 (t2) values from the Andersen model change values from Embretson model

  38. A Rasch model for longitudinal data Simulation results - group standard deviations notes: r(t0;change) = 0 for this table, results very similar for other r(t0;change); t0 (t1) and t1 (t2) values from the Andersen model change values from Embretson model

  39. A Rasch model for longitudinal data Simulation results - correlation t0 and change time point model • unbiased estimates from PVs, but large S.E. • biasin WLE estimates (regression to the mean?)

  40. A Rasch model for longitudinal data Conclusions - Simulation Using PVs to analyse results of a combined latent regression latent change model gives unbiased results • for all statistics under investigation • The two step procedure • might be used to analyze time point mean values on aggregated levels, • should not be used to analyze distributions of changeor correlations with change and other variables.

  41. Assessing student competencies in PISA A Rasch model for competency profiles skip

  42. A Rasch model for competency profiles:German Educational Standards for Mathematics Difficulty Levels Content(Overarching Ideas) curricular validity Competencies • Pragmatically differentiated in • 6 Competencies • 5 Overarching Ideas • 3 dificulty levels • a test of 313 Items was developed for grade 9

  43. A Rasch model for competency profiles: Test Construction – Competency Model • Overarching Ideas: • Number • Measurement • Space & Shape • Functional Relationships • Data & Stochastics • Difficulty • Theoretical levels of cognitive complexity • three levels • Competencies: • Mathematical arguing • Mathematical problem solving • Mathematical modeling • Using mathematical graphs • Using mathematicalsymbols & techniques • Mathematical communicating

  44. A Rasch model for competency profiles: Research Questions • What differential information do we get on the students? • On which reporting scales can (group) profiles be based? What about • the Overarching Ideas, the Competencies or • interactions of Overarching Ideas and Competencies?

  45. A Rasch model for competency profiles: Multidimensional Models I – Variance & Reliability

  46. A Rasch model for competency profiles: Interactions of Overarching Ideas and Competencies Why can’t the competencies measured with these items? • Do the competencies have different meanings within the 5 Overarching ideas? • Different meanings might indicate specific difficulties of the tasks for the competencies between different overarching ideas or might be due to the test construction (hopefully not) For the next step, Competencies will be estimated within Overarching Ideas (in 5 runs on separate sets of items)

  47. A Rasch model for competency profiles: Multidimensional Models II – Variance & Reliability Each row presents the results of one parameter estimation within the items of each Overarching Idea

  48. A Rasch model for competency profiles: Defining Interaction Models Looking at the covariances of the Competencies within Overarching Ideas, different interaction models were derived: • a long Interaction Model with 19 dimensions • a reduced Interaction Model with 15 dimensions • a short Interaction Model with 11 dimensions

  49. A Rasch model for competency profiles: Multidimensional Models III – Model Fit

  50. A Rasch model for competency profiles: Short Interaction Model – Variance & Reliability

More Related