Overview of field trial analysis procedures

1. Overview of field trial analysis procedures National Research Coordinators Meeting Windsor, June 2008

2. Content of presentation • Purposes of field trial analysis • Methodologies applied • IRT Rasch model • Factor analysis • Criteria used • Fit indices • Item and scale statistics

3. Purposes of field trial • Test feasibility of construct measurement • Review scalability of item material • Check test and questionnaire length • Compare different formats (items with and without “don’t know” category) • Inform on relationships between constructs and variables • Compare results from on-line and paper surveys

4. Data included in analysis • Data from 31 countries included in international analyses • Instrument data from • Student test (98 cognitive items) • Student questionnaire • Teacher questionnaire • School questionnaire • Regional instruments (cognitive and questionnaire data) • Comparison of on-line and paper surveys (international option)

5. Types of analysis • Review of frequencies and means • Review of correlations between variables and constructs • Review of reliabilities and item-score correlations • IRT (Rasch) Scaling results • Exploratory and Confirmatory Factor Analysis

6. Analysis reports • Part 1 in NRC(June08)2.pdf • Analysis of cognitive test items • Analysis of student questionnaire data • Four appendix documents (a, b, c and d) • Part 2 in NRC(June08)3.pdf • Analysis of teacher questionnaire data • Analysis of school questionnaire data • Two appendix documents (a and b) • Addition document on comparison of paper and online mode (NRC(June08)3c.pdf)

7. Cognitive test data analysis • Review of omitted, invalid and “not reached” responses • Analysis of item difficulties, discrimination and Rasch model fit • Differential item functioning • Gender groups • Countries • Analysis of dimensionality • Trend item analysis

8. Not reached items (medians)

9. The IRT “Rasch” Model • Modelling probability of getting a correct response • Modelling probability of getting an incorrect response

10. IRT curves

11. MC Item statistics (example)

12. Example

13. IRT models for categorical data • Extension of Rasch model with additional step parameters  • Partial credit model has different step parameters for each item

14. Partial Credit Model – probabilities

15. Test item difficulties and abilities

16. Gender DIF • Gender effect directly estimated with ACER ConQuest • Reflects difference in logits if item parameters had been estimated separately for males and females • Differences for combined effect > 0.3 flagged (effect * 2) • DPC item stats: separate effects reported • Generally few cognitive test items had Gender DIF

17. Item-by-country interaction

18. Item dimensionality • No clear pattern of dimensionality with regard to old CIVED and new ICCS items • High correlation between the two test parts (0.87) • High correlation for sub-dimensions • Cognitive dimensions: 0.89 • Content dimensions: 0.93

19. Review of CIVED trend items

20. Coder reliability

21. Summary for test item analysis • Very positive results regarding scalability of test items • Support for uni-dimensionality of test items • Few items to be deleted or modified • Open-ended items performed generally well (except one item)

22. National reports • Purpose: Checking of national item statistics and review of possible explanations • Only for cognitive test items • Graphical displays

23. Item Stats in graphical form

24. National item fit and discrimination

25. National Item difficulties and thresholds

26. National item review list

27. International Summary

28. Item statistics • DPC provided NRCs with item statistics • Review of • category frequencies • point biserials (correlations) • Rasch parameter and fit • Gender DIF information • Difficulty in percentage correct (national and international)

29. Questionnaire item analysis • Review of frequencies (including for omitted and invalid responses) • Comparison of different formats (with and without “don’t know” categories) • Analysis of scaling properties (reliabilities, Rasch modelling) • Analysis of dimensionality • Analysis of relationships between variables and constructs

30. Correlations • Reporting of Pearson’s correlation coefficients • Used to review whether expected relationships are found in data (e.g. correlations between indicators of social background) • Correlation with test performance regularly reported for student scales • Criteria (not “scientific”): • < 0.1 Not substantial • 0.1 – 0.2 Weak • 0.2 – 0.5 Moderate • > 0.5 Strong

31. Reliabilities • Cronbach’s alpha coefficient • Is influenced by number of items! • Item-by-total correlation • Criteria • < 0.60 Poor • 0.60 – 0.70 Marginally satisfactory • > 0.70 Good

32. Example of scale information

33. Exploratory factor analysis • Used for exploring dimensionality for sets of items • VARIMAX rotation • Assumes factors to be uncorrelated • PROMAX rotation • Assumes factors to be correlated • Not always reported as it was used primarily in preliminary analysis steps

34. Confirmatory Factor Analysis • Model estimation based on variances and covariances • LISREL and SAS CALIS estimates • Maximum Likelihood (items assumed to be continuous) • Analysis to confirm expected factor structure • Model fit indices indicate whether the model “fits the data”

35. Example of CFA

36. CFA by country

37. Fit indices • RMSEA (Root mean squared error approximation) • > 0.10 Poor model fit of) • 0.10 – 0.05 Marginally satisfactory model fit • < 0.05 Close model fit • RMR • > 0.10 Poor model fit • 0.10 – 0.05 Marginally satisfactory model fit • < 0.05 Close model fit • CFI (Comparative fit index) and NNFI (Non-normed fit index) • < 0.70 Poor model fit • 0.80 - 0.90 Marginally satisfactory model fit • > .90 Close model fit

38. IRT models for categorical items • Partial credit model models the response probability for each depending on the latent trait  • Item location parameter  • Step parameter 

39. ACER ConQuest models • ITEM+ITEM*STEP: Constrained model • Assumes all item parameters to be equal across countries • ITEM-CNT+ITEM*CNT+ITEM*STEP:Unconstrained model • Assumes item location parameters to be different across countries

40. Item-by-country interaction • Item-by-country interaction effects sum up to zero • For review, the median of the absolute values was taken as an indicator of measurement invariance across countries • Those values > 0.3 logits were interpreted as items with large item-by-country interaction

41. IRT result tables

42. Scope of analysis • Given the narrow timeframe for analysis not all of these analyses were carried out for all instruments • Reviews of frequencies, computation of scale reliabilities and exploratory factor analyses were done for all field trial instruments

43. Analysis of relationships between context variables • Correlation of school, teacher and student data aggregated at the school level • Results show quite a few of the expected relationships • Single-level regression analysis for test performance and expected electoral participation • 25 percent of variance in test performance and 21 percent of variance in index of expected electoral participation explained by models

44. General outcomes • Good scaling properties for most items • Some constructs and items not retained due to poor results • Comparison of formats • No substantial differences in outcomes but large differences in missing values • Proposal to omit “don’t know” categories • Encouraging results for measurement of socio-economic student background • Proposal not to retain household possession items

45. Questions or comments?