1 / 55

Making Computerized Adaptive Testing Diagnostic Tools for Schools

Making Computerized Adaptive Testing Diagnostic Tools for Schools. Hua-Hua Chang University of Illinois at Urbana-Champaign October 17, 2010. What is Adaptive Testing?. O riginally called tailored tests (Lord, 1970)

aure
Download Presentation

Making Computerized Adaptive Testing Diagnostic Tools for Schools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Making Computerized Adaptive Testing Diagnostic Tools for Schools Hua-Hua Chang University of Illinois at Urbana-Champaign October 17, 2010

  2. What is Adaptive Testing? • Originally called tailored tests (Lord, 1970) • Examinee are measured most effectively if items are neither too difficult nor too easy. • Θ: latent trait. Heuristically, • if the answer is correct, the next item should be more difficult; • If the answer is incorrect, the next item should be easier. • How adaptive test works? • An item pool, known item properties (such as difficulty level, discrimination level,.. • Algorithm, computer, and network • The core is the item selection algorithm • Need mathematicians help to design algorithm

  3. Sequential Design &Robbins-Monro Process (1955) • Numerous Refinements: • Engineering (Goodwin, Ramadge and Caines, 1981; Kumar, 1985) • Biomedical science (Finney, 1978) • Education (Lord, 1970)

  4. The Maximum Information Criterion (MIC) • Lord’s (1980) MIC method, the most commonly used method. • MIC would select items with high discrimination • There have been many other methods

  5. Item difficulty vs Item discrimination a=1 a=2 a=0.5

  6. In 2D Case: item whose volume is max should be selected Item inf surface Information volume Theta region

  7. From Theoretical Development to Large Scale Operation • Issues to be addressed: • Should CAT only use the best items? • It is common only 50% items are used • Is CAT more secure than paper/pencil test? • How to improve CAT test security? • How to control non-statistical constraints? • How to get diagnostic information? • How to make CAT affordable to many schools?

  8. Two NSF Grants and loads of Papers • Constraint-weighted design • Cheng, Chang, & Yi (2006), Cheng & Chang, (2008), Cheng, Chang Douglas, & Guo (2009), etc. • Establish theoretical foundation • Chang & Ying (2009) • Test Security • Chang & Zhang (2001), Zhang & Chang (2010) • Cognitive diagnostic CAT • McGlohen & Chang (2008) • Multi-dimensional CAT • Wang & Chang (in press), Wang & Chang (accepted for publication) • Large scale k-12 Applications in China • Liu, Yu, Wang, Ding & Chang (2010)

  9. CAT & Transformative Research • National Science Board (2007)unanimously approved a motion to enhance support of transformative research at the NSF. • All proposals received after Jan 5, 2008, will be reviewed against the criterion. • revolutionizing entire disciplines; • creating entirely new fields; or • disrupting accepted theories and perspectives • Many CAT researches are transformative!

  10. New Developments • Measuring Patient-Reported Outcomes • Conventional measures of disease such as lab results do not fully capture information about chronic diseases and how treatments affect patients. • CAT can be used to assess patients subjective experiences such as symptom severity, social well-being, and perceived level of health. • K-12 Applications • Large State testing • Teaching/learning, within School application • Diagnostic purpose • Web-based learning

  11. Challenges in NCLB Testing • Many items are too difficult to students • 70% math items may be too difficult • The influence of this kind of test taking experience on low-achieving students is not well-understood (e.g., Roderick & Engle, 2001, Ryan & Ryan, 2005; Ryan, Ryan, Arbuthnot, & Samuels, 2007). • Test security of NCLB • The # of security violations in P&P based NCLB testing in on the rise. • Documented cases of such incidents have been uncovered in numerous states including New York, Texas, California, Illinois, and Massachusetts. (Jacob & Levitt, 2003, and Texas Education Agency, 2007).

  12. CAT Has Glowing Future in the K-12 Context. • Why not use benchmark testing? • Adaptive Testing can do better. • Quellmalz & Pellegrino (2009): • more than 27 states currently have operational or pilot versions of online tests, including Oregon, North Carolina, Utah, Idaho, Kansas, Wyoming, and Maryland. • The landscape of educational assessment is changing rapidly with the growth of computer-administered tests.

  13. Objectives: 1. Making CAT diagnostic tool2. deliver the tool to schools

  14. How to get diagnostic information? • Post-hoc approach (non-adaptive) • perform CD after students completed CAT • Adaptive approach • Select the next item which provides the max info about the student’s strength and weakness • Need a model, item selection algorithm • Psychometric theory • Simulation study • Field test

  15. Cognitive Diagnosis Provide examinees with more information than just a single score. • How? By considering the different attributes measured by the test. • An attribute is a “task, subtask, cognitive process, or skill” assessed by the test, such as addition or reading comprehension.

  16. What should be reported to examinees? Traditional Testing: Cognitive Diagnosis: A set of scores: One for each attribute. (K is the total # of attributes.) A single score

  17. Why is this beneficial? Feedback from an exam can be more individualized to a student’s specific strengths and weaknesses. Julia R. Halle B.

  18. The Item-Attribute Relationship Which items measure which attributes is represented by the Q-matrix:

  19. Cognitive Diagnostic Models vector • Many models have been proposed • DINA model • Fusion model (Stout’s group) person Item

  20. The DINA Model Student i Item j

  21. How to adaptively select items? • No direct analogy to “match theta with b-parameter” • Regular CAT, b-parameter with • Now is a vector, called latent class

  22. The KL information Approach (Xu, Chang, & Douglass, 2004) • Let’s assume • The likelihood test is the most powerful test • Intuitively the j-th item selected should make large

  23. Taking expected value assume is true Select item j to make the following as large as possible

  24. j: item, c: attribute pattern 4 possible patterns Interim estimate for an examinee Item bank Demo Consider two attributes and four candidate items

  25. Demo Change the slipping/guessing parameters of the items • The magnitude of the non-zero values depends on the item slipping and guessing parameters

  26. Change the interim estimate to Demo The positions of the zero KL cells changed for item 2 & 3

  27. To explain the last table in the previous slide • “0” means this item provides no information to discriminate the interim estimate with another possible attribute pattern. • The magnitude of the non-zero values depends on the item slipping and guessing parameters • Which cell is zero depends on the q-vector and the examinee’s interim estimate. If for a particular item (e.g., item 4 in this demo), q-vector contains all zeros, all cells will be zero.

  28. Estimation Response data Students’ latent class Item parameters

  29. New Tests vs. Existing Tests • Existing Exams • Analyze the responses from an existing large-scale assessment from a Cognitive Diagnosis framework. • Examine the results across various methods of constructing a Q-matrix. • New Exams • Identify Attributes and Content validity structure • Writing items according to cognitive specifications • Pre-testing • Q-matrix validation

  30. Application 1: existing dataset • A simple random sample of 2000 examinees who took the • Grade 3 TAAS from Spring 2002 • Grade 11 TEKS from Spring 2003 • The Math & Reading portion of each test was analyzed by using the Fusion Model • Item selection methods • Kullback-Libler (KL) • Shannon Entropy (SHE), and etc. • Reference, e.g., • McGlohen & Chang (2008) • Download from http://www.psych.illinois.edu/people/showprofile.php?id=539 • Or, google Hua-Hua Chang

  31. Taxes 3rd grade reading assessment 6 attributes (Application 1) • The student will determine the meaning of words in a variety of written texts. • The student will identify supporting ideas in a variety of written texts. • The student will summarize a variety of written texts. • The student will perceive relationships and recognize outcomes in a variety of written texts. • The student will analyze information in a variety of written texts in order to make inferences and generalizations. • The student will recognize points of view, propaganda, and/or statements of fact and opinion in a variety of written texts.

  32. Why CD-CAT? How to help schools to Own and operate CD-CAT?

  33. Building CAT-Driven Assessment and Diagnosis to Improve StudentLearningChang & Ryan (IES Proposal) • Develop the technical foundations for a CAT system to meet NCLB accountability and to inform teaching and learning. • In alignment with race to the top (RTTT) priorities, the proposed CAT will include • individualized diagnostic information to provide teachers, schools, and states with more-precise information about student achievement levels along with valuable formative feedback to inform instructional planning.

  34. Address issues reviewers raised How to address issues such as schools have no money to buy and operate CD-CAT?

  35. New Technologies--- Schools can use existing PCs or MACs • Client/Server Architecture (CS) • CAT software has to be installed on each client computer ( large workload) • only applicable to Local Area Network (LAN) • Browser/Server Architecture (BS) • database is still on the server • nearly all the tasks concerning development, maintenance and upgrade, are carried out on the server.   • based on the Wide Area Network (WAN) • Advantages of BS • Low maintenance, no network programming

  36. Hardware and Network Design Item Bank

  37. Develop a CD-CAT system to show its applicability to improve teaching and learning Application 2: the china project

  38. Application 2:Level II English Proficiency Test • Pretest and Calibration of Item bank • Pretest • 38,662 students from 78 schools, 12 counties participated • Analyzing pretest data • Estimated the parameters of DINA model • Estimated the parameters of 3PLM model • Calibrate attributes of item again • If it fits well then stop, otherwise revise q-matrix and got 3 • Assembling the item bank with item parameters and specifications.

  39. Distribution of the students in pretest Red: Field Test Sampling Area Yellow and red: Current Implementation 40

  40. Linking Design Eg, this block has 10 anchor items, The locations of the anchor items in each booklet are the same (as they appear in anchor test). 41

  41. Item Writing • About 40 Excellent Teachers in Beijing • Process • Psychometric Training • Identify Attributes • Writing Items • Constructing Q-matrix • Pre-testing and check FITTING • Revise Q-matrix until fitting is ok; go to 5 if not • stop

  42. Item Selection Strategy • Shannon Entropy (SHE) procedure was applied to select next items • SHE (Tatsuoka, 2002, Xu, Chang, & Douglas, 2004, McGlohen & Chang, 2008) • Dual Information (McGlohen & Chang, 2004 and 2008) Cheng and Chang, 2007)

  43. Parameters Estimation • The knowledge state of examinee is estimated sequentially. • The Maximum posterior estimation (MAPE) method was used in the system. • The ability is estimated at the end of the test.

  44. Monte Carlo simulation Studies • Item selection rule • Content constraints (same test structure as Pretest) • Listening Dialog (item1-item10), the next items is selected within remaining Listening Dialog items in the item bank. • Short Talks (item11-item12), two items for a piece of speech is selected within the short-talk items in the item bank. • Grammar and Vocabulary (item17-item32), the next items is selected within remaining Grammar and Vocabulary items in the item bank. • Reading Comprehension (item33-item40), the next items is selected within remaining Reading Comprehension items in the item bank. • Item selection strategy • the item was selected according to Shannon entropy procedure

  45. Classification Accuracy & Evaluation Criteria • Evaluation criteria • Rate of pattern match (RPM) • Rate of marginal match (RMM) • average test information

  46. Field Test • SHE with content constraints • The adaptive test was web-based, consisting of 36 items and lasting for 40 minutes. • Number of Participants: 584 • 5th and 6th grade, from 8 schools in Beijing, China

  47. Validity Study • Evaluating the consistency of • CD-CAT system results with an existing English achievement test • a group of students took two exams • CD-CAT system results with Teachers’ evaluation outcomes.

  48. CD scores vs. scores of an achievement test The Consistence between levels and # of mastered attributes

More Related