NRTs and CRTs

1. NRTs and CRTs Group members: Camila, Ariel, Annie, William

2. NRTs—Norm-referenced Item Analysis • Item facility • Item discrimination

3. Why do we do item analysis? • Assemble a large number of items of the type you want on the test • make sure the items are well written and clear • Pilot the items • Analyze the results of the pilot testing • Select the most effective items/get rid of the ineffective items or revise the weak ones

4. The basic purpose of NRTs • To spread students out along a general continuum of language abilities making aptitude, proficiency, or placement decisions

5. Two item statistics used in IA of NRTS • Item facility(IF):the proportion of students who answered a particular item correctly. 45/50=.90---90% of the students answered the item correctly—the item is very easy • Item discrimination(ID):Caculate IF for the upper group and the lower group using AVERAGE (C2:C6) and AVERAGE (C15:C19) ID=IFu-IFl ex: IFu-IFl=.20

6. Item Discrimination (ID) • can be calculated by first figuring out who the upper and lower students are on the test • using their total scores to sort them from the highest score to the lowest • Equal numbers of students in three groups—high/middle/low • ID=Ifupper - Iflower

7. Ideal items in an NRT • Should have an average IFof .50 50% of the students answered correctly and 50% of them answered incorrectly • In reality, items rarely have an IF of exactly .50 • Those that fall in a range between .30 and .70 are usually considered acceptable for NRT purpose

8. Items within .30 to .70 range • The items among them that have the highest IDs should be further selected for inclusion in the revised test. • This process would help the test designer to keep only those items that are well centered and discriminate well between the high and the low scoring students

9. Conclusion for NRT • IF and ID are only appropriate for developing and analyzing norm-referenced tests • Used at the institutional level overall English language proficiency tests or placement tests • Not appropriate for developing and analyzing classroom oriented criterion-referenced tests like the diagnostic, progress, and achievement tests

10. CRTs-Criterion-referenced Item Analysis • Purpose of CRTs—to measure the amount (or percent) of material in a course or program of study that students know • Usually for purposes of making diagnostic, progress, or achievement

11. Two item statistics used in IA of CRTs 1.The difference index(DI):the item facility on the particular item for the posttest minus the item facility for that same item on the pretest pretest:10/50=.20; posttest:45/50=.90 DI=.90-.20=.70 • DI tells how much the students are improving between the pretest and posttest on each item • The higher the value of the DI, the better. • A value of 1.00 is a perfect difference index

12. Two item statistics used in IA of CRTs 2.The B-index:the item facility on the particular item for the students who passed the test minus the item facility for the students who failed • The B-index show how well each item is contributing to the pass/fail decisions that are often made with CRTs B-index=IFpass - IFfail=14/14-0/6=1.0-.00=1

13. P.21 • The B-index tell us how well each item is contributing to the pass/fail decision on this test at the cut-point. • Like ID and DI statistics, the higher the B-index, the better. • A perfect B-index would be 1.00.

14. Conclusion-CRTs • When should these indices be used? To analyze the items on a CRTs for purposes of revising the test • The items with the highest values should generally be kept in both cases, DI and BI • Making these decisions not as simple as it is for NRT development, because a CRT item may not be performing well in terms of these statistics for many reasons.

15. Reasons that a CRT item may not be performing well • The item is written/ working poorly • The objective the item is testing is vague • Ss are not ready to learn this particular objective • One/ all of the teachers did not teach this particular objective or teaching it poorly • The materials are confusing with regard to this particular objective • The statistics cannot tell you exactly what is wrong even though they can point you to places in your curriculumwhere something is not working well

16. Some common-sense analysis of the entire situation needs to be done • To revise the CRT or other aspects of your curriculum such as the objectives, the materials, the teaching, etc.,

17. What the statistics help you? • Help you figure out where to focus your energies • The DI will tell you how well each item fits the objectives of your curriculum • The BI will tell you how each item is contributing to the pass/fail decision that you must make at whatever cut-point you are using.

18. 2 Definitions for “Criterion” in CRT 1.Refers to the material being taught in the course --CRT would assess the particular learning points of a particular course or program • This definition fits very well with the difference index, which indicates how well each item fits objectives of the curriculum

19. 2 Definitions for “Criterion” in CRT 2. The standard of performance (or cut-point for decision making) that is expected for passing the test/course --CRTs –used to assess whether students pass or fail at a certain criterion level (or cut-point) • Fits very well with the B-index, which indicates how well each item is contributing to the pass/fail decision that you must make at whatever cut-point your are making

20. What should you focus? • If you are interested in the degree to which your items are reflecting the material in your course….DI • If you are interested in the degree to which your items are helping you make decisions at a certain cut-point…BI • If you are interested in both statistics…DI and BI

21. What should they not be used? • Not used to analyze the effectiveness of norm-referenced items

22. What is the Ultimate Goal? • To produce a curriculum and CRTs that match each other such that you get high difference indexes and high B-indexes.