1 / 48

New England Common Assessment Program

New England Common Assessment Program. NECAP Science Bias & Sensitivity Review With Reading Passage Review Supplement August 14-16, 2007 Killington, VT. Welcome and Introductions. State Departments of Education Staff: Michael Hock – Vermont Deb Fleurant – New Hampshire

travis-hodges
Download Presentation

New England Common Assessment Program

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New England Common Assessment Program NECAP Science Bias & Sensitivity Review With Reading Passage Review Supplement August 14-16, 2007 Killington, VT Science Bias & Sensitivity Review

  2. Welcome and Introductions State Departments of Education Staff: • Michael Hock – Vermont • Deb Fleurant – New Hampshire • Heather Heineke Agnew – Rhode Island Measured Progress Staff: • Harold Stephens – NECAP Program Director • Elliot Scharff – NECAP Science Program Manager • Amanda Smith – NECAP Program Manager • Josh Evans – NECAP Program Manager Science Bias & Sensitivity Review

  3. Logistics • Committee Member Expense Reimbursement Form • NECAP Nondisclosure Form • Bias & Sensitivity Guidelines for NECAP item development Science Bias & Sensitivity Review

  4. What Is Item Bias? • Bias is the presence of some characteristic of an assessment item that results in the differential performance of two individuals of the same ability but from different student subgroups. • Bias is not the same thing as stereotyping (but we don’t want stereotypes in NECAP either). • We need to ensure that ALL students have an equal opportunity to demonstrate their knowledge and skills. Science Bias & Sensitivity Review

  5. What is the purpose of sensitivity review? • The ability to answer an item correctly should reflect a student’s learning opportunities, not cultural opportunities or life experiences. • No student should be disadvantaged by item content that is insensitive or disrespectful to the student's ethnic, religious, or cultural background. • Think of "culture" in broad enough terms to include disability, language proficiency, socio-economic status, and regional differences. • Item insensitivity is not the same thing as bias, but it is a potential source of bias that should be identified and eliminated. Science Bias & Sensitivity Review

  6. The “Big Idea” of Bias-Sensitivity Review NECAP should measure the results of learning opportunities NOT life experiences Science Bias & Sensitivity Review

  7. How Do We Prevent Item Bias? • Item Development • Bias-Sensitivity Review • Item Review • Pilot-Testing Feedback • Pilot-Testing Data Analysis (DIF) Science Bias & Sensitivity Review

  8. Role of the Bias-Sensitivity Review Committee • Review assessment items • Complete independent ratings of each item and record comments • Be prepared to discuss “Mixed Reviews” and “Problem Items” with your review team; Finalize independent ratings Please Note: The Bias-Sensitivity Review Committee plays an advisory role. Final decisions on which passages and items to include in NECAP will be made by the NH, RI and VT DOEs. Science Bias & Sensitivity Review

  9. Role of the Bias-Sensitivity Review Committee The Bias-Sensitivity Review Committee DOES need to make recommendations that will help ensure… • Sensitivity to different cultures, religions, ethnic and socio-economic groups, and disabilities • Balanced gender roles that are free of gender stereotypes • Use of positive language, situations, and images • Use of context, situations, images, and examples that are universally understood across groups and cultures • Overall, that the test is free of items and text that may elicit strong emotions in specific groups of students, and as a result prevent those groups of students from accurately demonstrating their skills and knowledge • Overall, that the test is free of items that measure life experiences instead of learning experiences Science Bias & Sensitivity Review

  10. Role of the Bias-Sensitivity Review Committee • Reading Level • Grade Level Appropriateness • GSE Alignment • Language Structure and Complexity • Instructional Relevance • Accessibility • Overall Item Design The Bias-Sensitivity Review Committee DOESNOT need to make recommendations concerning… The Item Review Committees Are Addressing These Issues and Factors. Science Bias & Sensitivity Review

  11. The Bias-Sensitivity Review Process Process: • Day 1 – (1) Training, (2) Practice, (3) Clarification of Guidelines and Procedures, (4) The entire committee will review grade 8 science, (5) Break into two groups for independent review of item sets—Group 1 will review science for grade 4 and Group 2 will review science for grade 11 • Day 2 – (1) Break into small groups for re-evaluation and discussion of “Problem Items” (2) Finalize Independent Ratings, (3) Evaluate the Review Process and Provide Feedback • Day 3 – (1) Training on reading passage review (2) The entire committee will review grade 11 reading passages (3) Break into two groups for independent review of reading passages—Group 1 will review grades 4, 6, and 8 and Group 2 will review grades 3, 5, and 7 Science Bias & Sensitivity Review

  12. General Guidelines • Avoid issues and topics considered sensitive by the general public. • Keep in mind that some topics that are appropriate for classroom discussion may not be appropriate for assessment. • Consider context, setting, and instructional relevance; Bias-Sensitivity Review is not a “literal” process. • Think of “bias” as an assessment issue as well as a cultural issue, that is, a specific topic or feature of a passage or item that may disadvantage one of two otherwise equally matched groups of students. Science Bias & Sensitivity Review

  13. Abortion Birth control Child abuse or neglect Creationism Divorce Incest Occult/Witchcraft Rape Religious doctrine Sexual orientation Suicide Questioning/ignoring parent/teacher authority Topics to Avoid in Science Test Items Science Bias & Sensitivity Review

  14. Death Impact of humans on the environment Natural disasters Conservation, logging, fisheries Extinction Climate change, global warming Sexuality, pregnancy, reproduction, sexually transmitted disease Violence Human or animal research subjects, animal rights Pollution Obesity Science Topics that Need Careful Consideration Science Bias & Sensitivity Review

  15. Awareness Issues • Avoid economic, regional, cultural, or gender bias in questions, prompts, identified situations, graphics, or reading selections. • Avoid profanity unless it is used in a relevant direct quote. • Ensure that items and passages emphasize academic programs and learning rather than life experiences. • Remember that we are proud of our states and region and expect to see that pride reflected in assessment passages and items. • Remember that items should reflect scientific thinking, processes, and content and not political or personal opinions. Nor should items require students to present political or personal opinions on scientific issues. Science Bias & Sensitivity Review

  16. Item Review Rating Form New England Common Assessment Program Bias & Sensitivity Review Committee Item Review Evaluation Form Read the statement below and select the rating that best describes your evaluation of the item for potential bias and sensitivity concerns. “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Comments Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be impacted, and the extent of the potential impact within the subgroup(s). Science Bias & Sensitivity Review

  17. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  18. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  19. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  20. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  21. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  22. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  23. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  24. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  25. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  26. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 2  In a model of our solar system, a tennis ball is used to represent Earth. A marble would be a good representation of Mercury in this model because compared to Earth, Mercury is —A   brighterB   lighterC   smallerD   more solid 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  27. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  28. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  29. Bias-Sensitivity Examples and Non-Examples “This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions  If a woodland is cut down to make room for a large shopping mall, which of the following would be the most likely outcome for birds living in the area? a. They will build nests on rooftops b. They will move to a new woodland habitat c. They will add garbage to their diet d. They will stop mating 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  30. Information, Questions and Comments • Tim Kurtz Director of Assessment NH Department of Education (603) 271-3846 • Mary Ann Snider Director of Assessment and Accountability Rhode Island Department of Elementary and Secondary Education (401) 222-8492 • Michael Hock Director of Educational Assessment Vermont Department of Education (802) 828-3115 Science Bias & Sensitivity Review

  31. New England Common Assessment Program Bias & Sensitivity Review Reading Passage Supplement Science Bias & Sensitivity Review

  32. The “Big Idea” of Bias-Sensitivity Review NECAP should measure the results of learning opportunities NOT life experiences Science Bias & Sensitivity Review

  33. How Do We Prevent Item Bias? • Reading Passages are Selected by Item Developers (who are trained to recognize potential sources of bias) • Sensitivity Review of Reading Passages • Reading and Math Items are Developed • Bias-Sensitivity Review of Reading & Math Items • Item Review of Reading and Math Items • Field-Testing Feedback • Pilot-Testing Data Analysis (DIF) Science Bias & Sensitivity Review

  34. Abortion Birth Control Child Abuse/Neglect Creationism Divorce Incest Occult/Witchcraft Questioning Parental Authority Rape Religious Doctrine Sex/Sexuality Sexual Orientation Sexually Transmitted Diseases Suicide Obesity Topics to Avoid in Reading Passages & Test Items Science Bias & Sensitivity Review

  35. Death Drugs, Alcohol & Tobacco Family Issues Guns / Gun Control Handicapping Conditions Holidays Homelessness Misuse of Animals/ Animal Rights Murder Pregnancy Racism, Sexism & Ageism Religion (Except in Historical Context) Violence Topics Which May or May Not Be Acceptable Science Bias & Sensitivity Review

  36. Passage Review Rating Form Grade Level: “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students” Science Bias & Sensitivity Review

  37. Bias-Sensitivity Examples and Non-Examples My sister met Kyle at a high school dance and nine months later I was an uncle. When dad brought the broken-down mare home from the livestock auction, we all thought he had thrown his money away. Now there’s a handsome colt in the next stall, so I guess the old horse was a good deal after all. “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  38. What’s the Point? “Topics to Avoid: Sex/Sexuality” Not okay because the passage alludes to extra-marital sex between high school students Okay because our general public standard generally allows for the notion of animal parents and babies Science Bias & Sensitivity Review

  39. Bias-Sensitivity Examples and Non-Examples The Christmas Tree Fern is a common sight in the New England woodlands. Christmas is the most wonderful time of the year. “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  40. What’s the Point? “Topics Which May or May Not Be Acceptable: Holidays, Religion” Okay because “Christmas” refers to the name of the plant; Not a specific reference to religion Not okay because the reference is based on the writer’s cultural and religious values (not shared by all) Science Bias & Sensitivity Review

  41. Bias-Sensitivity Examples and Non-Examples Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives. Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.” “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  42. What’s the Point? “Avoid ECONOMIC, regional, CULTURAL or gender bias in questions, prompts, identified situations…” Okay because the writer’s culture and economic status are not factors in the passage Not okay because the passage assumes a cultural and economic status that won’t be shared by all test-takers Science Bias & Sensitivity Review

  43. Bias-Sensitivity Examples and Non-Examples It was ten years after Uncle Ned’s death when we found his old sea trunk in the attic. Mary cried and cried when she was told Uncle Ned had died. “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  44. What’s the Point? “Topics Which May or May Not Be Acceptable: Death” Okay because the death reference is used to set the passage’s time and place but is not a key plot feature Not okay because death is a key plot feature; deals with emotions that may be upsetting to some test-takers Science Bias & Sensitivity Review

  45. Bias-Sensitivity Examples and Non-Examples Pa and Uncle Jimmy drove the final nails into the new barn roof while Ma and I laid out supper. For settlers on the Great Plains in the 1840’s, a barn raising was hard work, but also a time for celebration Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach. “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  46. What’s the Point? “Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…” Okay because the gender specific roles in the passage are historically accurate and do not, per se, represent bias Not okay because the writer’s perspective represents a clear and unacceptable gender bias Science Bias & Sensitivity Review

  47. Bias-Sensitivity Examples and Non-Examples “Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.” That’s what I used to think until the day I spied Mary Jean McCubbins up in a tree, building the coolest tree fort I ever saw. “This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.” Rating Descriptions 4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree Science Bias & Sensitivity Review

  48. What’s the Point? “Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…” Okay because the writer identifies the gender bias in prior beliefs and attempts to correct the stereotype Science Bias & Sensitivity Review

More Related