Qualitative Study on Tox Town: Evaluation for Science Educators

1. Qualitative Research StudyonTox Town Prepared for: May 10, 2006

2. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

3. Background • As part of an ongoing qualitative evaluation of NLM websites, the National Library of Medicine conducted TWO online focus groups on Tox Town – a Web resource about environmental health and toxicology. • The primary objective of the study was to understand the value, usefulness, strengths, and weaknesses of the Tox Town site for potential users of the site who are high school and college educators in science fields. • The online focus groups on the Tox Town site were conducted on April 25, 2006.

4. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

5. Study Objectives • The main objective of this qualitative study on Tox Townwas to evaluate the site's value and usefulness to current and potential users. Additional objectives included: • Gauging the site's strengths and weaknesses, in terms of content and navigation • Understanding how respondents either currently use or intend to use the site (in and out of the classroom) • Getting reactions to terminology, acronyms, and other language-related content on Tox Town Target respondents. Screening parameters for respondents were as follows: • Group 1 was comprised of science educators employed primarily in high schools. • Group 2 w was comprised of science educators employed primarily on the college level. • Both groups were recruited primarily from the NSTA's 54th National Conference on Science Education, which took place in Anaheim, CA from April 6-9. A few additional respondents were recruited via listserv subscription. • All were asked to spend approximately 20 minutes viewing/navigating the site and completing a series of tasks prior to the discussion to familiarize themselves with Tox Town.

6. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

7. Methodology: Online Focus Group* • Testing consisted of TWO (2) 90-minute online focus groups. • Respondents were recruited via conference and listserv. • Each group consisted of SIX (6) participants. (See Appendix.) • The group was asked to visit the Tox Town site prior to the session. They were also asked to complete a series of tasks using the site. • All participants received an incentive payment of $45. • * The online focus group represents a qualitative methodology used for the purposes of ideation, brainstorming, and evaluation. Qualitative methodologies are based on a small sample size, and the findings are intended to be directional only, not projectable to the larger population.

8. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

9. Summary Conclusions • Tox Town • While very few respondents had known of Tox Town prior to the NSTA conference, reactions to the site were positive, and the majority of respondents anticipated incorporating its content into their teaching curricula. • According to respondents, simplicity, ease of use, reliability, and accuracy are the marks of a top science education site. Evidence in this evaluation yields data to support that Tox Town meets these criteria overall. • One critique of Tox Town, however, includes that the graphics seem more suited to a younger ("elementary") audience. This criticism was balanced with the general perception that the content is age-appropriate to secondary school and college-aged users. • While the neighborhoods on Tox Town (Town, City, Farm, and U.S.-Mexico border) are relevant scenes, the "town" vs. "city" distinction yields some confusion. • The idea of more comprehensive depictions of "rural" and "city" scenes emerges in the focus groups, such that they would include a non-farm Rural area and a more comprehensive City environment. • Other neighborhood scenes that might be relevant additions to the site include rivers, lakes, apartment complexes, landfills, and national parks.

10. Summary Conclusions (cont’d.) • User Preferences for Tox Town • Tox Town's greatest strengths are the neighborhood scenes, along with the A-Z list and the fact that the site content is offered in Spanish. • While the sound effects are appealing, there is some sense that it could be distracting, especially if several students are using the site simultaneously in a classroom environment. • On the other hand, the recommended mode for sound effects would be the ON mode, allowing for users to mute the sound if they choose (the argument being that users may not notice the sound effects if the default is OFF). • To some, greater "realism" is a desirable attribute in the depictions of the neighborhoods (Town, City, Farm, and U.S.-Mexico border). Others, however, feel that the current graphical provides an appropriately "universal" tone. • The graphics version of Tox Town is far preferable to the text version, though there is some concern that the graphics may slow information-gathering processes on the site.

11. Summary Conclusions (cont’d.) • Expectations for Tox Town • Anticipated usage of Tox Town is very high, with some respondents stating that they have already bookmarked the site and have been recommending it to colleagues and students. • Respondents agree that Tox Town is a highly valuable, reliable, and informative site for science education information. • Some recommend expanding the A-Z list of chemicals, while others feel that doing so could be overextending Tox Town's "reach." • There is a strong sense that with greater exposure, Tox Town could become an even greater educational resource within the science teachers' community. One respondent noted that Tox Town is a "very good resource that suffers only from being anonymous." Following are detailed findings from the testing on Tox Town.

12. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

13. Overall Findings: Tox Town • Teaching Science using Online Sources • Respondents in these focus groups included science teachers covering grades 8 through college and even graduate level. Subjects taught included biology, chemistry, earth science, environmental science, life science, physics, and toxicology. • The technology profiles of their schools and classrooms varied. Some had technologically-advanced schools, with multiple computer stations in the computer labs, while others are trying to "catch up." • Current sources of online science educational resources included word of mouth, colleagues/other teachers, listservs, Internet searches, and professional organizations such as the NSTA (National Science Teachers Association), the USGS (U.S. Geological Survey), and NSF (National Science Foundation). • Factors that impact a "standout" site included its simplicity, reliability, accuracy, and ease of access and navigation. • Some respondents also preferred that the site not require a plug-in. (Note: One user in the High School group was required to view the text-only version of Tox Town due to constraints on the school computer she used, which did not allow for downloads of any kind.)

14. Overall Findings: Tox Town • Prior familiarity with Tox Town and Initial Impressions • The majority of respondents were previously unfamiliar with Tox Town. • Impressions of Tox Town were positive, with respondents citing "good information" that was quick, easy, and engaging. • Some initial criticisms of Tox Town included comments that it was "too busy," that there was "too much information," and that it was easier to access the information using a text-only version, indicating that the graphics presenting an obstacle to speedy information-gathering. • A couple of respondents also felt that the graphics design seemed more appropriate for elementary students than for high school or college students. On the other hand, one respondents felt that the simple neighborhood design made them look "more universal." • The locations, however (e.g., town, city, U.S.-Mexico border, and farm), were useful and appealing overall.

15. Overall Findings: Tox Town • Best Features of Tox Town • Respondents felt that the best features of Tox Town were its neighborhood scenes/locations, links, and teacher activities. • Another strength of Tox Town was its availability in Spanish language. “access to good information from a variety of sources.” (Mary, High School) “The fact of the different locations yet having similar problems.” (Ken, College) “I appreciate its availability in Spanish. I have lots of ESL students.” (Richard, College)

16. Overall Findings: Tox Town • Areas of Improvement and Suggestions for Change • The audio or sound effects on Tox Town were distracting to most, appealing to few. With regard to the sound effects, one respondent noted that it might be "distracting in a classroom full of students on the site at the same time." • Interestingly, more respondents in the High School group than the College group felt negatively about Tox Town's sound effects. This discrepancy may be due in part to the age differences of their students, and possibly other factors as well. • Overall, respondents preferred to have the audio default to being in the ON mode, which allows for user choice to subsequently turn the audio OFF. • One respondent in the College group encountered difficulty in searching for information on a specific topic, as he describes here. • Another respondent suggested specific changes to the neighborhood graphical images. “When you are led to the site for medical problems [parasites, etc.] the selection process gets obscure … You are presented with a menu that is not intuitive.” (Ken, College) “Change the logos for the 4 sections. Make them bigger. If you use a taxi make it bright yellow. A boy and a dog doesn't signify much, etc.” (Sherm, College)

17. Overall Findings: Tox Town • Neighborhood Scenes • Most respondents both liked the neighborhood scenes depicted on Tox Town and felt that the number of scenes represented was appropriate. • Some were confused as to the distinction between Town and City. • Another respondent (in the High School group) noted that most of the vehicles in the U.S.-Mexico border graphic were headed toward Mexico rather than toward the U.S. Others felt that the images were not "realistic." “I found town and city confusing. In our area they pretty much mean the same thing.” (Mary, High School) “US-Mexico - the vehicles are headed into the country rather than to the U.S. Very traditional in scope.” (Cassie, High School) "I don't think the neighborhood scenes are very realistic.” (Tom, High School)

18. Overall Findings: Tox Town • Neighborhood Scenes (cont'd.) • The idea of multiple options or "areas" in each category. For instance, one respondent noted that there are rural areas that are not necessarily "farm" areas and also questioned the broad brush stroke implied with the term "city." My students are ‘wealthy’ in general … when you say [city] … they think Ghetto. The run down dirty aspects of a city. So the pics show a clean flourishing Toronto or Chicago they think big city outskirts with all of the problems that go along with it …” (Barbara, High School)

19. Overall Findings: Tox Town • Neighborhood Scenes (cont'd.) • There were suggestions for additional neighborhood scenes, if Tox Town were to expand, including the following: • Beach • Other shore location (ocean, river, or lake) • High-rise apartment or housing development • Landfill • Power plant • Recreational area (e.g., a national park)

20. Overall Findings: Tox Town • Other Topics • There was also high receptivity to new scenes being developed, particularly Arctic areas. Indoor environmental health concerns (e.g., indoor air quality, drinking water) were also confirmed as relevant topic areas for respondents' curricula. • The A-Z topic index was mentioned as an extremely useful tool in searching for relevant information on Tox Town but could be expanded further. "The index was the most convenient in finding information fast.” (Tom, High School) "Alphabetical, clean look, quick links, all I would add is a search function. I used it for brownfields and hair products. Graphics version index is on the home page, it should be added to all pages so that you can get to it from anywhere.” (Troy, High School) "Index was very helpful in searching.” (Jacqui, College) "The A-Z index may need to include a couple of items of toxics in candies and makeup used in some parts of Mexico and other Latin American countries … for the U.S.-Border section of this site … I suggest the section ‘A-Z’ disaster recovery include a very visible link on Hurricane Katrina and other in a section titled ‘Current Events.’” (Helda, College)

21. Overall Findings: Tox Town • Additional Chemicals • Some respondents felt that additional chemicals should be included in the A-Z index, such as carbon, nitrogen, and CFCs. • One respondent in the College group noted that the A-Z list was "too restrictive" and suggested adding some chemicals to the list. • On the other hand, some felt that the chemicals list was sufficient in its current form, and that expanding it might be too far-reaching in scope. “The list is far too restrictive for the graduate students once they've gone past a basic toxicology introduction. Even the ATSDR tox profile list is too restrictive, which is why HSDB is so good. Dioxin and Furans; Polycyclic Aromatic Hydrocarbons need to be on the list.” (Sharron, College) "The site is great for any level, but I don't think NLM should feel obligated to turn it into a comprehensive tox chemical site. Other NIOSH sites can fill that bill.” (Sherm, College)

22. Overall Findings: Tox Town • Readability Level and Age-Appropriateness • Most respondents felt that the reading level of Tox Town was appropriate for their students (both in the High School and the College groups). • Some felt that the graphics were targeted toward a younger (elementary) user but that the content was appropriate for secondary and post-secondary school users. "… I think you've likely hit 10-11 grade level. For my college students, UG and grad it is fine.” (Sharron, College) "Readability level appropriate for college level.” (Helda, College) "Grade level appropriate for secondary but not elementary (except for the graphics).” (Tom, High School) "Content: high school and up. Graphics--younger. Considering both, with content the prevailing factor, high school and up.” (Louise, High School)

23. Overall Findings: Tox Town • Additions to Tox Town -- Student Activities • Adding activities for students was a well-received idea. While the crossword puzzles and word searches were not popular, the idea of letter-writing to Congress was. Additionally, respondents had some suggestions, such as self-assessments and searches with links. "For high school - a link to current research and researchers. Maybe a place to download data. Using Tox Town as a springboard for research projects/links to their community data sources.” (Cassie, High School) "Student activity: track a chemical around the local ecosystem... and including the human being exposed to it...and the health aftermath. I also like the writing to your Congressperson idea.” (Louise, High School) "I think a simulation activity that allows students to be scientists that solve problems as a means of alternative assessment would be good.” (Tom, High School) "Webquests might be useful … A problem based activity using internet resources. There are a number of sites that include webquests. the National Air and Space Museum has a good one on black aviators, as an example.” (Ken, College) "It would be very useful to have a search engine in the home page and in each page of the site.” (Helda, College)

24. Overall Findings: Tox Town • Additions to Tox Town for Teachers • The "For Teachers" page was commonly overlooked. Once it was pointed out, this page was perceived as very useful. • One respondent suggested that more of the links could use descriptions. • Another respondent suggested adding more in-depth lessons and activities. “I liked some of the activity links. Descriptions on the links would be helpful.” (Troy, High School) “At this point the activity list is not very extensive. More in-depth lessons/activities would be helpful.” (Mary, High School)

25. Overall Findings: Tox Town • Anticipated Usage • Likelihood to return to Tox Town and use the site as an educational resource was high across both the High School and College groups. • Respondents also expect to recommend Tox Town to students and colleagues, other teachers. Very likely, will integrate it into my classroom lessons and for a research resource for assignments. I'll use it next year in both courses. I'll also use it as an informational resource." (Louise, High School) “I would return for information and quick links to other government sites on the topics; it would be a resource for students as well." (Tom, High School) “I plan to incorporate it in my next methods class." (Ken, College) "would definitely come back for personal use and to refer." (Jacqui, College) "I will definitely recommend this to our bio and environmental teachers.” (Barbara, High School) “I have already recommended it to several others.” (Sherm, College)

26. Overall Findings: Tox Town • Additional Exposure for Tox Town • Articles in the NSTA journal and other journals, an insert in science magazines (e.g., Science Scope or The Science Teacher), and biology and environmental science listservs were among the suggestions for informing potential users of Tox Town and periodic updates to the site. • Respondents concurred that Tox Town could benefit from greater exposure while simultaneously serving as a valuable tool for the science educator; some had already bookmarked Tox Town. Following is a series of recommendations that emerged from the results of these online focus groups. “NABT. College Board AP environmental science seminars this summer." (Troy, High School) "Journal of Environmental Education. I've seen Tox Town regularly at APHA, but not advertised in the journal." (Sharron, College)

27. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix

28. Recommendations • Recommendations that emerged from these focus groups included the following: • Consider the addition of neighborhood scenes such as a beach, apartment complex, landfill, and river or lake area. • Make the "For Teachers" page more prominent; this was an area of the site that was highly useful to teachers but commonly overlooked. • Add activities for students, such as self-assessments, quizzes, and problem-solving opportunities. • If appropriate, expand the chemicals list in the A-Z list. (Suggested additions included carbon, nitrogen, CFCs, and polycyclic aromatic hydrocarbons.) • Keep the sound effects default to ON, but provide a mechanism that easily permits the user to MUTE the sound effects in a situation that calls for such action. • Gain exposure for Tox Town via articles in science education journals (including the NSTA journal and/or newsletter), as well as via listservs and at conferences. Following are respondent profiles of the participants in these online focus groups.

29. Background • Study Objectives • Methodology: Online Focus Group • Summary Conclusions • Overall Findings: Tox Town • Recommendations • Appendix: Respondent Profile

30. Appendix: Respondent Profiles

31. Appendix: Respondent Profiles - Group 1 (High School educators)

32. Appendix: Respondent Profiles - Group 2 (College educators)

33. Contact: Mary Beth Solomon For inquiries and capabilities on Qualitative Analytics 201.434.0404 917.601.1273