1 / 19

Taxonomy Validation: Case Study

Taxonomy Validation: Case Study. Richard Morey, Public Health Advisor, SAMHSA Office of Communications Joseph A. Busch, Founder and Principal, Taxonomy Strategies. Agenda. SAMHSA Health Information Network Role of Taxonomy Taxonomy Validation. What Does SAMHSA Do?.

carl
Download Presentation

Taxonomy Validation: Case Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Taxonomy Validation: Case Study Richard Morey, Public Health Advisor, SAMHSA Office of Communications Joseph A. Busch, Founder and Principal, Taxonomy Strategies

  2. Agenda • SAMHSA Health Information Network • Role of Taxonomy • Taxonomy Validation

  3. What Does SAMHSA Do? • Substance Abuse and Mental Health Services Administration • Part of the US Dept of Health and Human Services (HHS) • Supports State & local efforts to improve quality & availability of SA & MH treatment & prevention services • More Specifically: • Federal Grants to States and Community Orgs • Drug and alcohol treatment & mental health services referral • Statistics on Drug Abuse & Mental Health

  4. What Does SAMHSA Do? SAMHSA’s Health Information Network Call center, publications & information Customer “Touch Points”* Customer requests (600K+ Inquiries) Customer orders (10.7M shipped) Customer on the Web (8M+ Web visits) Customer community (40K email subs) Customers at conferences (100+ exhibits) How do we make sense of all this rich data? *stats are from the SAMHSA Health Information Network Annual Report 2008

  5. What Does SAMHSA Do? SAMHSA’s Health Information Network Mission: To connect people working in substance abuse and mental health fields and the general public to the latest information on prevention and treatment Vision: Share data to develop a more proactive, interactive knowledge management system

  6. SAMHSA Knowledge Management Project

  7. Role of Taxonomy in SHIN • Tagging content • Navigating (top level, drill down & contextual filters) • Searching (categories & synonyms) • Aggregating & reporting

  8. SAMHSA Taxonomy • Eight facets. • High-level; especially Substances. • Designed to create compelling groups or “buckets” of similar content; but not point to individual pieces of content.

  9. SAMHSA SHIN Taxonomy overview Taxonomy Subjects Contextual Filters Population Groups 38 Audience 23 Filter Facets 59 terms Substances Content Type 33 36 Conditions & Disorders 36 Subject Facets 301 terms Intervention & Treatment Topics 48 Professional & Research Topics 37 Geography & Locations 109

  10. SAMHSA SHIN Taxonomy validation exercise • Demonstrate that SAMHSA staff will be able to use the Taxonomy to easily tag and find content. • Validation tests: • Walk through & review candidate taxonomy. • Place 30 most popular queries (words & phrases) from search logs in the correct Taxonomy facet. • Tag content from a test collection and compare those tags to an established baseline. • 5 items selected from collection of 20. • 14 SAMHSA & IQ Solutions staff participated in one-hour one-on-one test sessions over three days.

  11. Term sorting data collection form

  12. Blind sorting of popular search terms (n=12) 84% of terms were correctly sorted 60-100% of the time. Results: Excellent • Difficulties • For Methadone, confusion when, in this case, a substance is a treatment. • For general terms such as Smoking, Substance Abuse and Suicide, confusion about whether these are Conditions or Research topics.

  13. Search terms sorting task user rating (n=12)

  14. Tagging template filled in Title: American Indian/Alaska Native Substance Abuse Treatment Services: 2004 Image Number: 18 Add any additional keywords that you think would be helpful in finding this item (that are not in the title or taxonomy): _JB_ Initials Was it easy / medium / difficult to tag this item? (circle one)

  15. Tagged examples test collection

  16. Content tagging consensus (n=244) Test subjects tagged content consistent with the baseline 41% of the time. Results: Good • Observations • Many other tags were reasonable alternatives. • Correct + Alternative tags accounted for 83% of tags. • Over tagging is a minor problem.

  17. Tagging exercise test subject rating (n=42) • Only 7% rated the task as difficult!

  18. Examples of taxonomy changes informed by validation exercise

  19. Contact Info Richard Morey, 240-276-2131, rich.morey@samhsa.hhs.gov Joseph Busch, 415-377-7912, jbusch@taxonomystrategies.com

More Related