1 / 8

HTST Evaluation Notes: Outline of Stable Version of HTST

The stable version of HTST includes an HT sense knowledge base, auxiliary sub-lexicons, and contextual features extracted from OED word sense definitions. It is evaluated on six test texts annotated by experts, focusing on precision, impact of OED contextual information, and time filtering. Observations highlight the potential for further improvement in precision and thematic code tagging. The integration of OED data and the need for larger, reliable test sets are emphasized for future development.

patterson
Download Presentation

HTST Evaluation Notes: Outline of Stable Version of HTST

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HTST Evaluation Notes

  2. Outline of Stable Version of HTST • Stable version of HTST contains: • HT sense knowledge base • Auxiliary sub-lexicons and data extracted from HT, e.g. highly polysemous words, polyseme density etc. • Context feature (USAS tags) model data extracted from OED word sense definitions. • Main software modules • CLAWS • USAS • VARD • HT-OED based components developed in SAMUELS

  3. Evaluation • HTST is evaluated on six manually annotated test texts. • Test data: • Five test texts manually annotated by Fraser; • One EEBO test text manually annotated by Jane. • Full test data set will contain ten texts. • Evaluation criteria • General performance in terms of precision • Impact of OED contextual information • Impact of time filtering • Further evaluation on full test data is under way

  4. HTST Overall Performance Note: VARD is used for EEBO sample “1621-Newes-out-of-France”, but not used for other test data.

  5. Experiment with OED Information Note: OED information helped in most cases (modern English), but decreased precision for EEBO sample. Possible cause is that OED definitions are all written in modern English.

  6. Time Filtering on EEBO sample Published in 1621 (Main HT Codes)

  7. Time Filtering on EEBO Sample Published in 1621 (Thematic Codes)

  8. Observation • On average, about 82% precision is expected. • With proper parameter setting, thematic code tagging can reach nearly 88% on some types of texts. • Need further improvement by tuning implemented methods and introducing more reliable methods. • OED data contains noise caused by the inconsistent HT versions. If OED entries can be precisely mapped to latest HT codes in future, it should improve the tagger. • Larger reliable test data is needed for further development.

More Related