1 / 23

The ILK Suite of Text Tools

The ILK Suite of Text Tools. Antal van den Bosch ILK Research Group Faculty of Humanties, Tilburg University http://ilk.uvt.nl Political Mashup Meeting Amsterdam, March 19, 2008. The ILK Text Tools. Text Quality Management Text normalization Spelling and grammar checking

gale
Download Presentation

The ILK Suite of Text Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The ILK Suite of Text Tools Antal van den Bosch ILK Research Group Faculty of Humanties, Tilburg University http://ilk.uvt.nl Political Mashup Meeting Amsterdam, March 19, 2008

  2. The ILK Text Tools • Text Quality Management • Text normalization • Spelling and grammar checking • Structured data cleaning • Text Mining • Entity recognition • Relation finding • Text Recommendation • Document recommendation • Expert recommendation

  3. ILK Text Tools Applications • Cultural Heritage • Historical texts: Royal Library, DBNL • Entity recognition: Naturalis field books • Structured data cleaning: Naturalis, Beeld & Geluid, Army Museum, Meertens • Service and media industries • Text mining: Textkernel B.V. • Recommendation: Trouw

  4. TICCL • Text-induced corpus cleanup • Martin Reynaert • Robust, scalable method for finding wordform variants • Sensitive to morphology and context • Knowledge-free Very large corpus indexes Dirty word list Linked word list

  5. TICCL • hartstochtelijk hartstochtelyk hartstochtelyke hartstochtlijk hartstochtlijke hartstochtlyk hartstogtelijk hartstogtelijke hartstogtelijks hartstogtelyk • wenkbrauwen wenkbraauwen wenkbraeuwen wenkbrauwen winkbraauwen wynbraauwen wynbrauwen • Nederland NEDERLANDEN Nederlan Nederland Nederlanden Nederlander Nederlandse Nederlandt Nederlandts Nederlandze Nederlansch Nederlanse Nederlant Nederlants Neederland Neerland Neerlands Neerlandts Neerlants Netherlands

  6. Other Text QM Tools • Knowledge-free, corpus-driven • Tokenization and sentence splitting • Grammar checking • All d/t/dt errors • gebeurd/gebeurt, word/wordt • Inflectional and derivational errors • Run-on/split detection • Word completion Dirty corpus Disambiguator Cleaner corpus

  7. MITCH: Mining Natural History • Piroska Lendvai, Marieke van Erp, Steve Hunt • Field books and registers describe objects in many valuable facets, • In ambiguous, elliptic language • In multiple languages • Describing animals, people, biotopes, geographical names, time expressions

  8. Cleaning and overhauling data Rhabdo-phis? ? Geophis Geophis? Actual value: Geophis Expected: Rhapdophis

  9. Entity type correction

  10. Entity detection in fieldbooks 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  11. Entity detection in fieldbooks → Number 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  12. Entity detection in fieldbooks → Number, Genus 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  13. Entity detection in fieldbooks → Number, Genus, Species 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  14. Entity detection in fieldbooks → Number, Genus, Species, Biotope 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  15. Entity detection in fieldbooks → Number, Genus, Species, Biotope, Collection Time 1 ex. Leptodactylus wagneri At base of tree on small island, primary forest, 20.45-22.00 u. RMNH 23865 Lithodytes lineatus, Brownsberg, aan voet, onder stuk rot hout, 13.07.1968, 8.45 u., RMNH 26076 Dorsolateraal strepen heldergeel, tekening op dijen vuurrood, veel feller als bij P. femoralis. Gonyocephalus auritus Meyer, 3 ex. (1 juv.), Misool. Hoedt 1867. RMNH 17656 Eleutherodactylus zeuctotylus 1 [vrouw] Lelygebergte, 4 km N.O. van airstrip, distr. Marowijne, Suriname, 19-VIII-1975, onder stuk hout, 610m, l [plus] d M. S. Hoogmoed.

  16. Entity detection in fieldbooks • Training on labeled examples • Easy: short, regular entities • Hard: longer textual descriptions • Metadata detection in description entities • Types of forest, soil, … in biotopes • Physical appearance, … in special comments • By automatically learning the “grammar” of these entities (ABL)

  17. Expert search Toine Bogers, A Propos project Two types Expert finding Expert profiling Evidence of expertise Content-based evidence Evidence from social networks Activity-based evidence Current results on academic workgroup Content-based not better than citation-base Number of citations just as good as PageRank “authorship = expertise”? not 100%

  18. Trouw Recommender • news article recommender for Trouw • recommend related stories for article posted online • editors provide feedback on recommendations • approved recommendations are automatically placed online

  19. Other ILK Text Tools • Translation • Memory-based, any pair of languages • Morpho-syntactic analysis: Tadpole • Part-of-speech tagging, lemmatization • Dependency parsing, 20 languages • Text-to-speech conversion • Dutch speech synthesizer: NeXTeNS • Word sense disambiguation, co-reference resolution, paraphrasing, named entity recognition.

  20. Thank you http://ilk.uvt.nl  Toine Bogers, Martin Reynaert, Piroska Lendvai, Marieke van Erp, Steve Hunt, Peter Berck, Ko van der Sloot, Herman Stehouwer, Menno van Zaanen, Tanja Gaustad, Sebastiaan Tesink, Erwin Marsi, Iris Hendrickx, Antal van den Bosch, Walter Daelemans   

More Related