Text m ining

1. Textmining michel.bruley@teradata.com Extract from various presentations: Temis, URI-INIST-CNRS, Aster Data …

2. Information context • Big amount of information is available in textual form in databases and online sources • In this context, manual analysis and effective extraction of useful information are not possible • It is relevant to provide automatic tools for analyzing large textual collections

3. Text mining definition The objective of Text Mining is to exploit information contained in textual documents in various ways, including … discovery of patterns and trends in data, associations among entities, predictive rules, etc. The results can be important both for: • the analysis of the collection, and • providing intelligent navigation and browsing methods

4. Text mining pipeline Unstructured Text (implicit knowledge) Information Retrieval Information extraction Knowledge Discovery Semantic metadata Structured content (explicit knowledge) Semantic Search/ Data Mining

5. Text mining process Text preprocessing Syntactic/Semantic text analysis Features Generation Bag of words Features Selection Simple counting Statistics Text/Data Mining Classification- Supervised learning Clustering- Unsupervised learning Analyzing results Mapping/Visualization Result interpretation Iterative and interactive process

6. Text mining actors Publishers Enriched content Annotation tools Tools for authors New applications based on annotation layers Richer cross linking based on content… Analysts Empowers them Annotating research output Hypothesis generation Summarisation of findings Focused semantic search… Libraries Linking between Institutional repositories Access to richer metadata Aggregation Aids to subject analysis/classification …

7. Challenges in text mining • Data collection is “free text”, is not well-organized (Semi-structured or unstructured) • No uniform access over all sources, each source has separate storage and algebra, examples: email, databases, applications, web • A quintuple heterogeneity: semantic, linguistic, structure, format, size of unit information • Learning techniques for processing text typically need annotated training • XML as the common model, it allows: • Manipulation data with standards • Mining becomes more data mining • RDF emerging as a complementary model • The more structure you can explore the better you can do mining

8. Data source administration File System Databases EDMS Intranet Internet XML Normalisation -subject -Author -text corpora -keywords Web Crawling On-line Databank Information Provider Format filter

9. Name Extractions Term Extraction Abbreviation Extraction Relationship Extraction Feature extraction Categorization Summarization Clustering Text Analysis Tools Hierarchical Clustering Binary relational Clustering TM Text search engine Web Searching Tools NetQuestion Solution Web Crawler Text mining tasks

10. Information extraction Extract domain-specific information from natural language text • Need a dictionary of extraction patterns (e.g., “traveled to <x>” or “presidents of <x>”) • Constructed by hand • Automatically learned from hand-annotated training data • Need a semantic lexicon (dictionary of words with semantic category labels) • Typically constructed by hand Keyword Ranking Link Analysis Query Log Analysis Metadata Extraction Intelligent Match Duplicate Elimination

11. Categorization Document collections treatment Clustering

12. Text Mining example:Obama vs. McCain

13. Aster Data position for Text Analysis Data Acquisition Pre-Processing Mining Analytic Applications Gather text from relevant sources (web crawling, document scanning, news feeds, Twitter feeds, …) Perform processing required to transform and store text data and information (stemming, parsing, indexing, entity extraction, …) Apply data mining techniques to derive insights about stored information (statistical analysis, classification, natural language processing, …) Leverage insights from text mining to provide information that improves decisions and processes (sentiment analysis, document management, fraud analysis, e-discovery, ...) Aster Data Fit Third-Party Tools Fit Aster Data Value: Massive scalability of text storage and processing, Functions for text processing, Flexibility to develop diverse custom analytics and incorporate third-party libraries

14. Ability to store and process massive volumes of text data Massively parallel data stores and massively parallel analytics engine SQL-MapReduce framework enables in-database processing for specialized text analytics tools Tools and extensibility for processing diverse text data SQL-MapReduce framework enables loading and transforming diverse sources and types of text data Pre-built functions for text processing Flexible platform for building and processing diverse analytics SQL-MapReduce framework enables creation of flexible, reusable analytics Embedded MapReduce processing engine for high-performance analytics Aster Data Value for Text Analytics

15. Aster Data Capabilities for Text Data • Data transformation utilities • Pack: compress multi-column data into a single column • Unpack: extract nested data for further analysis • Web log analysis • Sessionization: identify unique browsing sessions in clickstream data • Text analysis • Text parser: general tool for tokenizing, stemming, and counting text data • nGram: split text into component parts (words & phrases) • Levenstein distance: compute “distance” between words • Pre-built SQL-MapReduce functions for text processing Custom and Packaged Analytics Aster Data nCluster App App App App App App SQL-MapReduce SQL Aster Data Analytic Foundation Data Data Data