Uncovering Hidden Knowledge through Data Mining

1. Chapter 7 Pages 304-309, 311, Sections 7.3, 7.5, 7.6 DATA, TEXT, AND WEB MINING

2. Data mining A process that uses statistical, mathematical, artificial intelligence and machine-learning techniques to extract and identify new knowledge from large databases Recognizes the untapped value of data in large databases You may unexpectedly strike rich in understanding relationships among data

3. Example Task: Find the best route to cover the territory

4. Challenge of finding relationships in large databases

5. Connect equal elevation points to make a contour map The dark vertical line shows the best route to cross the territory without falling off a cliff.

6. Once relationships are discovered, they can be used for prediction

7. Uses of Data Mining-1 • Classification Identify attribute of interest (eg. You want to classify who is likely to pay late) Examine all other attribute values of customer from data warehouse and locate the one that is most related to the attribute of interest (eg. monthly income level) • Mining Algorithm The most common algorithm used for Classification is Decision trees Gini Index: helps to determine where to find the split between two classes (eg. at what income level) - used in developing decision trees (see example on page 316)

8. Which product class is the best seller? Conclusion: Clay products with a price below $25!

9. Uses of Data Mining-2 • Segmentation Partitioning a database into groups in which the members of each group share similar characteristics • Mining Algorithm Clustering: The object is to sort cases into groups so that the similarities within the group are strong among members of the same cluster and weak between members of different clusters Eg. Companies with over 100 employees may share similar characteristics (eg. revenue size) than those with less than 100 employees. Knowledge can help with developing different policies when dealing with different type of companies

10. Uses of Data Mining-3 • Association A category of data mining algorithm that establishes relationships about items that occur together in a given record Eg. You may discover from data that senior students take elective courses together in the final semester Can be helpful to schedule courses People who buy a suit may also buy dress shirt People who buy swimwear may buy fins, goggles, cap, etc.

11. Uses of Data Mining-4 • Sequence discovery The identification of associations over time. Discovering the order in which events occur. • The algorithm can examine data and predict what event is most likely to occur next. Widely used in studying how visitors navigate a Web site. Helps to improve chances of making a sale.

12. Uses of Data Mining-5 Regression is a statistical technique that is used to map data to a prediction value • Forecasting estimates future values based on patterns within large sets of data Eg. Gasoline prices this month may predict next month’s sales of SUVs

13. Marketing Banking Retailing and sales Manufacturing and production Brokerage and securities trading Insurance Computer hardware and software Government and defense Airlines Health care Broadcasting Police Homeland security Data Mining Concepts and Applications Data mining applications

14. Text Mining Application of data mining to text files, typically freestyle text material Discovers new knowledge that is not obvious Examples: Examine all news services, cluster similar topics, create a new summary for each topic Find the “hidden” content of documents, including additional useful relationships, eg. Lies, deceptions, scams Not same as the search engine on the Web.

15. Text Mining – how is it done? It entails the generation of meaningful numerical indices/factors from the unstructured text and then processing these indices using various data mining algorithms Example: Extract each word from the document being text mined Eliminate commonly used words (the, and, other, etc) Combine synonyms and phrases Calculate weights for each term: tf factor (term frequency) – actual number of times a word appears in a document idf factor (inter document frequency) – across multiple documents High tf factor value of a given term indicates that the document topic is probably around the meaning of that term!

16. Text Mining - applications • Automatic detection of e-mail spam or phishing through analysis of the document content • Automatic processing of messages or e-mails to route a message to the most appropriate party to process that message • Analysis of warranty claims, help desk calls/reports, and so on to identify the most common problems and relevant responses

17. Web Mining The discovery and analysis of interesting and useful information from the Web

18. Web content mining The extraction of useful information from Web pages Eg. Search with the help of keywords in the Meta tags of the web page You can analyze the document content of the first 10 links of Google in a search response You can generate a summary of the contents automatically in a new document!

19. Web structure mining The development of useful information from the links included in the Web documents If a web site’s pages predominantly link to each other, you may consider the site to exist ‘independent’ If a collection of web sites are linked to each other heavily, it points to a web community or clan that share common interests Example application: Web structure mining can lead to better understanding of extremist groups

20. Web usage mining The extraction of useful information from the data being generated through webpage visits, transaction, etc. Clickstream analysis Uses cookies, number of logs, time of log, etc Can help profile users

21. Uses for Web mining • Determine the lifetime value of clients • Design cross-marketing strategies across products • Evaluate promotional campaigns • Target electronic ads and coupons at user groups • Predict user behavior • Present dynamic information to users

22. Data Mining Project Processes

23. Steps for Data Mining • Problem definition: Decide the measure to study and the suitable mining algorithm (see Exercise 11) • Data preparation: Design the cube and populate it relevant data from the data warehouse • Training: Run the mining algorithm on a subset of the data warehouse data for the system to learn to find segments, associations, etc among data • Validation: Run the ‘learnt’ model from previous step to the remaining subset of data and try to ‘predict’. Since you have historical data, you can verify if the ‘learnt’ model is any good. • Deploy: Implement to predict in real environment where you do not know the actual results.