Data Warehousing, Mining, and Security: Developments and Directions

1. Data and Applications Security Developments and Directions Dr. Bhavani Thuraisingham The University of Texas at Dallas Data Warehousing, Data Mining and Security March 2012

2. Outline • Background on Data Warehousing • Security Issues for Data Warehousing • Data Mining and Security

3. What is a Data Warehouse? • A Data Warehouse is a: • Subject-oriented • Integrated • Nonvolatile • Time variant • Collection of data in support of management’s decisions • From: Building the Data Warehouse by W. H. Inmon, John Wiley and Sons • Integration of heterogeneous data sources into a repository • Summary reports, aggregate functions, etc.

4. Example Data Warehouse Data Warehouse: Data correlating Employees With Medical Benefits and Projects Users Query the Warehouse Could be any DBMS; Usually based on the relational data model Oracle DBMS for Employees Sybase DBMS for Projects Informix DBMS for Medical

5. Some Data Warehousing Technologies • Heterogeneous Database Integration • Statistical Databases • Data Modeling • Metadata • Access Methods and Indexing • Language Interface • Database Administration • Parallel Database Management

6. Data Warehouse Design • Appropriate Data Model is key to designing the Warehouse • Higher Level Model in stages • Stage 1: Corporate data model • Stage 2: Enterprise data model • Stage 3: Warehouse data model • Middle-level data model • A model for possibly for each subject area in the higher level model • Physical data model • Include features such as keys in the middle-level model • Need to determine appropriate levels of granularity of data in order to build a good data warehouse

7. Distributing the Data Warehouse • Issues similar to distributed database systems Branch A Branch A Branch B Branch B Branch B Warehouse Branch A Warehouse Central Bank Central Bank Central Warehouse Central Warehouse Distributed Warehouse Non-distributed Warehouse

8. Multidimensional Data Model

9. Indexing for Data Warehousing • Bit-Maps • Multi-level indexing • Storing parts or all of the index files in main memory • Dynamic indexing

10. Metadata Mappings

11. Data Warehousing and Security • Security for integrating the heterogeneous data sources into the repository • e.g., Heterogeneity Database System Security, Statistical Database Security • Security for maintaining the warehouse • Query, Updates, Auditing, Administration, Metadata • Multilevel Security • Multilevel Data Models, Trusted Components

12. Example Secure Data Warehouse

13. Secure Data Warehouse Technologies

14. Security for Integrating Heterogeneous Data Sources • Integrating multiple security policies into a single policy for the warehouse • Apply techniques for federated database security? • Need to transform the access control rules • Security impact on schema integration and metadata • Maintaining transformations and mappings • Statistical database security • Inference and aggregation • e.g., Average salary in the warehouse could be unclassified while the individual salaries in the databases could be classified • Administration and auditing

15. Security Policy for the Warehouse Federated Policy Federated Policy for Federation for Federation F2 F1 Export Policy Export Policy Export Policy Export Policy for Component A for Component B for Component B for Component C Generic Policy Generic Policy Generic policy for Component A for Component B for Component C Component Policy Component Policy Component Policy for Component A for Component B for Component C Security Policy Integration and Transformation Federated policies become warehouse policies?

16. Security Policy for the Warehouse - II

17. Secure Data Warehouse Model

18. Methodology for Developing a Secure Data Warehouse

19. Multi-Tier Architecture Tier N: Data Warehouse Tier N: Secure Data Warehouse Builds on Tier N Builds on Tier N - - 1 1 * * Each layer builds on the Previous Layer Schemas/Metadata/Policies * * Tier 2: Builds on Tier 1 Tier 2: Builds on Tier 1 Tier 1:Secure Data Sources Tier 1:Secure Data Sources

20. Administration • Roles of Database Administrators, Warehouse Administrators, Database System Security officers, and Warehouse System Security Officers? • When databases are updated, can trigger mechanism be used to automatically update the warehouse? • i.e., Will the individual database administrators permit such mechanism?

21. Auditing • Should the Warehouse be audited? • Advantages • Keep up-to-date information on access to the warehouse • Disadvantages • May need to keep unnecessary data in the warehouse • May need a lower level granularity of data • May cause changes to the timing of data entry to the warehouse as well as backup and recovery restrictions • Need to determine the relationships between auditing the warehouse and auditing the databases

22. Multilevel Security • Multilevel data models • Extensions to the data warehouse model to support classification levels • Trusted Components • How much of the warehouse should be trusted? • Should the transformations be trusted? • Covert channels, inference problem

23. Inference Controller

24. Status and Directions • Commercial data warehouse vendors are incorporating role-based security (e.g., Oracle) • Many topics need further investigation • Building a secure data warehouse • Policy integration • Secure data model • Inference control

25. Data Mining for Counter-terrorism

26. Data Mining Needs for Counterterrorism: Non-real-time Data Mining • Gather data from multiple sources • Information on terrorist attacks: who, what, where, when, how • Personal and business data: place of birth, ethnic origin, religion, education, work history, finances, criminal record, relatives, friends and associates, travel history, . . . • Unstructured data: newspaper articles, video clips, speeches, emails, phone records, . . . • Integrate the data, build warehouses and federations • Develop profiles of terrorists, activities/threats • Mine the data to extract patterns of potential terrorists and predict future activities and targets • Find the “needle in the haystack” - suspicious needles? • Data integrity is important • Techniques have to SCALE

27. Data Mining for Non Real-time Threats Clean/ Integrate Build modify data Profiles data of Terrorists sources and Activities sources Mine Data sources the with information about terrorists data and terrorist activities Report Examine final results/ results Prune results

28. Data Mining Needs for Counterterrorism: Real-time Data Mining • Nature of data • Data arriving from sensors and other devices • Continuous data streams • Breaking news, video releases, satellite images • Some critical data may also reside in caches • Rapidly sift through the data and discard unwanted data for later use and analysis (non-real-time data mining) • Data mining techniques need to meet timing constraints • Quality of service (QoS) tradeoffs among timeliness, precision and accuracy • Presentation of results, visualization, real-time alerts and triggers

29. Data Mining for Real-time Threats Rapidly Integrate Build sift through data and data real - time discard models sources in irrelevant real - time data Mine Data sources the with information about terrorists data and terrorist activities Report Examine final Results in results Real - time

30. Data Mining Outcomes and Techniques for Counter-terrorism

31. Example Success Story - COPLINK • COPLINK developed at University of Arizona • Research transferred to an operational system currently in use by Law Enforcement Agencies • What does COPLINK do? • Provides integrated system for law enforcement; integrating law enforcement databases • If a crime occurs in one state, this information is linked to similar cases in other states • It has been stated that the sniper shooting case may have been solved earlier if COPLINK had been operational at that time

32. Where are we now? • We have some tools for • building data warehouses from structured data • integrating structured heterogeneous databases • mining structured data • forming some links and associations • information retrieval tools • image processing and analysis • pattern recognition • video information processing • visualizing data • managing metadata

33. What are our challenges? • Do the tools scale for large heterogeneous databases and petabyte sized databases? • Building models in real-time; need training data • Extracting metadata from unstructured data • Mining unstructured data • Extracting useful patterns from knowledge-directed data mining • Rapidly forming links and associations; get the big picture for real-time data mining • Detecting/preventing cyber attacks • Mining the web • Evaluating data mining algorithms • Conducting risks analysis / economic impact • Building testbeds

34. IN SUMMARY: • Data Mining is very useful to solve Security Problems • Data mining tools could be used to examine audit data and flag abnormal behavior • Much recent work in Intrusion detection (unit #18) • e.g., Neural networks to detect abnormal patterns • Tools are being examined to determine abnormal patterns for national security • Classification techniques, Link analysis • Fraud detection • Credit cards, calling cards, identity theft etc. BUT CONCERNS FOR PRIVACY