Data Integration: Achievements and P erspectives in the Last Ten Years

1. Data Integration:Achievements and Perspectives in the Last Ten Years AiJing

2. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

3. Motivation & Background • Data integration is a pervasive challenge faced in applications that need to query across multiple autonomous and heterogeneous data sources. • Data integration is crucial in large enterprises that own a multitude of data sources. • For better cooperation among agencies, each with their own data sources.

4. Enterprise Databases Legacy Databases Services and Applications Data Integration

5. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

6. Ten-Year Best Paper Querying Heterogeneous Information Sources using Source Descriptions. VLDB96 Alon Halevy a principal member of technical staff at AT&T Bell Laboratories, and then at AT&T Laboratories. • Main idea: the Information Manifold • led to tremendous progress on data integration • and to quite a few commercial data integration • products.

7. The Information Manifold • An implemented data integration system • Goal: provide a uniform query interface to a heterogeneous collection of Web data sources • Main contribution: the way it described the contents of the data sources it knew about. • IM contains declarative descriptions of the contents and capabilities of the information sources. (Source Description)

8. An example of complex query find reviews of movie directed by Woody Allen playing in my area three web sites join! 1. a movie site containing actor and director information (IMDB) 2. movie playing sources(e.g.,777film.com) 3. movie review sites (e.g., a newspaper)

9. wrapper wrapper wrapper wrapper wrapper Design time Run time Mediated Schema query reformulation Semantic mappings optimization & execution

10. Semantic Mappings Mediated Schema CD: ASIN, Title, Genre,… Artist: ASIN, name, … Information sources Mapping logic CDs Album ASIN Price DiscountPrice Studio Books Title ISBN Price DiscountPrice Edition Authors ISBN FirstName LastName Artists ASIN ArtistName GroupName CDCategories ASIN Category BookCategories ISBN Category

11. Source Source Source Source Source Global-as-View (GAV)(Previous approaches) Mapping: Mediated Schema CD: ASIN, Title, Genre,… Artist: ASIN, name, … R1 R2 R3 R4 R5

12. Source Source Source Source Source Local-as-View (LAV) Mapping: Mediated Schema CD: ASIN, Title, Genre, Year Artist: ASIN, Name, … Mediated View Mediated View Mediated View Mediated View Mediated View R1 R2 R3 R4 R5

13. benefits of LAV • Describing information sources became easier a data integration system could accommodate new sources easily • The descriptions of the information sources could be more precise describe precise constraints on the contents of the sources become easier

14. Query reformulation Mediated Schema A query posed over CD: ASIN, Title, Genre,… CD(A,T,G) a set of queries on the data sources CDs Album ASIN Price DiscountPrice Studio Books Title ISBN Price DiscountPrice Edition Authors ISBN FirstName LastName Artists ASIN ArtistName GroupName CDCategories ASIN Category BookCategories ISBN Category

15. Query Answering in LAV =Answering queries using views (AQUV) • a problem which was earlier considered in the context of query optimization Given a set of views V1,…,Vn, And a query Q, Can we answer Q using only the answers to V1,…,Vn?

16. AQUV • Query optimization & Supporting physical data independence • AQUV for data integration: • Not necessarily equivalent rewriting • Find maximally contained rewriting • Main AQUV Algorithms: • Bucket • Inverse rules • Minicon

17. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

18. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

19. Generating Schema Mappings • Look at that observation: • Who’s going to write all these LAV/GAV formulas (the semantic mappings between the sources and the mediated schema)? 1.create the source descriptions 2. writing the semantic mappings • This was the main bottleneck.

20. Techniques for Schema Mapping • semi-automatically generating schema mappings • Goal: create tools that speed up the creation of the mappings and reduce the amount of human effort involved. • Compare schema elements based on: • Linguisticsimilarities • overlaps in datavalues or data types • schema mapping tasksare often repetitive.

21. A Machine Learning Approach Mediated schema • Map multiple schemas in the same domain to the same mediated schema. • Learn from previous experience: • the manually created schema mappingsas training data • generalize from them to predict mappingsbetween unseen schemas. Given matches Predict new ones

22. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

23. Adaptive query processing • look at that observation: • Once we have mappings, how can we execute queries? • Traditional plan-then-execute doesn’t work. • Root: the dynamic nature of data integration contexts

24. Adaptive query processing • data integration system: the context is very dynamic and the optimizer has much less information than the traditional setting. • Two results: • the optimizer can’t decide a good plan • a plan may be arbitrarily bad. • Dynamic adjust query plan

25. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

26. XML characters for data integration • XML offered a common syntactic format for sharing data among data sources. • since it appeared as if data could actually be shared • integration systems using XML as the underlying data Model and XML query languages (XQuery)

27. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

28. Model Management • Goal: provide an algebra for manipulating schemas and mappings • With such analgebra: • complex operations on data sources simple sequences of operators in the algebra • Some of the operators in Model Management • create & compose mappings, merge & diff models

29. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

30. Q3 Q1 Q4 Q5 Q6 Q Q2 Peer Data Management Systems UW (Wisconsin) Stanford Berkeley LAV, GLAV DBLP CiteSeer UW (Washington) UW (Waterloo)

31. Two Additional Benefits • A P2P architecture offers a truly distributed mechanism for sharing data. • Every data sourceonly provide semantic mappings to a set of neighbors. • complex integrations emerge follows semantic paths • P2P architecture is more appropriate than a single mediated schema in data sharing context. • there is never a single global mediated schema • data sharing occurs in local neighborhoods of the network.

32. Buildingonthe Foundation • Generating Schema mappings • Adaptive query processing • XML • Model management • Peer-to-Peer Data Management • The Role of Artificial Intelligence

33. The Role of Artificial Intelligence • Description Logics describe relationships between data sources • data sources need to be represented declaratively • the mediated schema of IM was based on Classic Description Logic • Description Logics offered more flexible mechanisms for representing a mediated schema • Recent work: combine the expressive power of Description Logics with the ability to manage large amounts of data.

34. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

35. The Data Integration Industry • Late 90’s——commercialization • Enterprise Information Integration (EII): withouthaving to first load all the data into a central warehouse • the development of the EII industry • Technologies from research labs matured enough • The needs of data management • XML • Inappropriate: data warehousing solutions, ad-hoc solutions

36. A data integration scenario will participate in the application Query processing data sources a query posedoverthe virtual schema semantic mappings Execute with an engine that create plans that span multiple data sources a query over the data sources query reformulation mediated schema build query applications applications

37. Other EII Products • XML data model and XQuery Challenge: the research on integration for XML was only in its infancy • customer-relationship management Challenge: how to provide the customer-facing worker a global viewof a customer whose data is residing in multiple sources, and track information from multiple sources in real time.

38. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

39. Future Challenges • The factors of data integration challenges: • Social: Data integration is fundamentally about getting people to collaborate and share data. • complexity of integration • Data integration has been referred to as a problem as hard as AI, maybe even harder! • Our goal: create tools that facilitate data integration in a variety of scenarios.

40. Several Specific Challenges • Dataspaces: Pay-as-you-go data management • Uncertainty and lineage • Reusing human attention

41. Dataspaces • database system: create the schema first! • data integration system: create the semantic mappings first! fundamental shortcoming: long setup time! • Dataspaces: the idea of pay-as-you-go data management

42. Pay-as-you-go • offer some services immediately without any setup time, and improve the services as more investment is made into creating semantic relationships. • A dataspace should offer keyword search over any data in any source with no setup time.

43. Pay-as-you-go Data Management Dataspaces: Franklin, Halevy, Maier [see PODS 2006] Benefit Dataspaces Data integration solutions Investment (time, cost)

44. Several Specific Challenges • Dataspaces: Pay-as-you-go data management • Uncertainty and lineage • Reusing human attention

45. Uncertain data & data lineage • A necessity in data integration system • introspect about the certainty of the data • when not automatically determine its certainty, refer the user to the lineage of the data • Web search engines provide URLs along with their search results, so users can consider the URLs in the decision of which results to explore further.

46. Several Specific Challenges • Dataspaces: Pay-as-you-go data management • Uncertainty and lineage • Reusing human attention

47. Reusing human attention • achieving tighter semantic integration among data sources • Users’ any operation to data sources: Giving a semantic clue about the data or about relationships between data sources • Systems that leverage these semantic clues: obtain semantic integration much faster • an area for additional research and development

48. Outline • Motivation & Background • Best Paper: Information Manifold • Buildingonthe Foundation • Data Integration Industry • Future Challenges • Conclusion

49. Conclusion time data integration not so long ago a nice feature and an area for intellectual curiosity today a necessity • Today’s economy further emphasize the need for data integration solutions. • Thomas Friedman: The World is Flat.

50. A Framework for Deep Web Integration Our focuses Developing issue Developed issue Undeveloped issue