Information Integration

1. Information Integration 4/5 Information Integration on the Web (SA-2)

2. Information Integration on the WebAAAI Tutorial (SA2)Rao Kambhampati&Craig Knoblock Slides for Parts 1 and 5 are Available in hardcopy at the Front of the room Monday July 22nd 2007. 9am-1pm Information Integration on the Web (SA-2)

3. Information Integration • Combining information from multiple autonomous information sources • And answering queries using the combined information • Many Applications • WWW: • Comparison shopping • Portals integrating data from multiple sources • B2B, electronic marketplaces • Mashups, service composion • Science informatics • Integrating genomic data, geographic data, archaeological data, astro-physical data etc. • Enterprise data integration • An average company has 49 different databases and spends 35% of its IT dollars on integration efforts Information Integration on the Web (SA-2)

4. Deployed information integration systems… • Travel sites: Kayak, Expedia etc.. • Google Base, DBPedia • Map Mashups • Libra; Citeseer; Google-squared etc Information Integration on the Web (SA-2)

5. Deep Web as a Motivation for II • The surface web of crawlable pages is only a part of the overall web. • There are many databases on the web • How many? • 25 million (or more) • Containing more than 80% of the accessible content • Mediator-based information integration is needed. Information Integration on the Web (MA-1)

6. Services Webpages Structured data Sensors (streaming Data) ~25M Information Integration on the Web (SA-2)

7. Database View Integration of autonomous structured data sources Challenges: Schema mapping, query reformulation, query processing Web service view Combining/composing information provided by multiple web-sources Challenges: learning source descriptions; source mapping, record linkage etc. Blind Men & the Elephant: Differing views on Information Integration • IR/NLP view • Computing textual entailment from the information in disparate web/text sources • Challenges: Information Extraction Information Integration on the Web (SA-2)

8. Services --Warehouse Model-- --Get all the data and put into a local DB --Support structured queries on the warehouse Webpages Structured data Sensors (streaming Data) --Mediator Model-- --Design a mediator --Reformulate queries --Access sources --Collate results --Search Model-- --Materialize the pages --crawl &index them --include them in search results ~25M Information Integration on the Web (SA-2)

10. Information Integration on the Web (SA-2)

11. Dimensions of Variation • Conceptualization of (and approaches to) information integration vary widely based on • Type of data sources: being integrated (text; structured; images etc.) • Type of integration: vertical vs. horizontal vs. both • Level of up-front work:Ad hoc vs. pre-orchestrated • Control over sources: Cooperative sources vs. Autonomous sources • Type of output: Giving answers vs. Giving pointers • Generality of Solution: Task-specific (Mashups) vs. Task-independent (Mediator architectures) Information Integration on the Web (SA-2)

12. Dimensions: Type of Data Sources • Data sources can be • Structured (e.g. relational data) • Schema mapping (GAV/LAV) • Precise integration (Could I have done better by going to the sources myself?) • Text oriented • Information Extraction • Mixed Information Integration on the Web (SA-2)

13. Dimensions: Type of Output(Pointers vs. Answers) • The cost-effective approach may depend on the quality guarantees we would want to give. • At one extreme, it is possible to take a “web search” perspective—provide potential answer pointers to keyword queries • Materialize the data records in the sources as HTML pages and add them to the index • Give it a sexy name: Surfacing the deep web • At the other, it is possible to take a “database/knowledge base” perspective • View the individual records in the data sources as assertions in a knowledge base and support inference over the entire knowledge. • Extraction, Alignment etc. needed Information Integration on the Web (SA-2)

14. Dimensions: Vertical vs. Horizontal • Vertical: Sources being integrated are all exporting same type of information. The objective is to collate their results • Eg. Meta-search engines, comparison shopping, bibliographic search etc. • Challenges: Handling overlap, duplicate detection, source selection • Horizontal: Sources being integrated are exporting different types of information • E.g. Composed services, Mashups, • Challenges: Handling “joins” • Both.. Information Integration on the Web (SA-2)

15. Fully Query-time II (blue sky for now) Get a query from the user on the mediator schema Go “discover” relevant data sources Figure out their “schemas” Map the schemas on to the mediator schema Reformulate the user query into data source queries Optimize and execute the queries Return the answers Fully pre-fixed II Decide on the only query you want to support Write a (java)script that supports the query by accessing specific (pre-determined) sources, piping results (through known APIs) to specific other sources Examples include Google Map Mashups Dimensions: Level of Up-front workAd hoc vs. Pre-orchestrated (most interesting action is “in between”) E.g. We may start with known sources and their known schemas, do hand-mapping and support automated reformulation and optimization Information Integration on the Web (SA-2)

16. Dimensions: Control over Sources(Cooperative vs. Autonomous) • Cooperative sources can (depending on their level of kindness) • Export meta-data (e.g. schema) information • Provide mappings between their meta-data and other ontologies • Could be done with Semantic Web standards… • Provide unrestricted access • Examples: Distributed databases; Sources following semantic web standards • …for uncooperative sources all this information has to be gathered by the mediator • Examples: Most current integration scenarios on the web Information Integration on the Web (SA-2)

17. [Figure courtesy Halevy et. Al.] Information Integration on the Web (SA-2)

18. Collated Challenges • Source Selection • Which sources are likely to be most relevant to answer the query? • Source quality • Source overlap • Schema Mapping • How to map schemas of the sources to the mediator? • Record Linkage • How to couple data items that refer to the same entity • Information quality • Data with null values • Inconsistent data • Query reformulation • To convert query from the mediator schema to the source schema • To bring out relevant data despite incompleteness and inconsistency

19. Services Webpages Structured data Sensors (streaming Data) Rao’s Solution… ~25M Information Integration on the Web (SA-2)

20. Idea: Consider “endorsement” (a la A/H or PageRank) But there are no explicit links…. Information Integration on the Web (SA-2)

21. Information Integration on the Web (SA-2)

22. Information Integration on the Web (SA-2)

23. Services Webpages Structured data Sensors (streaming Data) Rao’s Solution… ~25M Information Integration on the Web (SA-2)

24. Case Study: Query Rewriting in QPIAD Given a query Q:(Body style=Convt) retrieve all relevant tuples Base Result Set AFD: Model~> Body style • Rewritten queries • Q1’: Model=A4 • Q2’: Model=Z4 • Q3’: Model=Boxster Re-order queries based on Estimated Precision Ranked Relevant Uncertain Answers We can select top K rewritten queries using F-measure F-Measure = (1+α)*P*R/(α*P+R) P – Estimated Precision R – Estimated Recall based on P and Estimated Selectivity

25. Services Webpages Structured data Sensors (streaming Data) Rao’s Solution… ~25M Information Integration on the Web (SA-2)

26. A Feasible Query Make =“Toyota”, Model=“Camry”, Price ≤ $7000 Toyota Camry $7000 1999 Want a ‘sedan’ priced around $7000 Toyota Camry $7000 2001 Camry Toyota $6700 2000 Toyota Camry $6500 1998 ……… What about the price of a Honda Accord? Is there a Camry for $7100? Solution: Support Imprecise Queries Imprecise Queries ?

28. Things beyond this were not covered in Spring 2010 Information Integration on the Web (SA-2)

29. Review of Topic 3: Finding, Representing & Exploiting Structure • Getting Structure: • Allow structure specification languages •  XML? [More structured than text and less structured than databases] • If structure is not explicitly specified (or is obfuscated), can we extract it? • Wrapper generation/Information Extraction • Using Structure: • For retrieval: • Extend IR techniques to use the additional structure • For query processing: (Joins/Aggregations etc) • Extend database techniques to use the partial structure • For reasoning with structured knowledge • Semantic web ideas.. • Structure in the context of multiple sources: • How to align structure • How to support integrated querying on pages/sources (after alignment)