1 / 27

Information Integration Across Heterogeneous Sources: Where Do We Stand and How to Proceed?

Information Integration Across Heterogeneous Sources: Where Do We Stand and How to Proceed?. Aditya Telang Sharma Chakravarthy, Yan Huang. Motivation. “ Retrieve castles near London that are reachable by train in less than 2 hours”

gladys
Download Presentation

Information Integration Across Heterogeneous Sources: Where Do We Stand and How to Proceed?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Information Integration Across Heterogeneous Sources: Where Do We Stand and How to Proceed? Aditya Telang Sharma Chakravarthy, Yan Huang

  2. Motivation • “Retrieve castles near London that are reachable by train in less than 2 hours” • “Find 3-bedroom houses in Houston within 2 miles of a school and within 5 miles of a highway and priced under 250,000$” • “Retrieve French restaurants within 1 mile of IMAX Theater in Dallas, Texas” • …

  3. Motivation Search engines Meta-search engines Faceted search engines Domain-specific portals

  4. Current Scenario • “Retrieve castles near London that are • reachable by train in less than 2 hours” - Decision Making Process - Manually Combine Results to arrive at a decision London Train schedules Trains from London Castles Near London

  5. Ideal Scenario Intent: Retrieve castles near London that are reachable by train in less than 2 hours Information Integration System Actual Results for the intent Challenges

  6. Focus of the Paper • Identify the salient challenges needed to be encountered to address this problem • Survey existing work to identify the challenges for which acceptable solutions are available • Propose a framework that could provide potential solutions towards the problem

  7. Broader Challenges Intent specification and formulation Query processing and optimization Discover of sources, their schemas and characteristics Data Extraction, Integration and Ranking Result Visualization Issues with inconsistency, security, privacy, …

  8. Intent Specification • “Retrieve castles near London that are reachable by train in less than 2 hours” • Keyword-based (e.g., search engine query)? • Structured (e.g., SQL) ? • Unstructured (e.g., natural language) ? • Template/Form/Menu-based (deep Web query) ?

  9. Query Processing • The number of sources to be integrated are much larger than in a normal database environment. • Heterogeneous sources (RDBMS, websites, web services, etc.) do not provide the same processing capabilities found in a typical database system (such as the ability to perform joins). • Unlike relational databases, there might be restrictions on how a source can be accessed.

  10. Query Processing • In contrast to query optimization in DBMS, the query optimizer in information integration has little information about the data since it resides in remote autonomous sources • Web data sources are not necessarily database systems and may have different processing capabilities. • Hence, the query optimizer must consider the possibility of exploiting a data source’s query-processing capabilities.

  11. Discovery • Source discovery • Given the domain of travel, determine all possible source providing airfare information • Not a simple crawling process since categorization is necessary after crawling [Gal:VLDB’06] • Use of • search engines ? • web directories ?

  12. Discovery • Discovery of source schema and characteristics • Understanding source schema • Understanding query mechanism (for deep Web sources) • Understanding characteristics of sources

  13. Data Extraction • How to extract data for individual sub-queries? • APIs, Web services for deep Web? • Data extractors (e.g., Lixto, Florid) for surface Web? • Temporary storage of extracted data (becomes a critical issue when data can be large in size such as spatial data)

  14. Data Integration • Schema integration a complex challenge across domains [Gal:VLDB’06] • Additional challenges while integrating data • Inefficient execution of recursive integration plans • No support to dynamic service composition • Lack of operators to support GeoSpatial data types • No support for record linkage and object consolidation in the mediator can incorporate the source into a new or existing workflow

  15. Ranking • In context of integration, ranking has not been addressed as a significant challenge [Telang:ICDE’07] • When to rank? • Before integrating sub-query results? • After integrating sub-query results? • Source-independent ranking possible?

  16. Other Challenges • Visualization of results • Handling inconsistencies • Ensuring no breach of privacy and security • ….

  17. The Current Big Players • Industry-level • Google (Google Base) [Madhavan:CIDR’07] • IBM (Web Sphere) • Yahoo (Trip Planner) • Academia-level • Havasu [Kambhampatti:ICDE’05] • MetaQuerier [Chang et. al: VLDB’05, CIDR’07] • Ariadne [Knoblock: VLDB’02,03] • …

  18. The InfoMosaic Approach

  19. Knowledge-Base Knowledge Base • Identify different types of information needed for the domains and sources to answer a query. • Domain Knowledge – • Necessary information/knowledge required for elaborating and refining the query based on the domains and keywords provided by the user • Source Semantics – • Information store for modeling and maintaining all the necessary information for each source within a given domain Statistics Vocabulary Operators Domain Knowledge Source Semantics Metadata Ontology Attributes Schemas

  20. User Intent Specification • Specify intent that is more precise than a “search” but less rigid than a “SQL-query” • Ability to resolve concepts and their attributes elegantly with minimal user interaction • Effectiveness depends on user feedback and past query statistics [Telang:COMAD’08] Feedback-centric Query Specification Feedback Refined Query User Intent Knowledge-Base

  21. Multi-level Query Planning Query Planner & Optimizer Domain Level Plan • Evaluation is made at each stage to prune plans using relevant cost metrics. • Some of the additional cost metrics – • volume of data retrieved from each source • number of calls made to and amount of data sent by each source • quantity of data processed • the number of integration queries executed Domain-Level Refined Query … … Source-Level Source-Level SP-1 SP-2 Knowledge-Base

  22. Query Execution & Data Extraction Internet • Checking availability of sources, identifying attributes to be extracted (using the source semantics) and extracting data • Determining the output in XML and spatial data formats for storage and further querying • Reuse of previously retrieved results is an integral part of this task Query Results Data Store Spatial Data Repository XMLData Repository Query Plan Query Executor & Data Extractor Extracted Results Knowledge-Base

  23. Integration of Results Domain Level Plan • Generation of XQueries for combining extracted data • Develop external functions for XQueries to access spatial data • The result of the query will be transformed into a homogeneous schema for understanding and analyzing the results. Data Store Spatial Data Repository XMLData Repository Result Set Query Integrator Results Knowledge-Base

  24. Ranking Query Executor & Data Extractor • Two approaches to ranking [Telang:DBRank’07]– • Rank Before Integration: Applicable when user-specified metrics can be decomposed and applied to individual sub-queries • Rank After Integration: Applicable when user-specified metrics CANNOT be decomposed and applied to individual sub-queries Ranking Integrator

  25. To Conclude • Ideally, an information integration system should allow users to specify what information is needed without having to provide detailed instructions on how or from where to obtain the information. • A number of challenges need to be addressed by different research communities (AI, DB, IR, NLP, Semantic Web, …) • Existing work suggests we are on the right track • Our proposed framework (InfoMosaic) could be a further step in this direction

  26. Thank You !

More Related