1 / 34

An Efficient Multi-Dimensional Index for Cloud Data Management

An Efficient Multi-Dimensional Index for Cloud Data Management. Xiangyu Zhang Jing Ai Zhongyuan Wang Jiaheng Lu Xiaofeng Meng School of Information Renmin University of China. Outline . Motivation Query Answering on the Cloud Related Work

myron
Download Presentation

An Efficient Multi-Dimensional Index for Cloud Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Efficient Multi-Dimensional Index for Cloud DataManagement Xiangyu Zhang Jing Ai Zhongyuan Wang Jiaheng Lu XiaofengMeng School of Information Renmin University of China

  2. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  3. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  4. Motivation • Cloud systems have been justified as brilliant for web search applications • Simple structure, mostly key-value pairs • Flexible, efficient for analytic work • However, they are insufficient for complex data management needs • No powerful language as SQL • Hard to process complex queries • Lack of efficient index structures

  5. Distributed Cloud base? • BigTable How to query on other attributes besides primary key? • HBase

  6. Motivation • As part of our Cloud-based DBMS project, we aim to build efficient index structure on the Cloud.

  7. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  8. Query Answering in the Cloud Fast locating of relevant slave nodes Efficient lookup on each slave nodes

  9. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  10. Related Work • S. Wu and K.-L. Wu, “An indexing framework for efficient retrieval on the cloud,” IEEE Data Eng. Bull., vol. 32, pp.75–82, 2009. • H. chih Yang and D. S. Parker, “Traverse: Simplified indexing on large map-reduce-merge clusters,” in Proceedings of DASFAA 2009, Brisbane, Australia, April 2009, pp. 308–322. • M. K. Aguilera, W. Golab, and M. A. Shah, “A practical scalable distributed b-tree,” in Proceedings of VLDB’08, Auckland, New Zealand, August 2008, pp. 598–609.

  11. Distributed Database • Data slicing in DDBS • Horizontal, vertical, etc. • Slice based on conditions • Check condition conflict on query processing • Data distribution on the Cloud is different and could be very complex if expressed as set of conditions • Condition check is too expensive

  12. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  13. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  14. EMINC: Node Bounding • Node cube of a table on a slave node • Value range of table on this node Node Cube: (1,1), (6,10)

  15. EMINC: Architecture Each leaf node corresponds to one node cube Use KD-Tree to maintain local index on slave nodes

  16. EMINC: Query Processing • Get query cube of the query and look up in the R-Tree to get relevant data nodes • 1<x<2, 3<y<4 => Query Cube: (1,3),(2,4) Query Cube Query Cube No Yes Node Cube Node Cube

  17. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  18. EMINC: Extended Node Bounding • Problem with single bounding • Bad performance for sparse node Many queries will be mislead to this node

  19. EMINC: Cube Cutting Single Node Cube with Low Accuracy Multiple Node Cube with High Accuracy

  20. EMINC: Cube Methods Random cutting Equal cutting Clustering-based cutting

  21. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  22. EMINC: Index Update Strategy • Index update issues: • Cubes may invalidate themselves after certain data update, thus need reconstruction • Insertion invalidates cube • Create a node cube containing new data • For regular maintenance of index • Cost estimation based update strategy

  23. EMINC: Cost Estimation Strategy • Cost of index update: • Recalculate cubes on local node • Transfer to master node and maintain R-Tree • Query performance will be affected • Benefit of index update: • More accurate query directing, less waste

  24. EMINC: Two Phase Method • After one update: • Wait for a time period of deltaT • deltaT expires, check if an update is needed • DetermindeltaT • Check for update • Assumption : Number of queries to be processed Total size of node cubes of this node

  25. EMINC: Phase One • After pervious update: • benefit = decrement-of-query/time* deltaT • We enjoy the benefit of pervious update for deltaT time period • cost = number-of-queries missed • Number of queries we could process if we use pervious update time to answer queries

  26. EMINC: Phase Two • benefit > cost => deltaT • After deltaTexpires, check if an update is needed. This check involves following: • Record update frequency • Expected benefit ratio • Performance requirement • We leave this as future work

  27. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  28. Evaluation • 6 machines • 1as master node • 5 slave nodes simulating 100~1000 nodes • Each machine had a 2.33GHz Intel Core2 Quad CPU, 4GB of main memory, and a 320G disk. • Machines ran Ubuntu 9.04 Server OS.

  29. Evaluation: Point Query

  30. Evaluation: Range Query

  31. Outline • Motivation • Query Answering on the Cloud • Related Work • EMINC: Index the Cloud Efficiently • Node Bounding • Extended Node Bounding • Cost Estimation based Index Update • Evaluation • Conclusion & Future Work

  32. Conclusion • In this paper we presented a series of approaches on building efficient multi-dimensional index on Cloud platform. • We developed the node bounding technique to reduce query processing cost on the cloud platform. • In order to maintain efficiency of the index, we proposed a cost estimation-based approach for index update.

  33. Future Work • Complete cost estimation model • Take replication of datainto consideration • Implement in Hbase to further verify performance

  34. Thanks Please visit our lab for more information:http://idke.ruc.edu.cn/

More Related