1 / 28

Service - Oriented Middleware for Distributed Data Mining on the Grid

Service - Oriented Middleware for Distributed Data Mining on the Grid. Antonio C., Domenico T., and Paolo T. Journal of Parallel and Distributed Computing, 2008. 89721006 ，劉妘鑏. Outlines. Abstract Introduction Distributed Data Mining and Grids Grid services for Distributed Data Mining

fausto
Download Presentation

Service - Oriented Middleware for Distributed Data Mining on the Grid

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Service - Oriented Middleware for Distributed Data Mining on the Grid Antonio C., Domenico T., and Paolo T.Journal of Parallel and Distributed Computing, 2008 89721006，劉妘鑏

  2. Outlines • Abstract • Introduction • Distributed Data Mining and Grids • Grid services for Distributed Data Mining • The Knowledge Grid Framework • Weka4WS • Concluding Remarks

  3. Abstract (Cont.) • Distribution of data and computation allows for solving larger problems and executing applications that are distributed in nature. • The grid is a distributed computing infrastructure that enables coordinated resource sharing within dynamic organizations consisting of individuals, institutions, and resources.

  4. Abstract （Cont.） • Distributed data mining (DDM) is widely used in industrial, scientific and commercial applications to analyze large data sets maintained over geographically distributed sites. • The current study on DDM is mainly focused on two aspects：One is the architecture of DDM, and the other is the algorithm of DDM.

  5. Abstract • This paper discusses： --how grid computing can be used to support distributed data mining. --Two systems developed according to this approach are presented and discussed.

  6. Introduction（Cont.） • Data mining in distributed environments like the Internet, virtual organization networks, corporate intranets, sensor networks, and other decentralized infrastructures questions the suitability of centralized knowledge discovery architectures for large-scale data analysis in a networked environment.

  7. Introduction（Cont.） • Distributed data mining works by analyzing data in a distributed fashion and pays careful attention to the trade-off between centralized collection and distributed analysis of data. • When data sets are large, scaling up the speed of a data mining task is crucial. • Parallel knowledge discovery techniques address this problem by using high-performance multi-computer machines.

  8. Introduction（Cont.） • The increasing availability of such machines calls for extensive development of data mining algorithms that can scale up as we attempt to analyze data sets measured in terabytes and petabytes on parallel machines with hundreds or thousands of processors. • This technology is particularly suitable for applications that typically deal with very large amount of data (e.g., transaction data, scientific simulation, and telecom data) that cannot be analyzed on traditional machines in acceptable times.

  9. Introduction（Cont.） • Grid technology integrates both distributed computing and parallel computing, thus it represents a critical infrastructure for high-performance distributed knowledge discovery. • Grid computing can leverage the computing power of a large number of server computers, desktop PCs, clusters, and other kinds of hardware. Therefore, it can help to increase efficiency and reduce the cost of computing networks by decreasing data processing time, optimizing resources and distributing workloads, thereby allowing users to achieve much faster results on large operations and at lower costs.

  10. Introduction（Cont.） • Data mining algorithms and knowledge discovery processes are both compute and data intensive, therefore the grid offers a computing and data management infrastructure for supporting decentralized and parallel data analysis. • the grid provide general data mining services, algorithms, and applications that help analysts, scientists, organizations, and professionals to leverage grid capacity in supporting high performance distributed computing for solving their data mining problems in a distributed way.

  11. Distributed data mining on the Grid (Cont.) • the OGSA (Open Grid Services Architecture), defined grid services as an extension of Web services for providing a standard model for using the grid resources and composing distributed applications as composed of several grid services . • the OGSA provides a well-defined set of basic interfaces for the development of interoperable grid systems and applications.

  12. Grid services for distributed data mining (Cont.) • OGSA defines standard mechanisms for creating, naming, and discovering transient grid service instances, provides location transparency and multiple protocol bindings for service instances. • OGSA also defines mechanisms required for creating and composing sophisticated distributed systems, including lifetime management, change management, and notification.

  13. Grid services for distributed data mining (Cont.) • WSRF provides the means to express stateful resources and codifies the relationship between WS and stateful resources. • WSRF define basic services for supporting distributed data mining tasks in grids. Those services can address all the aspects that must be considered in data mining and in knowledge discovery processes from data selection and transport to data analysis, knowledge models representation, and visualization.

  14. the data mining phase takes from 96.1% to 99.6% of the total execution time if client and service are located in a LAG（Local Area Grid）. • the execution time in the WAG（Wide Area Grid）takes from 84.6% to 88.3% of the total time. • WSRF Overhead：sum of the various WSRF-related steps（resource creation, notification subscription, task submission, results notification, resource destruction.）accounts for an amount of time ranging from about 0.4% to 4.1%.

  15. The Knowledge Grid Framework (Cont.) • The Knowledge Grid framework is a system implemented to support the development of distributed KDD processes in a grid. It uses basic grid mechanisms to build specific knowledge discovery services. • The Knowledge Grid provides users with high-level abstractions and a set of services by which it is possible to integrate grid resources to support all the phases of the knowledge discovery process, as well as basic, related tasks like data management, data mining, and knowledge representation.

  16. The Knowledge Grid Framework (Cont.) DAS ：Data Access Services TAAS：Tools and Algorithms Access Services EPMS：Execution Plan Management Service RPS ：Result Presentation Services RAEMS：Resource Allocation and Execution Management Service KDS：Knowledge Directory Service KBR：Knowledge Base Repository KMR：Knowledge Metadata Repository KEPR：Knowledge Execution Plan Repository

  17. EPMS：Execution Plan Management Service RAEMS：Resource Allocation and Execution Management Service KDS：Knowledge Directory Service KBR：Knowledge Base Repository KMR：Knowledge Metadata Repository KEPR：Knowledge Execution Plan Repository

  18. The Knowledge Grid Framework (Cont.)

  19. The Knowledge Grid Framework (Cont.)

  20. Weka4WS (Cont.) • Weka4WS is a framework that extends the widely used open source Weka toolkit for supporting distributed data mining on WSRF-enabled grids. • Weka4WS adopts the WSRF technology for running remote data mining algorithms and managing distributed computations. • TheWeka4WS software prototype has been developed by using the Java WSRF library provided by Globus Toolkit 4 (GT4). All involved grid nodes inWeka4WS applications use the GT4 services for standard grid functionality, such as security, data management, and so on.

  21. Weka4WS (Cont.)

  22. Weka4WS (Cont.)

  23. 1.Resource Creation. 2.Notification Subscription. 3.Task Submission. 4.Dataset Download. 5. Data Mining. 6.Results notification 7.Resource Destruction.

  24. Concluding Remarks • Many experts in IT, science, finance, and commerce are recognizing the importance of scalable data mining solutions in their business. • From Bill Gates’ keynote speech at Supercomputing05: . . . And so now we need to take the techniques that have been developed for things like business intelligence and data mining that goes on around that and think how we can apply those in these realms as well, how we can take every step of the process and have it be very visual and only require as much software understanding as is absolutely necessary..

  25. Concluding Remarks (Cont.) • the grid can offer an effective infrastructure for deploying data mining and knowledge discovery applications. • we discussed systems and services for implementing grid-enabled knowledge discovery services by using dispersed resources connected through a grid. These services allow professionals and scientists to create and manage complex knowledge discovery applications composed as workflows that integrate data sets and mining tools provided as distributed services on a grid.

  26. Concluding Remarks • As an example of this approach, we described how the Knowledge Grid and the Weka4WS systems provide a higher level of abstraction of the grid resources for distributed knowledge discovery activities, thus allowing the end-users to concentrate on the knowledge discovery process without worrying about grid infrastructure details.

More Related