250 likes | 336 Views
Multi-Site Virtual Cluster: A User-Oriented, Distributed Deployment and Management Mechanism for Grid Computing Environments. Takahiro Hirofuchi, Takeshi Yokoi, Tadashi Ebara, Yusuke Tanimura, Hirotaka Ogawa, Hidetomo Nakada, Yoshio Tanaka, and Satoshi Sekiguchi
E N D
Multi-Site Virtual Cluster: A User-Oriented, Distributed Deployment and Management Mechanism for Grid Computing Environments Takahiro Hirofuchi, Takeshi Yokoi, Tadashi Ebara,Yusuke Tanimura, Hirotaka Ogawa, Hidetomo Nakada, Yoshio Tanaka, and Satoshi Sekiguchi National Institute of Advanced Industrial Science and Technology (AIST) EVGM08
Background • Grid Computing • Large-scale • Distributed • Heterogeneous • Grid Middleware • Globus toolkit • gLite EVGM08
x Sites Management Cost Explosion Software Deployment and Management • Problems • Software complexity • Dependency • Configuration • Resource diversity • Hardware • x86, x64, memory, storage, … • Software • CentOS 4, CentOS 5, Solaris, … • Library versions • Organizations • Administration policy EVGM08
My Experience • VOMS Installation at 2007 spring • Virtual organization management system • Supports only SL3 (based on RHEL3 since 2003-12) • It was nightmare VOMS requires old GCC for C++. VOMS depends on GT4. Me, too…? GT4 requires new GCC for its patched OpenSSL Debian Etch CentOS 5 EVGM08
Our Concept • Virtualization • Isolate resources • Capsulate environments • Create new administrative domains Site A Site B Create virtual machines at each site Site C EVGM08
Our Concept • Virtualization • Isolate resources • Capsulate environments • Create new administrative domains Site A Site B Group distributed VMsfor a Virtual Organization Site C EVGM08
Our Concept • Virtualization • Isolate resources • Capsulate environments • Create new administrative domains Site A Site B Site C Multi-Site Virtual Cluster EVGM08
Multi-Site Virtual Cluster • Integrate distributed VMs • Single cluster view. • Allow single administrative domain • OS installation and full configuration • Enable easy system deployment • Large-scale nodes Application Application Deployment and configuration Scientific application Emulation testbed EVGM08 Site B Site A
System Components (1) Resource Virtualization Mechanism Site A Site B Site C EVGM08
System Components (2) Web Service API for Virtualized Resource Control Site A Site B Site C EVGM08
System Components (3) Easy management systemfor large-scale, distributed nodes Site A Site B Site C EVGM08
Resource Virtualization Mechanism • Design criteria • Create completely-isolated VMs (Virtual Cluster) • Our virtual cluster system • Physical nodes • VMware Server and Xen • VLAN • LVM and iSCSI • Free and Open Source • http://code.google.com/p/grivon EVGM08
Allocating a Virtual Cluster (1) Public Network Cluster Manager VMM VMM VMM Private Network EVGM08
Allocating a Virtual Cluster (2) Cluster Manager VMM VMM VMM Bridge to VLAN Create a new VLAN (eth0.1234) EVGM08
Allocating a Virtual Cluster (3) Cluster Manager VMM VMM VMM Attach via iSCSI Create new storage volumes LVM EVGM08
Allocating a Virtual Cluster (4) VM VM VM Cluster Manager VMM VMM VMM Attach via iSCSI Launch VMs on VMM LVM EVGM08
Web Service API • REST API • https://example.com/api/vc/ • /api/vc • List virtual clusters (GET), Create a new virtual cluster (POST) • /api/vc/1234/ • Get the status of VC 1234 (GET) • /api/vc/1234/vm • List the VMs (GET), Add/Delete a new VM(POST) • /api/vc/1234/vm/{0, 1, 2, 3, 4, 5} • Get the status of a VM, Start/Stop a VM (POST) • /api/vc/1234/vpn • /api/vc/1234/vpn/{0, 1, 2} EVGM08
Distributed Node Management • Exploit an existing cluster management system for the inside of a multi-site virtual cluster • Designed for physical clusters • Powerful node management • Node database, parallel command execution, automatic node installation • Integrate distributed VMs by Ethernet VPN • Bridge internal networks of single-site virtual clusters • Transparency for admins and users • Transparent package caching at each site • Package-based software installer • Quick installation and reconfiguration • Flexible customizability EVGM08
Managed by Rocks toolkit VM VM VM VM VM VM Package cacherepository Frontend Node Console Node DB PXE installation server VM VM VM Ethernet VPN among site-local VLANs EVGM08
PrototypeImplementation • Reservation portal • Site resource monitoring • Automatic reservation • NPACI Rocks 4.2 • VMware Server • VLAN, iSCSI • OpenVPN 2.0 • Squid 3.0 EVGM08
Evaluation • Demo over the Pacific • Software configuration • Parallel command line tools • Node status monitoring • Condor job sumission • Evaluate Scalability • WAN • Large number of VMs • Reconfiguration time • Network traffic EVGM08
Experiment Setting Node Config. DB 134 nodes, reconfigured with 900MByte packages 16 nodes AMD Opteron 244, 3GB Mem, Gb Eth x2 AMD Opteron 246, 6GB Mem, Gb Eth x2 EVGM08
134 Node Reinstallation over WAN 900MB programs per node Only 20 minutes for a134-nodes virtual cluster EVGM08
Cache Disabled WAN Traffic over VPN RTT 20ms 30MB/s Pre Cached 800KB/s Cache Enabled 10MB/s A cache server minimizesVPN traffic for reinstallation. EVGM08
Conclusion • Multi-site virtual cluster • Virtualization for Grid computing • Isolate resources • Give independent administrative domains • Easy-to-use UI • Future work • Amazon EC2 support • Live Migration (demo sc08@Authtin) EVGM08