1 / 18

NTU Cloud

NTU Cloud. 2010/05/30. System Diagram. Architecture. Gluster File System Provide a distributed shared file system for migration NFS A Prototype Image storage space. Compute Img. C- Img. C- Img. C- Img. Storage Img. S - Img. Prototype Img. NFS. Gluster File System.

idona-armstrong
Download Presentation

NTU Cloud

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NTU Cloud 2010/05/30

  2. System Diagram

  3. Architecture • Gluster File System • Provide a distributed shared file system for migration • NFS • A Prototype Image storage space Compute Img C- Img C- Img C- Img Storage Img S- Img Prototype Img NFS Gluster File System

  4. Architecture • Prototype Image • Original Image e.g. Hadoop MPI • Compute Image • Modified Images for user • Do not preserve the content after cluster shutdown Compute Img C- Img C- Img C- Img Storage Img S- Img Prototype Img NFS Gluster File System

  5. XEN • A hypervisor • Virtualization

  6. Cloud Master • Monitor system state • Scheduling • Use NFS to store Prototype Image • Web server

  7. OpenNebula • A middleware • Provides an interface to manage virtual infrastructure (computation and network) • VM Migration => We use OpenNebula to manage VM deployment, migration and set up virtual local area network(VLAN).

  8. Gluster file system • User level distributed file system • Client/Server Architecture • Use TCP/IP to transfer data =>We use GlusterFS to build our share file system environment for VM live migration. =>Our deployment is "symmetrical" - every machine is both a server and a client.

  9. System Flow

  10. Hadoop Benchmark • Case 1 • M1 : Master + Slave-01 + Slave02 • Case 2 • M1 : Master • M2 : Slave-01 + Slave-02 • Case 3 • M1 : Master • M3 : Slave-01 + Slave-02 • Case 4 • M1 : Master • M2 : Slave-01 • M3 : Slave-02

  11. Sec Iteration

  12. Set 1 • M1&M2 has same CPU and Memory size. • HADOOP_HEAPSIZE=500MB • mapred.child.java.opts=100MB • RandomWriter 10M for 30Maps • Sortting • HDFS_BYTES_READ=210543161 • HDFS_BYTES_WRITTEN=210541669

  13. Reduce shuffle bytes=199629523 Sort Reduce shuffle bytes=203039958 Launched reduce tasks=4 Others=3 Exactly the same! Therefore, putting two VM into one machine performance slowdown to 88.92% two machine / single machine = 88.92 %

  14. Set 2 1.RandomWriter 10M for 30Maps 2.Sort HADOOP_HEAPSIZE=500MB mapred.child.java.opts=100MB

  15. RandomWriter Therefore, putting two VM into one machine performance slowdown to 80.70% two machine / single machine = 80.70 %

  16. RandomWriter

  17. Sort

  18. Current Progress • Xen 4.0 is ready on each node. • We can offer two kinds of images • Hadoop • MPI • Start up VMs to destination node automatically. • Configure MPI and Hadoop environment for use automatically.

More Related