1 / 16

Aspects of a Processing Grid Peter T. Kirstein P.Kirstein@cs.ucl.ac.uk

Aspects of a Processing Grid Peter T. Kirstein P.Kirstein@cs.ucl.ac.uk S ø ren-Aksel S ø rensen S.Sorensen@cs.ucl.ac.uk Stefano A. Street S.Street@cs.ucl.ac.uk Sheng Jiang S.Jiang@cs.ucl.ac.uk Department of Computer Science University College London. Processing scenarios.

errin
Download Presentation

Aspects of a Processing Grid Peter T. Kirstein P.Kirstein@cs.ucl.ac.uk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Aspects of a Processing Grid Peter T. Kirstein P.Kirstein@cs.ucl.ac.uk Søren-Aksel Sørensen S.Sorensen@cs.ucl.ac.uk Stefano A. Street S.Street@cs.ucl.ac.uk Sheng Jiang S.Jiang@cs.ucl.ac.ukDepartment of Computer Science University College London

  2. Processing scenarios • Batch processing • Large number of small (10’s mins) jobs • Typical for Bioinformatics and HEP. • Resource hungry but not fussy. • Large data sets. • Interactive experimentation • Single distributed job • User controlled (haptic steering) • Dynamic resource requirements • Performance sensitive (latency) • Visualization and steering

  3. Switch Cluster computing • Advantages • Cost effective solution to resource shortage • Simple security (isolated environment) • Easy management. • Problems • Switching cost (Ethernet) • Task mixing (delay vs. throughput) • Resource management (time sharing, performance protection)

  4. Switch Switch Switch Gigabit switch Cluster of clusters • Large systems can be created from a hierarchy of sub-clusters. • The UCL cluster uses this architecture • Still growth problems (eg. power and cooling) • Utilisation problems

  5. Internet Multi-domain clusters • Problems • Security • Load balancing • Resource discovery

  6. http://bioinf.cs.ucl.ac.uk/GTD

  7. http://bioinf.cs.ucl.ac.uk/GTD

  8. Web interface @ UCL Web interface @ EBI Web interface @ IC RDB of annotation @ UCL RDB of annotation @ IC RDB of annotation @ EBI Resource Management under GLOBUS Transparent Processing System CPUs @ EBI CPUs @ UCL-CS CPUs @ Biochem CPUs @ IC User interface via DAS Application

  9. Web-based Submission Interface Grid Distribution Manager Globus • • • • • • • • • SGE SGE SGE GenTREADER GenTREADER GenTREADER GenTREADER GenTREADER GenTREADER Distributed GenTHREADER Configuration • User ‘annotation’ submission committed via Web Form • Web service filters & divides sequence file into sub jobs

  10. JANet IC Front End UCL BioChem UCL CS UCL Backbone Front End Front End eProtein current production setup SGE GT2 IPv4

  11. JANet IC EBI Front End Front End UCL BioChem UCL CS UCL Backbone Front End Front End eProtein future setup SGE GT3 IPv6 VPFS

  12. IPv6 Advantages • Larger Address Space • Mobility Support • Built-in Security • Hooks for QoS • Better Auto-configuration • End-to-end Addressing • Deploys New Technologies

  13. Grid over IPv6 • Full availability of IPv6 will allow provision of better Grid services • Mobility support in distributed networks or global-scale networks • Performance potentially much better • Auto-configuration for resource discovery • Allow separation of addressing and routing • Better group communication • Better security

  14. Demonstrator Setup • Intel • Linux (Red Hat 8 – 2.4.18) • GT3 (alpha) • Java (Sun SDK 1.4.1) • IP-independent - Class InetAddress used everywhere • Inet4Address • Inet6Address • Tomcat “lightweight edition”

  15. Switch Switch Switch Gigabit switch Testing • GenTREADER setup • GT3 GRAM services • Local subnets

  16. Interactive use • Distributed systems use PVM/MPI2 • Dynamic scheduling through dummies • Dummy process scheduled by SGE. • Extra resource communicated to application. • Dummy replaced by application process. • Priorities used to favour interactive processes • Cluster wide process control.

More Related