Millennium Overview and Status

1. Millennium Overview and Status David Culler and Jim Demmel Computer Science Division culler,demmel@cs.berkeley.edu http://www.millennium.berkeley.edu

2. MM Proposed Cluster of Clusters Business SIMS BMRC Chemistry C.S. E.E. Gigabit Ethernet Astro NERSC M.E. Physics N.E. Math IEOR Transport Economy C. E. MSME Millennium

3. Physical Connectivity Millennium

4. Associated Commitments • Intel provides 6 M$ in equipment • Sun provides all Solaris x86 software • Microsoft provides all NT software • Campus provides staff to support the core infrastructure, networking • CS will try to raise funds for the network and campus cluster infrastructure • Departments provide 1/2 of system admin for own side and 20 K$ for group cluster • racks, network, software, ... Millennium

5. Where are we? • Evolving Vision => SimMillennium • Hardware deployment • Software availability • Cluster Environment • Grants • Networking Millennium

6. The Vision • To work, think, and study in a computationally rich environment with deep information stores and powerful services • test ideas through simulation • explore and investigate data and information • share, manipulate, and interact through natural actions • Organized in a manner consistent with the University setting Millennium

7. SimMillennium Project Goals • Enable major advances in Computational Science and Engineering • Simulation, Modeling, and Information Processing becoming ubiquitous • Explore novel design techniques for large, complex systems • Fundamental Computer Science problems ahead are problems of scale • Develop fundamentally better ways of assimilating and interactingwith large volumes of information • and with each other • Explore emerging technologies • networking, OS, devices Millennium

8. Components of the Effort • Community • Cluster-based Resources • Connectivity • User Interaction • Computational Economics Millennium

9. Electrical Engineering Transportation Mechanical Engineering Computer Science Astronomy Civil Engineering Physics Math SIMS Component 0: Community • An inter-disciplinary community with common interests and shared view of the future • strong momentum in computational science and engineering • Members of 17 campus units and NERSC in Intel Millennium • Need and commitment required for participation • Key subset represented in SimMillennium proposal Strong Progress with Pacific Earthquake Engineering Research Center New Thrust: NSF Science and Technology Center for the Study of Turbulence in Geophysical and Astrophysical Flows Millennium

10. Dept. SMP Group Cluster of SMPs NT 3D Desktop Component 1: Resources (Millennium) • An environment with vast cluster-based computing power and storage (CLUMPS) behind a personal 3D desktop Campus Cluster Millennium

11. Resource Component Support • Computers via IntelTechnology 2000 grant • 200 NT desktops • 16 department 4-way SMPs • 8 5x4 Group Clusters, • 1 ~100x4 Campus Cluster • PPro => Pentium II => Merced • Additional storage via IBM SUR grant • 0.5 TB this year => 4 TB • NT tools via Microsoft grant • Solaris x86 tools via SMCC grant • Campus provides Technical staff • Research provides the prog. and system support 200 Gflop/s 150 GB memory 8 TB disk Millennium

12. NSF investment: Cluster Network • Transforms large collection of individual resources into a powerful system • can be focus on a problem • High Bandwidth • scales with the number of processors (Gb/s per proc) • Low Latency • Low Overhead • Low Cost • Simple and Flexible • Almost no errors • Low Risk • Today: Myrinet Millennium

13. Cluster Research Agenda • Applications grow into resources • huge range of needs • require Algorithmic Innovation, Prog. Tools, & Performance • Dealing Deep Memory Hierarchy • New numerical algorithms on CLUMPs • New compiler techniques for parallel object language • Fast Multi-protocol Communication • Global system at large scale • Unix vs. NT, single system image vs. objects • Exciting technology turnover • VIA, SANs, Gigabit Ethernet Millennium

14. Component 2: Connectivity • Create a richly interconnected pool of resources owned by members of the community • Enable transportation of huge data sets and computation • Enable remote visualization and collaboration • Enable extensive sharing of resources • Expand networking technology CS Cluster EE Cluster CE Cluster ME Cluster Astro/Phys Cluster xport Cluster BIO Cluster Econ/Math Cluster Campus Cluster Millennium

15. NSF Investment: Inter-cluster network • Gigabit Ethernet connecting group clusters and campus cluster • Bay Networks provides 70% discount • Campus provides fiber plant, maintenance, and staff Millennium

16. Inter-Cluster Research Agenda • Vastly expands the scope of systems challenge • integrate well-connected resources according application needs, rather than physical packaging • resource allocation, management, and administration • Network bandwidth matches display BW • Protocols and run-time sys. for visualization, media transport, interaction, and collaboration. • Community can share non-trivial resources while preserving sense of ownership • Bandwidth translates into efficiency of exchange • Data can be anywhere • Important networking technology in its own right. • Emulate networks on Internet Scale! Millennium

17. Component 3: User Interaction • High-quality 3D graphics emerging on cost-effective platforms • desktops and dedicated cluster nodes • NERSC team provides modern scientific visualization support • Gigabit network allows this to be remote. • New displays create “workbench” environment where large volumes of information can be viewed and manipulated. • Trackers and Haptic interfaces greatly enhance degrees of user input • 3D capture Millennium

18. NSF Investment: UI Technology • Two Projection Table • large field of view in horizontal (or vertical) orientation • Phantom Haptic Interface • 3D force feedback • Motion Tracker • untethered position • 3D Shutter Glasses • low cost visualization Millennium

19. User Interaction Research Agenda • Expand access to 3D visualization • Explore any data anywhere • Ease development • Develop lab-bench metaphor for Viz • two hands, physical icons • Fast prototyping and exchange through Informal Interfaces • sketching • Dealing with large volumes of information • lenses, brushing and linking • 3D collaboration and interaction Millennium

20. Component 4: Computational Economy • How is this vast, integrated pool of resources managed? • Traditional system approach: empower global OS to provide “optimal” allocation to blind applications • predefined metric, tuned to fixed workload • ignores the inherent adaptation of demand • Computer Center • charge => director-to-user feedback according to cost • Economic view: decentralized allocation according to perceived value • pricing => user-to-user feedback • compatible niches,sense of control, cooperation • idea has been around, why now? Millennium

21. Research Agenda • Natural fit to academic structure • members want control over own resources, and each has varying needs that far exceed dedicated resources • incentive for maintaining resources up to par • Address partial or delayed information, component failure, and user satisfaction from the start • Framework for elevating design from resources to services • Rich body of theory, little empirical validation • experts in several parts of the community • New paradigm for algorithms & perf. Analysis • Complex, large-scale systems Millennium

22. Basic Approach • Desktop an active agent conducting automated negotiation for resources • Servers provide resources to highest bidders • monitor usage and enforce limits within remote execution environment • placement based on economic advantage • Higher level system functions are self-supporting • resource availability, brokering, directories • Useful applications packaged as services • may charge more than resources cost Millennium

23. NSF Investment: Staff Support • Provide enabling technology and let it evolve • monitoring, enforcement • exchange • negotiation tools • Integrate it into users enviroment • Tools and measurements to determine effectiveness Millennium

24. Integrated Research Agenda • Advance the State of Computational Science and Engineering • immerse a community in a computationally rich environment with the right tools: algorithms, programming & system support • Path to exploiting novel techniques and technology • Explore design techniques for robust large-scale distributed systems • economic (or ecologic) approach • Explore new ways of interacting with information • large paste-ups, two hands, sketching, 3D collaboration • Investigate new technology • SMP nodes, gigabit Ethernet, SANs, VIA • NT, dCOM, Java beans, directory services • workbench displays, 3D icons, haptics, position sensors Millennium

25. Perspective • Highly leveraged investment in a large scale infrastructure for studying problems of scale • Deep commitment across the campus • Sense of ownership and participation • Rich research agenda Millennium

26. Nuts and Bolts Millennium

27. Current Environment • All projects have servers and several desktops • last few in current shipment • millennium.berkeley.edu domain established • solaris/x86 shared servers with 60GB disks • www.millennium.berkeley.edu, MM.millennium.berkeley.edu • imap, ... • solaris/x86 sww served from CS servers • NT domain server with 60 GB disks • MMNT.millennium.berkeley.edu • exchange, file, ... • First cluster ready for use • NSF SimMillennium grant to cover network • Two of three Millennium staff hired Millennium

28. Hardware Deployment (per Q2_98) Millennium

29. Basic Software Tools • Standard Unix Tools /usr/sww/bin • 484 packages • GNU Tools • gcc, g++ (version 2.8.1), gdb, g... • Sun ProWorks • workshop “development environment” • C, C++, F77 • Debugger • Sun Performance Library /usr/sww/lib • Sun Math Library • NT dialtone Millennium

30. Cluster Developments • 4 x 4 PentiumPro Cluster ready • Full NOW environment • Glunix, MPI, Active Messages, Split-C • titanium protoype • Transfering to Cory to debug deployment • CS / Civil Eng. Consortium has been shaking it out • petc graph partitioner, finite element classes, … • ScaLapack (???) • Have parts for three 8x2 Pentium II clusters • AstroPhysics • Soda Solaris x86 to be shipped out • Soda NT cluster Millennium

31. Grants • NSF CISE Research Infrastructure grant • cluster networking • inter-cluster networking (campus mgmt) • computational economy staff support • visualization • devoted to CS research with disciplinary applns • Bay Networks 70% discount up to 5 M$ • gigabit networks • IBM SUR disk towers • IBM CS extension to PDA access • NERSC/LBL - DOE2000 Initiative • => NSF Science and Tech. Centers in process Millennium

32. Networking • Campus using Millennium to drive planning • CNS is working closely with us. • Testing gigabit ethernet • Campbell, Evans, Davis moving forward • CS will be able to cover the intracluster network (Myrinet) and gigabit switch in the group clusters • frees up some of the committed resources Millennium

33. Going forward • Roll out group clusters over next few quarters • Roll out gigabit interconnect • Build up the cluster programming tools • Intel Merced pushed out • campus cluster delayed • utilize NOW as an alternative • combine with NPACI infrastructure • Exploring cluster technology • VIA • Synfinity, ServerNet, Gigabit Ethernet • Exploring NT Millennium

34. Gigabit Ethernet A New Tier to Millennium Wireless Infrastructure Future Devices Cell Phones PDAs Millennium