1 / 33

High End Computing with K42

C O M P U T A T I O N A L R E S E A R C H D I V I S I O N. High End Computing with K42. FastOS PI Meeting June 9, 2005.

felix
Download Presentation

High End Computing with K42

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N High End Computing with K42 FastOS PI Meeting June 9, 2005

  2. The HECRTF and FastOS reports enumerate unmet needs in the area of Operating Systems for HEC, including • Availability of Research Frameworks • Support for Architectural Innovation • Performance Visibility • Adaptability to Application Requirements • This project uses the K42 Operating System to address these needs

  3. What is K42? • K42 is a GPLed research O/S at IBM • Framework for PERCS (DARPA HPCS) work • O/S research and architectural innovation • API and ABI compatible w/ Linux on PPC64 • Runs most Linux kernel modules (fs, etc) • Many OS services implemented in user-space • Object oriented • Every virtual or physical instance is an object • Every class may have multiple implementations • Implementations can be hot-swapped, per-instance • Very modular for easy addition/modification • Has extensive performance/tracing capabilities • Design/implementation is very SMP-scalable

  4. Who? • IBM • The group most experienced with K42 and responsible for its continued development • The group performing O/S research for PERCS • Has contributed K42-developed ideas to Linux • Linux Trace Toolkit (LTT) • Object-based reverse mapping (memory mgmt) • Read-Copy-Update (RCU)

  5. Who? • LBNL • DOE applications and application scientists • Especially via BIPS and SciDAC PERC project • Scalable Systems Software SciDAC project • BLCR (system-initiated checkpointing for Linux) • Access to OpenMPI team (former LAM/MPI) • UPC and Titanium teams (GASNet runtime) • Linux kernel experience • Including M-VIA and BLCR

  6. Who? • University of New Mexico • Experience with implementing and porting light-weight kernels • SUNMOS, Puma, Cougar and Catamount • Experience with the development of the Portals API • Experience in configurable/adaptable systems software • X-kernel, Scout and Cactus

  7. Who? • University of Toronto • A prominent member of the existing K42 research community • Origin of Tornado, the direct predecessor to K42 • Key preliminary work in arbiter-object technology for hot swap and dynamic adaptation

  8. What are we doing? • Work divides into three major areas: • Framework for OS/Runtime research for HEC applications • Dynamic Adaptation • Architecture of a parallel operating system

  9. Framework for OS/Runtime research for HEC • Make K42 usable as a platform • To perform basic O/S and Runtime research of importance to HEC • To develop/run/debug/tune HEC applications

  10. Framework (1) Issue: K42 is hard to build/install Approach: Create a distribution for a dual-boot K42/Linux system (src and bin) Issue: K42 runs only on PPC64 Approach: Port to AMD64 (maybe EM64T?) Issue: K42 lacks a full HEC environment Approach: Build/port the required environment (SSS OSCAR) • Numeric libraries (OSCAR) • Batch system (Scalable System Software suite) • Programming models (MPI UPC Titanium CAF)

  11. Dynamic Adaptation • Utilize K42’s design • To expose performance information below the app-O/S interface • To allow static and dynamic specialization of O/S and runtime services

  12. Dynamic Adaptation (1) • Issue: O/Ses and runtimes are performance-opaque • Approach: Extend what K42 has • K42 already has extensive performance/tracing capabilities • Expose K42’s object structure • Use arbiter objects for per-object collection of h/w counter data • Develop graphical tools to connect performance data to OS/runtime objects

  13. Dynamic Adaptation (2) • Issue: What to adapt? • Approach: Study HEC Apps • Use performance tools to identify OS/Runtime objects which are bottlenecks and in what situations • Investigate what alternative implementations offer better performance in those situations • Add these implementations to K42

  14. Dynamic Adaptation (3) • Issue: When to adapt? • Approach 1: user-directed customization • Approach 2: compiler-directed customization • Approach 3: Runtime adaptation using arbiter objects to monitor and adapt to changing conditions

  15. Dynamic Adaptation (4) • Issue: How to adapt? • Approach: Use hot-swapping of object implementations in K42 • Allows one to replace implementations, per-instance, without the need to block the application

  16. Dynamic Adaptation (5) • Issue: Need a small set of applications, representative of HEC today and in the future • Approach: LBNL has identified a set of applications that we feel are a good starting set. • separate presentation if time allows

  17. Building a Parallel O/S • K42’s design principles yield excellent scaling on SMPs, with minimal UP impact • Apply these principles to parallel runtime services • Integrate these services with the O/S

  18. Build a Parallel O/S (1) • Issue: K42 lacks a native RPC mechanism • Approach: • Adapt the best design features of • Protected Procedure Calls (inter-address space) • Active Messages (inter-node) • Simple but powerful AM-style mechanism for parallel runtime services • Reduce to protected procedure call in the intra-node case

  19. Build a Parallel O/S (2) • Issue: AM has no “name service” • Approach: Design and prototype a simple mechanism for locating required services • Services may be load-balanced • Services may migrate/fail-over

  20. Build a Parallel O/S (3) • Issue: Asynchronous events are common in a parallel environment • Approach: Reusable event service • K42 is already an event-driven system • Design and prototype a distributed event service • Simple Publish/Subscribe API?

  21. Build a Parallel O/S (4) • Issue: Parallel job management • Spawn, signal, ps • Approach: Extended Process Spaces • Not the same as SSI, more like PAGs • Process id tuple: (ProcSpace, ID) • Each parallel job is a ProcSpace • A process can see those ProcSpaces to which it is “attached” (creator, member or observer)

  22. Build a Parallel O/S (5) • Issue: Just TCP/IP sockets in K42 • Approach: • Characterize the application impact of s/w communication overheads • Investigate App/kernel/NIC APIs • Offload of communication processing • Application-specific customization • Implement O/S support for other communication abstractions • Active Messages • RDMA

  23. Applications Performance Evaluation • Can K42 be a production HEC environment?

  24. Applications Performance Evaluation • Head-to-head Linux-vs-K42 comparisons • Many comparisons already possible • Port of HEC environment will allow more complete comparison • Port to Opteron will allow head-to-head comparison to Catamount • Interesting: Apps programming in K42 is as easy as Linux, but can K42 perform as well as Catamount on HEC?

  25. And then what? • What can/should one do with the resulting framework?

  26. Possible Follow-on Work (1) • The research framework will allow/ease many potentially interesting research areas • Some cut from our proposal • Some are part of other FastOS projects • Others are new • Presented on following slides in no particular order

  27. Possible Follow-on Work (2) • Filesystem work via native RPC mechanisms and RDMA networking • Cluster software management • K42 hot-swapping can allow full OS/Runtime upgrade of a live system with no downtime(like TELCO equipment) • Co-scheduling • Reduce O/S-induced load-imbalance (“noise”) that perturbs collective operations (especially barriers)

  28. Possible Follow-on Work (3) • Scheduling and resource management for multi-threaded and multi-core processors • Example: page coloring for cache partitioning • Virtualization, checkpoint/restart and migration • Interposing objects makes virtualization trivial • Use of RCU makes most (all?) quiescing unnecessary when checkpointing • High performance network drivers for K42 • InfiniBand, Quadrics QSNetII, MyriNet

  29. In Conclusion…

  30. Summary (1) • Produce a K42 platform that: • Is easy to install and use for O/S and runtime research in HEC • Includes nearly all of the HEC environment a user is expecting • Helps users to track/understand performance within the O/S and runtime • Accepts static customization hints from users and/or compilers

  31. Summary (2) • Produce a K42 platform that: • Will dynamically identify performance bottlenecks in the O/S and runtime and dynamically switch to more appropriate object implementations • Includes custom HEC-appropriate application/kernel/network APIs • Includes an infrastructure for building of parallel operating environments • Includes a scalable mechanism for parallel job control

  32. Summary (3) • Work with DOE SC applications • To determine HEC-appropriate implementations/policies/APIs • To improve applications performance • Evaluate the performance of K42 as a production HEC platform • Head-to-head vs. Linux • Head-to-head vs. Catamount

More Related