Xports: Network-Based Macro-Molecular Structure Determination

1. Xports: Network-Based Macro-Molecular Structure Determination R. Bramley, D. F. McMullen, J. Huffman Indiana University - Bloomington I. Foster, G. von Laszewski, M. Westbrook Argonne National Laboratory E. Westbrook, Lawrence Berkeley Laboratory

2. Overview Application and goals X-ray crystallography; remote instrument control and collaboration Software architecture Explore use of DoE Common Component Architecture framework Extensions to non-compute components, network awareness Explore use of CoG Kits (commodity Grid components) Network requirements and research End-to-end reservation QoS requirements Capability for dynamic network access, control

3. Application and Goals • X-Ray crystallography to determine molecular structure • Better resolution from high-energy, high-flux beam lines at LBL and ANL • Current procedure: • Ship crystal sample to LBL • Lab tech mounts sample, runs predetermined scan procedure • Data collected, written to DAT, shipped FedEx • If sample is flawed/mounted incorrectly, multi-hour run wasted

4. Application and Goals • Desired procedure • Scientist sends sample to LBL • With scientist’s advice via teleconferencing, technician mounts sample • Scientist sends initial set of scan commands • Preliminary frames are filtered via local/remote components, get sent to scientist who manipulates images to check • mounting is proper • sample is still good • whether sample is subject to twinning • Scientist sends new scan parameters - or terminates beam-line session • At any time, scientist can allow remote colleagues to view partial results and confer

5. Application Beam Source Goniostat Local Store Sample Crystal CCD Array CCD Data Collector Local DSP, Buffering Structure Analysis User Ctl Scan Set Collaborative Visualization

6. Application Beam Source Status Image Frames to Struct. Analysis Goniostat Local Store Sample Crystal Ctl Signals CCD Array CCD Data Collector Local DSP, Buffering Image Frames Structure Analysis User Ctl Scan Set Image Frames from Local St User Ctl Munged Frames Collaborative Visualization

7. Xports Software Architecture Common component architecture (CCA) mechanisms Utilize CoG Kits: Commodity Grid Kits CCA defines minimal feature set for HPC software “components” - but is not a component framework Provides for interoperability between components developed by different sites using different frameworks. CORBA, DCOM, Java Beans are related systems but don’t handle HPC and parallel components. Fail on “minimum surface area” test. Basic ideas: CCAServices objects, and CCAPorts

8. Component System Advantages • Dynamic composition: run-time construction • Run-time customization via configuration parameters • Components may have state, multiple interfaces. • Can move location without recompiling/relinking • “Multicast” capability • Supports composition of components in different languages and software systems

9. Comm Channel Source Target InPort OutPort CCA Ports • Output ports provide methods to other components; example might be SendData( ) • Input ports are remote references to output port methods • Only required method: PortInfo( ) giving port type and name as strings. Type matching via the strings. • Fast “direct connect” for components in same address space; has uses port = connected provides port

10. CCA Services • Each component is provided with “CCA.services” object • Used by component to tell framework about the ports it has • Access to services provided by framework for components • Services methods include (SIDL interface descriptions) • array<PortInfo> getProvidesPorts( ); • array<PortInfo> getUsesPorts( ); • Port getPort(in string name); • registerUsesPort(in PortInfo name_and_type); • addProvidesPort(in ProvidesPort inPort, in PortInfo name); • releasePort(in string name); • ComponentID getComponentID( );

11. Indiana CCA Toolkit • An implementation of CCA with • Directory Service: for browsing remote directories containing data about components and their specifications. • Registry Service: for browsing remote directories with data about running instances of components. • Creation Service: instantiates another component. • Connection Service: connects port of one component to another. • Event Channel: publication and filtering of events. Push-based publisher channel/subscriber model with type filtering. • Services above are themselves “pseudocomponents” - CCA components with some internal method access

12. CCAT Extensions for XPorts • CCA model based on computational components • Components consisting of parallel programs • Mixture of shared memory, distributed memory components • Xports must extend to • Remote instrument access • Remote control • Large-scale, multi-source data management • Network capabilities access

13. Xports Components • Components match block model shown earlier • Handle difficulty of proprietary HW/SW of beam-line equipment • Control port takes user-specified scan parameters • Frame output port multiplexed to avoid loss of data from network problems • May require caching/sequencing component • Data analysis components dynamically instantiated where most useful - on site, or at user location. • User interface components: dynamically allow scripting, GUI, or auto-default as needed.

14. Xports Components • Network capabilities and CCA components • Network aware computational/data store components • Dynamic negotiation of protocols and capabilities, on a per-message basis if necessary • Ability to restore lost links, re-establish state • Interfaces to DiffServ, other systems (?) • Ability to vary QoS as analysis proceeds • Network status/statistics interface components • Give user information about overall system • Co-allocation of multiple QoS streams with differing service levels (more later)

15. Network requirements and research • Bulk data transfer • QoS requirements • Network related component enhancements

16. Bulk data transfer • Rates: 12MB/sec for several hours • Up to 16 MB/frame uncompressed (for 4k x 4k detector) • 9-900 frames • One frame per 1.5 sec • Other apps (e.g., CMT) have one frame per 0.1 sec, but total bandwidth reservation time reduced by same factor • Frames “demultiplexed” for reliability to • local buffer/cache • local or remote computational component • mass storage component • (beam-line time rarer, more expensive than compute/network time).

17. Xport QoS Requirements • Multiple streams: • CCD frame transfer: initial frames of critical importance • 2 to 3 video cameras: operator, crystal, crystal alignment, user(s) • Audio • Real-time visualization and shared virtual spaces (collaborative CAVE data presentation and analysis) • Co-allocation of multiple service levels • Video and audio • Bulk data transfer to/from HPSS and local cache • Data transfer between components during execution of distributed computations

18. Beamline Network app. services End-user services CCD Detector On-line data reduction View CCD images crystal monitoring camera Image analysis Work with beamline tech. conference camera Goniostat control and status, temperature monochromator Scan setup and control Structure solution and refinement Structure visualization QoS required Mass storage for CCD output QoS not required Structure refinement control Offload data Xports QoS Between Functional Blocks

19. Network related component enhancements • Determine and verify routing and link characteristics • Components may connect temporarily, drop and reconnect • During its lifetime a component should be aware of routing or link capacity changes • Determine link characteristics (throughput, latency(?), optimum protocol and socket settings) • DiffServ QoS support at component level • Determine classes of service and permission to request them intra- and inter- domain (?!) • Set a class of service for a flow • Determine ad hoc performance of a “QoS” flow

20. Xport Summary • Application has full range of network requirements • Good candidate for CCA component architecture extended to non-compute services • Network requirements range from premium to default • Require simultaneous management of multiple QoS streams with differing priorities. • Component-to-component CCA port connection will have QoS requirements that vary dynamically