SPRACE KyaTera / UltraLight Proposal

1. SPRACE KyaTera / UltraLight Proposal VI D0SAR Workshop São Paulo, Brazil September 16, 2005 Rogério L. Iope Universidade de Sao Paulo (Grad. Research Assistant for SPRACE)

2. e-Science: Data Gathering, Analysis, Simulation, Collaboration • Scientific discoveries increasingly driven by data collection • Computationally intensive analyses • Massive data collections • Data distributed across networks of varying capability • Internationally distributed collaborations • New approaches to enquiry based on • Deep analysis of huge quantities of data • Interdisciplinary collaboration • Large-scale simulation • Smart instrumentation • e-Science methods no longer optional but now vital to scientific competitiveness

3. e-Science: Driving Global Cyberinfrastructure e-Science is about providing significantly enhanced research infrastructure by utilizing distributed resources such as computers, storage devices, scientific instruments, and experts using information technology CMS TOTEM ALICE : HI ATLAS LHCb: B-physics

4. e-Science: Driving Global Cyberinfrastructure • The enormous speedups of computers and networks have enabled simulations of far more complex systems and phenomena, as well as visualizing the results from many perspectives • Advanced computing no longer restricted to a few research groups in a few fields, but pervades scientific and engineering research • New data-intensive applications are driving seemingly insatiable demand for more bandwidth • Groups collaborate across institutions and time zones, sharing data, complementary expertise, ideas, and access to special facilities without traveling • Optical Networks are key to this vision • Massive scalable bandwidth • Protocol and bit-rate independence • The ability to launch and scale new services on demand • Photonic Networking: the way to cope with IP traffic explosion

5. Overview of the UltraLight Project • UltraLight is • A collaboration of experimental physicists, computer scientists, and network engineers from BNL, Caltech, CERN, UF, FIU, FNAL, Internet2, UM, MIT, SLAC... • …to provide the network advances required to enable petabyte-scale analysis of globally distributed data • An application-driven network R&D program to explore the integration of cutting-edge network technology with the Grid computing and data infrastructure of HEP/Astronomy • A non-standard core network with dynamic and varying bandwidth interconnecting globally distributed nodes • An NSF-funded 4 year program to deliver a new, high-performance, network-integrated infrastructure • Two primary, synergistic activities (source: S. McKee) • Network “Backbone”: Perform R&D / engineering • Application “Driver”: System Services R&D / engineering

6. Overview of the UltraLight Project • Main goals • Engineer and operate a trans- and intercontinental optical network testbed • Promote the network as an actively managed component • Develop and deploy prototype global services which broaden existing Grid computing systems • Enable physics analysis and discoveries by integrating and testing UltraLight in Grid-based physics production and analysis systems currently under development in ATLAS and CMS • A three-phased plan • Phase 1 (12 months): Implementation of network, equipment and initial services • Phase 2 (18 Months): Integration and footprint expansion • Phase 3 (18 Months): Transition to production (LHC physics + eVLBI astronomy)

7. Overview of the UltraLight Project • Project Management Team • PI: Harvey Newman (Caltech) • Project Coordinator: Rick Cavanaugh (UF) • Network Coordinator: Shawn McKee (UM) • Applications Coordinator: Frank van Lingen (Caltech) • Education&Outreach Coordinator: Laird Kramer (FIU) • Physics Analysis User Community Coordinator: Dimitri Bourilkov (UF) • “Wan-In-Lab”: Steven Low (Caltech) • Project Coordination activities • Regularly scheduled phone and video meetings • Periodic face-to-face focus workshops (semi-annually or quarterly) • Persistent VRVS room for collaboration • Mail-lists • Web-page portal (first prototype)

8. Overview of the UltraLight Project • Some important UltraLight R&D goals • Basic Network Services • Data transport protocols • MPLS/QoS Services and Planning • Optical Path Management Plans • Optical Testbed • Optical Exchange Point • Network Monitoring • Network Management and AAA • Disk-to-disk data transfers • Wan-In-Lab / DISUN • HEP Application Services

10. Connectivity Diagram for UltraLight Source: http://ultralight.caltech.edu/

11. The KyaTera Project • A cooperative program proposed by FAPESP, as part of the TIDIA Program • Main goal: • The establishment of an optical fiber network infrastructure connecting laboratories for research, development and demonstrations of technologies for advanced Internet applications • Network infrastructure based upon the concept of dark fibers reaching directly to the research laboratories (FTTLab) • The name KyaTera comes from • Kya (“net” in Tupi-Guarani) • Tera (greek teras = monster)

12. The KyaTera Project • Composed by a dark fiber mesh spread over several cities among the State of São Paulo • A large, geographically distributed laboratory facility for experimental tests of new network concepts and optical devices, new network protocols and services • A platform for developing and deploying new high performance e-Science applications • A stable, high performance network always co-exists with the experimental network • new developments in the last do not interfere with the operation of the first

13. The KyaTera Project • Research subjects for KyaTera organized in 3 layers • Physical Layer • optical communications, new developments on fiber infrastructure • Transport Layer • protocols, interface standards, maanagement, monitoring, interoperability, etc, in optical networks • Applications Layer • automation and computer control of scientific instruments, Grid applications, HDTV, etc

15. WDM Fundamentals • Wavelength-Division Multiplexing – WDM • An approach that can exploit the huge bandwidth available on fiber optic links • Can manyfold the capacity of existing networks by transmitting many channels simultaneously on a single fiber optic line • The optical transmition spectrum is carved up into a number of non-overlapping wavelength (or frequency) bands • Multiple WDM channels from different end-users may be multiplexed on the same fiber • Each wavelength supports a single communication channel operating at peak electronic speed • By allowing multiple WDM channels to coexist on a single fiber, one can tap into the huge fiber bandwidth • A more cost-effective alternative compared to laying more fibers

16. WDM - Parallelism on Optical Networking (WDM) Source: Steve Wallach, Chiaro Networks “Lambdas” Parallel lambdas will drive this decade the way parallel processors drove the 1990s !

17. WDM Fundamentals • WDM building blocks • Light sources (laser diodes) and detectors (photodetectors, filters) • Optical fibers (single-mode, multi-mode) • Multiplexers and Demultiplexers • Optical Add/Drop Multiplexers • Optical amplifiers (e.g. EDFA) • Photonic cross-connect switches • Transponders

18. WDM Fundamentals • A wavelength-routed optical WDM network consists of a photonic switching fabric comprising active optical switches connected by fiber links forming any arbitrary physical topology • Each node equipped with a set of transmitters and receivers (which may be “wavelength tunable”) • The basic mechanism of communication in such a network is a lightpath • Lightpath: an all-optical communication channel (a path) between 2 nodes (it can span more than one fiber link!) • The intermediate nodes in this fiber path route the lightpath in the optical domain using their active optical (photonic) switches • The end-nodes of the lightpath access the signal with transmitters and receivers that are tuned to the wavelength on which the lightpath operates

19. WDM Fundamentals • Photonic switches & protocols like GMPLS are key elements to address new goals, and implement a multi-tiered and scalable IP/Optical network

20. WDM Fundamentals • In wavelength-routed WDM networks, a control mechanism is needed to set up and take down the optical connections (lightpaths) • A successful data transfer event between 2 nodes has three phases • Connection establishment • Data transfer • Connection release • During first phase, a few control signaling packets are exchanged between network resources, aiming to establish a lightpath with an assigned wavelength • If it succeeds, a lightpath is established, and data transfer occur through this circuit from source to destination • When the transfer is completed, control packets are again exchanged between the nodes, and the resources are released and made ready to be assigned for another connection

21. WDM Fundamentals • A challenging networking problem is that, given a set of lightpaths that need to be established on the network, and given a constraint on the number of wavelengths, • determine the routes over which these lightpaths should be set up • determine the wavelengths that should be assigned to them so that the maximum number of lightpaths may be established • If any switching/router node is also equipped with a wavelength-converter facility, then lightpaths can be established using diferent wavelengths on their routes from origin to destination • This problem is referred to as the RWA problem

22. WDM Systems: General layout . . . . . . . . . . . . . . . . . . Transmissor DWDM MUX DWDM DEMUX DWDM Transponder DWDM 1 fiber fiber Core Router Core Router 1 OXC EDFA EDFA GBIC n GBIC 1 GBIC n GBIC 1 n n Transponder CWDM Transponder CWDM cn c1 cn c1 DEMUX CWDM MUX CWDM MUX CWDM DEMUX CWDM Border Router Border Router OADM 1 OADM 1 Border Router Border Router c1 c1 CWDM CWDM cn cn OADM n OADM n (Source: M. Stanton - GIGA Project)

23. WDM Systems: R-OADM Conceptual Architecture drop 1 1 3 Software Controlled Selectors (Pass-through/Add/Block) Pass Splitter West Pass-Through Wavelengths DWDM Signal Add Pass Software Controlled DEMUX Add block block drop Add Wavelengths Drop Wavelengths Transponder Module Network Element Network Element Network Element Network Element 3 Transponder Module Drop Wavelengths Add Wavelengths Software Controlled DEMUX drop block drop block Add DWDM Signal Pass Pass-Through Wavelengths East Add Splitter Pass Software Controlled Selectors (Pass-through/Add/Block)

24. The KyaTera testbed: Reference Architecture Ethernet Aggregation Switch IP Router 10 GbE <-> l MUX/DEMUX & R-OADM MUX/DEMUX & R-OADM Photonic Switch Ethernet Aggregation Switch Ethernet Aggregation Switch MUX/DEMUX & R-OADM MUX/DEMUX & R-OADM MUX/DEMUX & R-OADM MUX/DEMUX & R-OADM IP Router 10 GbE <-> l IP Router 10 GbE <-> l Photonic Switch Photonic Switch

25. The KyaTera testbed (example of a proposed solution)

26. Enabling e-Science: The KyaTera / UltraLight Proposal • Network support: a critical aspect of Grid-enabled environments • Commodity Internet is based on a best-effort delivery model, a vehicle excessively slow and unreliable for the huge masses of data being generated in emerging e-Science applications • Deployment of Grids on wide-area scales is being severely restricted

27. Enabling e-Science: The KyaTera / UltraLight Proposal • Optical networing: a promising solution to these limitations • Emerging lightpaths technologies are becoming more and more popular in the Grid community • They can include the network resources as an integral Grid component, controlled by Grid schedulers in the same way as computing elements and storage resources • The challenge: • A new management technology is needed to allow end-users to acquire network resources on demand, control end-to-end interconnections between peers (lightpaths), and share unused bandwidth in a flexible and collaborative way

28. The KyaTera / UltraLight Proposal • Our project proposal: • To work on the problem of monitoring, managing, and optimizing the use of the networking resources present in next-generation user-controlled optical networks in real time • To work in close partnership with the UltraLight Project and the KyaTera Project • To use the optical networking infrastructure that is being made available by the KyaTera Project • The KyaTera network insfrastructure, enhanced by an intelligent optical control plane middleware, will provide the basement for the deployment of the Grid-enabled Analysis Environment Service Architecture (GAE), a project being developed at Caltech and University of Florida, coordinated by Prof. Harvey Newman

29. The KyaTera / UltraLight Proposal • Research will be done on provisioning end-to-end survivable optical connections in the testbed, as in a Grid environment, with an innovative use of the GMPLS control plane • (this will be accomplished in a close partnership with the OptiNet lab experts) (Drawing and text courtesy of Gustavo Pavani – OptiNet / UNICAMP)

30. Project Planning: Milestones and Timeframe • Milestones • Provisioning of end-to-end optical connections between pairs of nodes • Provisioning multilayer protocols and intelligent monitoring software agents, and research on RWA algorithms • Deployment of routing/switching and control protocols to locate suitable lightpaths and schedule the networking resources • Deployment of Grid Analysis Environment • Job submissions and data transfers between sites over the distributed computing infrastructure looking for failures, malfunctioning and bottlenecks

31. Project Planning: Milestones and Timeframe I II III IV V