1 / 36

TO-DAY

TO-DAY. 1 High Performance Networking as sign of its time. A Historical Overview of Hardware and Protocols Yesterdays High Performance Networks Ultranet, HIPPI, Fibre Channel, Myrinet, Gigabit Ethernet GSN ( the first 10 Gbit/s network and secure )

bparedes
Download Presentation

TO-DAY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TO-DAY • 1 High Performance Networking as sign of its time. • A Historical Overview of Hardware and Protocols • Yesterdays High Performance Networks • Ultranet, HIPPI, Fibre Channel, Myrinet, Gigabit Ethernet • GSN ( the first 10 Gbit/s network and secure ) • Physical Layer, Error Correction, ST Protocol, SCSI-ST • Infiniband ( the imitating 2.5 – 30 Gbit/s interconnect ) • Physical Layer, Protocols, Network Management • 5 SONET and some facts about DWDM, 10 Gigabit Ethernet, Physical Layers, Coupling to the WAN Arie Van Praag CERN IT/ADC 1211 Geneva 23 Switzerland E-mail a.van.praag@cern.ch

  2. S O N E T 1972 The first Electro optical specifications and synchronization protocol are published by an industry consortium called :SynchronousOptical NETwork or SONET.Bandwidth 3.125 Mbit/s 1985SONET was adapted by the ANSI standardsbody T1 X1 as Synchronous Fibre Optics Network for Digital communications. 1986 CCITT ( now ITU ) joined the movement. Implemented Optical Level Europe Electrical Line Rate Payload Overhead H Equivalent ITU Level (Mbps) (Mbps) (Mbps) 1989 OC - 1 --- STS - 1 51.840 50.112 1.728 --- 1992 OC - 3 SDH1 STS - 3 155.520 150.336 5.184 STM- 1 1995 OC - 12 SDH4 STS - 12 622.080 601.344 20.736 STM- 4 1999 OC - 48 SDH16 STS - 48 2488.320 2405.376 82.944 STM- 16 2002 OC-192 SDH48 STS-192 9953.280 9621.504 331.776 STM- 64 OC-768 SDH192 STS-768 39613.120 39486.016 1327.064 STM-256

  3. Standards SONET is a multitude of standards: It defines the Physical transfer and the protocols; ATM, POS, It defines hardware Interfaces at different levels; Utopia, XSBI, XAUI, etc. It defines the Bandwidth running up with a factor 4; SONET/SDH It defines Wavelength for DWDM. It defines switching layers as proposed by ISO; Level 2 VLAN DA SA Level 3 IP Addresses Level 4 Sessions Level 5-7 URLs, H.323 information etc. It defines routing and switching strategies such that different manufacturers stay compatible without limiting design freedom.

  4. Message Data Packet Packet Packet CONNECT BYTES 3 3 2 0 - 65.527 0 – 47 4 4 LLC OUI PID Data Payload PADING UU/CPI + Length FCS BYTES 48 48 48 48 48 48 48 48 48 48 48 48 ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM ATM Bits 4 8 12 3 1 VPI VCI PTI CLP Data GFC SONET - ATM GFC = Generic Flow Control VPI = Virtual Path Identifier VCI = Virtual Channel Identifier PTI = Payload Type Identifier CLP = Cell Loss Priority

  5. Message Data User Frame User Frame User Frame CONNECT 1 1 1 2 POS 0 - 1500 X85 0 - 1600 4 1 POH ADDR CNT ProtocolSAPI Data Payload FCS Flag TOH SOH Data Payload Flag HDLC Frame Header SONET - POS Data Space in POS mode up to 32 K – 65 K Will this be good place to put IP and Ethernet data ?may belet’s see it with 10GE

  6. DWDM DWDM Dense Wavelength Division Multiplexing

  7. Graded Index Reflex Plate Or Holographic Diffraction Plate Old Technology: but still much used By heating the laser the reflecting resonance cavity is expanded which changes the frequency. Problem: Stability Arrayed Wavelength Gratins = ¼  Selection Platealso called “Phase Array” or “Phaser Plate” Modern Technology: Using MEMS technology the mirror is moved by electro static forces, and as such expands the lasing cavity.The laser can be tuned with a DC voltage. What is Wavelength Multiplexing Sending Multiple colors ( wavelength ) over a single fiber How is it done ( simplified )

  8. DWDM Input Wave-Guides Field Lens with ¼  interference Single Color outputs Field Lens How the AWG works

  9. Fibers are still a medium with a lot of critical parameters, such that ongoing research is an important factor to limit theme to a minimum by choosing materials and doping, and optimise production methods. Some Physical Problems Are: Optical Return Loss:Change of material and interconnecting surfaces both reflect small parts of light, As different material may have a different index ( transfer speed ) a frequency shift is not excluded that may provoke cross-talk. Chromatic Dispersion: The group delay per Unit Wavelength is not equal and gives different travel speeds to the channels. Chromatic Dispersion accumulates with distance. Polarization Mode Dispersion: Simplified; Horizontal and Vertical polarization do not travel at the same speed which occasions pulse broadening. It is an unstable stochastic process that is due to all kind of stress on the fiber. It gets more sensitive at higher bit rates . Second Order Polarization Mode Dispersion: A dispersion of the polarization that is occasioned by chromatic Dispersion, only if there is Polarization Mode Dispersion. It is mainly a problem with very long distances. Brillouin Backscattering: Backwards reflexions from connectors and strong bends create in the laser an acoustic wave that travels in the fiber and occasions optical disturbances. The Doppler effect is source of frequency shift and increases the risk of cross talk or cross modulation. Solutions: Much progress is made on connectors and on the fibers itself, better glass ( plastic ) material, sophisticated doping and cladding, and stress limiting production methods have limited most of this problems to a minimum. Problems with DWDM Fibers

  10. DWDM - Equipment Capacity is up to 2048 and more Channels. Common in the Communication Industry is 1024 Each color with its modulation logic and its stabilization electronics is a plug in unit Take 25 Units in a crate, and 10 crates in a rack = 4 racks Transmitters and 4 racks Receivers Is DWDM technology a good solution for CERN To Expensive To Complicated To Much Maintenance To Much Place CERN can solve its internal future data transport problems with bandwidth extension to 10 Gbit/s and higher using one stream per Fiber. For external connections it is up to the Service Provider but 10GE in POS mode or 10GE in Native mode seems a good solution to communicate physics data to outside institutes and to handle GRID Data distribution.

  11. DWDM and some Simple Economic Aspects Up to 1990 traffic was mainly some e-mail and remote maintenance. 1990 The web was born incrementing traffic. Fiber connections ware view, single stream and at best OC12/SDH4 at + 622 Mbit/s. 1995 – 2000 Enormous investments ware made in larger cables (1000 fibers and more). Cable laying ships ware build, and cables pulled all calculated at Single stream traffic. 1998 and later: bandwidth goes up OC48/SDH16 at 2.5 Gbit/s and even to OC192/SDH48 with 10 Gbit/s and DWDM and multiplies the capacity by 1000 such that a 100 fiber cable now moves 1 000 000 streams at 10 Gbit/s. Overcapacity let prices fall and communication companies go bankrupt on their enormous investments.

  12. CALIENT MIRRORS ARE POSITIONED BY ANALOG VOLTAGES THE LIGHT BEAM HAS TO BE TARGETED AT A SINGLE MODE FIBER OF A VIEW MICRONS. Conclusion: EVEN IF INTEGRATED THIS WAY OF SWITCHING IS VERY SENSITIVE TO THE STEERING VOLTAGE AND TO TEMPERATURE Optical Switching LUCENT Can it be done more Reliable ?

  13. OMM Switching Speed + 12 msec. More Stable Optical Routing Switches If there is a On/Off Mechanism we are back to a Binary Function.

  14. 128 wave-guides Solid State Optical Switch In Gallium Arsenid (GaAs) optical properties can be influenced by an electric field. The electrical influenced refraction index delays the passing light. A voltage gradient in multiple waveguides turns the wavefront ( ¼ λ interference ) and with it the direction of the output beam. The result is an “Optical Phase Array” or “Beam Deflector” Can be produced with Semiconductor manufacturing technology, no moving parts. Switching speed 20 to 30 nsec. Fast enough to do IP routing of 10 Gbit/s and 40 Gbit/s networks. Format independence gives high scalability. Large switches possible ( 64 X 64 ) demonstrated. * According to CHIARO Networks

  15. 10GE

  16. 802.3 Ballot Study GroupFormed 80.3ae Formed Sponsor Ballot 1999 2000 2001 2002 Final Draft 80.3ae Standard First Draft 10 Gigabit Ethernet The 10 Gigabit Ethernet started in 1999 in the IEEE 802.3 working group Standard IEEE 802.3ae is accepted 17 June 2002 June 2002 First Commercial hardware: may be 4 Q 2002 2003

  17. 10 Gigabit Media Independent Interface XGMII 2X 32bit data + 2X 5 bit Control PCS 64B/66B PCS 64B/66B PCS 8B/10B Physical Coding Sublayer WIS WAN Interface Sublayer WAN compatible framing PMA PMA PMA Physical Medium Attachment Retime, SerDes, CDR PMD PMD PMD Physical Medium Dependent Optical Transceiver Connecting medium Medium Dependent Interface MDI MDI MDI MEDIUM10GBase-R MEDIUM10GBase-W MEDIUM10Gbase-X Fiber type etc 10 GE Physical Payload: 10 Gbit/s Transfer: Full Duplex Media: Fiber only ( for the moment at least ) Upcoding 8B/10B or 64B/66B

  18. OC – 48/SDH16 2488.320 OC – 192/SDH48 9953.280 OC – 768SDH/192 39813.12 Wavelength: 1275.7 nm, 1300.2 nm, 1324.7 nm, 1349.2 nm. 3.125 Gbit/s / channel Standard Interface foreseen For 4X INFINIBAND PMD Type of Fiber Target DistanceOptical TRC Meters 850 nm serial Multi Mode 65 1310 nm CWDM Multi Mode 300 1310 nm CWDM Single Mode 10 000 1310 serial Single Mode 10 000 1550 serial Single mode 40 000 10GE Optical & Fiber Type

  19. Problems with CWDM or WDWM Chromatic Dispersion travel speed in a fiber is not equal for all colors Wavelength Drift Transmitter Data drifts out of the filter slot and can not be received Wavelength Drift The receiver filter drifts and does not select the correct wavelength Wavelength Drift wavelength comes in the region where couples to other channels ( cross-talk ) Chromatic Dispersion Source of limited distance covered, and stops use of fiber amplifiers Conclusion 10 GE will only be popular if a cheap and reliable single fiber single wavelength solution is available

  20. Bandwidth: 12.5 Gbit/s Protocol: TCP/IP follows IEEE 802.3 full 48 bit addressing Frame size: 1500 Bytes Ethernet 10 Gbit/s = 830 000 frames of 1500 bytes, or 1.2 s / frame. = 2 X 830 000 Interrupts/s for transmission and for reception. Without an Operating System Bypass it will be extremely difficult OC-768 1 000 000 100 000 10 000 1000 1000 10 OC-192 GP MIPS Trend OC-48 MIPS OC-12 OC-3 MIPS Needed for Communication Applications Technology 1 0.7 0.5 0.35 0.25 0.18 0.13 0.1 0.07 0.05 0.03 0.02 10 GE Networking

  21. 6 5 4 3 2 1 10 10 10 10 10 10 10 Gbit/s 1024 10 Gbit/s 160 10 Gbit/s 32 10 Gbit/s 16 4 10 Gbit/s 8 10 Gbit/s 4 10 Gbit/s 2 I/0 Rates = Optical Wavelength Capacity Important Threshold OC-192c 10-GE 1.7 Gbit/s OC-48c OC-48c System Capacity (Mbit/s) 565 Mbit/s GigE OC-12c Fast Ethernet 135 Mbit/s OC-3c Optical DWDM Capacity Ethernet Internet Backbone T3 Ethernet T1 Year 1985 1990 1995 2000 2005 OC-768c 40-GE

  22. LINK/SWITCH Fabric queuing NP-1 TOPmodify engines memory TOPresolve engines memory External memory TOPsearch engines memory TOPparse engines memory MAC LINK/SWITCH Fabric EZ-Chip a Solution TOP = Task Operating Processor Existing Technology for 10 Gbit networks Ready for next generation for 40 Gbit Networks 32 TOP search engines, 64 processors total Onboard Memory up to 5 MByte 256 to 512 bit wide with 200 MHz clock Processes all 7 network layers: Level 2 VLAN DA SA Level 3 IP Addresses Level 4 Sessions Level 5-7 URLs, H.323 information etc. Capacity 8 X GigE or 1 X 10GE or 1 X OC192 Future versions Ready for 8X 10GE or 1 X 40 GE or OC768

  23. Does the I/O have the bandwidth Data given by PCI-SIG

  24. PPP Prot. Field 2 PPP PADDING Flag 8 Address 8 Control 8 PPP IP Packet FCS 16 / 32 Flag 8 IP to SONET Header Conversion SONET/SDH OC48c Conversion Hardware PPP IP Frames HDLC GSN DES. ADDR. 6 MAC SRC-ADDR. 6 IP Packet M LENGTH 4 Processor DSAD 2 SSAD 2 SNAP ctl x03 1 org x00 3 ETHERTYPE 2 40 IP Packet PAYLOAD Compliant to RFC 2615 IP - Internet Protocol IPv4: 020b

  25. 17280 Bytes 576 Bytes Section SONET/SDH Payload Envelope (SPE) FRAME FRAME FRAME TRANSPORT OVERHEAD 9 rows 9 rows O V E R H E A D FIXED HEADER 63 Columns SONET/SDH Payload Up to 64 full Ethernet Frames Line 16640 Bytes 16704 Bytes POS Packing ATM is not efficient anymore at bandwidth over 1 GHz, and even less in safe mode. The previous High Speed Way to move Ethernet packets over SONET/SDH used Byte Stuffing at 2.5 MByte/s for OC48/SDH16 in POS mode. It is also used in OC192/SDH48 An extension on the 10GE standard to be accepted by ANSI and UCI will transfer 10GE Packets directly over OC192/SDH48. The difference in bandwidth will be covered with by inserting IDLE’s at regular distances in the Ethernet data stream.

  26. PPP IP IP PPP Data Padding CONNECT PPP IP Data IP Padding PPP CONNECT Ethernet Ethernet IP IP Data Data IP IP Padding Padding Eth. Eth. CONNECT CONNECT IP IP Data Data IP IP CONNECT CONNECT SONET Ethernet IP Data IP Padding Eth. CONNECT Ethernet Frame IP packet 10GE direct on OC192c SONET PPP packet Ethernet Frame IP packet

  27. Up to 10:1 price advantage in upfront costs Up to 5:1 advantage in bandwidth provisioning expenses Up to 5:1 advantage in annual maintenance Provides bandwidth on demand without costly truck rolls Economics of 10 GE Ethernetoffers a superior price/performance and TCO over alternative technologies (SONET/ATM) SONET Ethernet

  28. Products Products start to arrive on the market now, mostly “Prove of Concept” commercially available are view. Silicon: Infineon, Sierra, Agilenta, Broadband and some small development houses are all advertising NIC’s and SERDES circuits, sometimes for 10 GHz/s and for 40 GHz/s. No general interfaces NIC’s are around and will not be before PCI-X2 is available Switches and routers are announced. The most common are line concentrators with 10 X GE to 1X 10GE 10GE Alliance makes a large effort by organizing compatibility workshops with all manufacturers concerned. It also does a general marketing job to show products and its compatibility efforts ( plug Fest ) on trade shows.

  29. 10 GigE Examples Examples of Future Applications by Cisco and the 10 Gigabit Ethernet Alliance

  30. VHDL using standard twisted telephone wire is developing rapidly to faster and higher reliability with new technology standards ( moving from “Discreet Multitone Modulation” to “Quadrature Amplitude Modulation” ) New VHDL technology will such allow a bandwidth good enough for 10/100 Base T Ethernet The Metropolitan Network will move to double arbitrated ring structures ( 802.17 Resilient Packet Ring ) And 802.17 introduces “Virtual Concatenation” for efficient packing of non corresponding frame sizes Example: SONET STS 1 = 51.84 Mb/s, STS 3 = 155.52 Mb/s. Standard: With RPR Ethernet 100 Base T needs STS 3 and wastes 55.52 Mb/s Ethernet 100 Base T can use STS1 and wastes 3.6 Mb/s RESULT: Your future Internet connection will be cheaper for the Service provider, and such for the user, to use the same old phone line that brings 10/100 Base T Ethernet to the end user. And the Last Mile

  31. Industrial Partners • Datamat (Italy) • IBM-UK (UK) • CS-SI (France) Grid and the Network As Grid started the interconnections planned used OC12 at 622 Mbit/s and OC3 at 155 Mbit/s. with a single exception at OC48 at 2.5 Gbit/s To move the large data files from LHC Experiments • Research and Academic Institutes • CESNET (Czech Republic) • Commissariat à l'énergie atomique (CEA) – France • Computer and Automation Research Institute,  Hungarian Academy of Sciences (MTA SZTAKI) • Consiglio Nazionale delle Ricerche (Italy) • Helsinki Institute of Physics – Finland • Institut de Fisica d'Altes Energies (IFAE) - Spain • Istituto Trentino di Cultura (IRST) – Italy • Konrad-Zuse-Zentrum für Informationstechnik Berlin - Germany • Royal Netherlands Meteorological Institute (KNMI) • Ruprecht-Karls-Universität Heidelberg - Germany • Stichting Academisch Rekencentrum Amsterdam (SARA) – Netherlands • Swedish Research Council - Sweden

  32. Trans-Atlantic Connections 10 GE and the GRID We can now foresee that in the near Future Grid communications will move to 10 GE Ethernet. But this depends if there are sufficient good quality Single Mode Fiber available And if Service Providers are ready To Handle Gigabit Ethernet.

  33. Ethernet T base 100 Gigabit Ethernet Fibre Channel ATM ( as computer interconnect ) HIPPI HIPPI-Serial 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 01 02 03 04 05 Standards & Popularity( made in 1995 and extended 2000 ) GSN ( Gigabyte System Network ) 10 Gigabyte Ethernet Infiniband PCI / PCI-X / PCI-X2

  34. 10 GE Conclusions 10GE is Ethernet, the name everybody knows. Also the manager that decides the IT Budget. Two standards for data transfer: 4 parallel streams using WWDM and a single stream. WWDM is difficult to stabilize, and can not easily be coupled to telecommunication channels. The Single Stream version connects the Local Network, to the Metropolitan Network and to the Wide Area Network within one protocol environment. Using PPP encapsulation in POS mode every standard communication channel can be used immediately, including DWDM channels 10GE Silicon with 110 nm and 90 nm technology is able to handle the bandwidth and will come at reasonable prices. The optical parts will stay high price, even if moved to VCSEL’s The small 1500 Byte frames will stay a handicap in local data handling and RDMA will only solve half of the problem. TOE engines are necessary, but some transfer latency stays We will see 10GE at CERN, as backbone for the network, as backbone in the computer-center and as a data-link between LHC experiments and the central computing facilities.

  35. Guide to WDM Technology,A. Girard, et all, EXFO Electro Engineering Inc. Quebec City, Canada, 2000, ISBN 1-55342-000-4 Design Trade-offs for Arrayed Waveguide Grating DWDM MUX/DMUX Jane Lam, Ph.D. and Liang Zhao, Ph.D. Lamwhitepaper.pdf Digital MEMS switch for planar photonic crossconnects, L. Fan, et all, OMM, Inc San Diego, 1999, OCIS codes 060.1810 http://www.omminc.com/technology/whitepapers.html. And much more interesting documentation on this site. Optical Phased Array Technology for High-Speed Switching, http://www.chiaro.com/pdf/CHI100_OPA_1.pdfwith more interesting white papers under http://www.chiaro.com/proof_points/index.jsp 10GEA 10 Gigabit Ethernet Alliance, http://www.10gea.org/ with many white papers and links to other cites. IEEE P802.3ae 10Gb/s Ethernet Task Force http://grouper.ieee.org/groups/802/3/ae/index.htmlThis is the IEEE working group that made the 802.3ae standard Strategic Directions Moving the Decimal Point: An introduction to 10 Gigabit Ethernet. B Tolley, Cisco, Jan.5, 2001.http://www.cisco.com/warp/public/cc/techno/lnty/etty/ggetty/tech/10gig_wp.htm The jump to 40GB Ethernet, P. Judge, Oct. 5, 2002 ZDNET (UK) Ethernet becomes king of the networking world, L. E. Frenzel, Electronic Design, December 9, 2002, p 45-52. References:

  36. END LASTPART Thank you for your attention during this long and not always easy material, where my hope is that you learned about 10 Gbit/s problems and highlights

More Related