1 / 28

POTENTIAL HEP APPLICATIONS OF A NEW HIGH PERFORMANCE NETWORKING TECHNOLOGY

POTENTIAL HEP APPLICATIONS OF A NEW HIGH PERFORMANCE NETWORKING TECHNOLOGY. 1 A SHORT Introduction to The GSN Specifications HIPPI 6400 PH ST Scheduled Transfer SCSI over ST 2 Applications in High Performance Computing 3 Applications in High Energy Physics.

cruz
Download Presentation

POTENTIAL HEP APPLICATIONS OF A NEW HIGH PERFORMANCE NETWORKING TECHNOLOGY

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. POTENTIAL HEP APPLICATIONS OF A NEW HIGH PERFORMANCE NETWORKING TECHNOLOGY 1 A SHORT Introduction to The GSN Specifications HIPPI 6400 PH ST Scheduled Transfer SCSI over ST 2 Applications in High Performance Computing 3 Applications in High Energy Physics Arie Van Praag & Ben Segal CERN1211 Geneva 23 Switzerland E-mail a.van.praag@cern.ch http://www.cern.ch/HSI/gsn

  2. Development Project: HIPPI-6400 Commercial name GSN ( Gigabyte System Network ) DATA 16 (8) Control 4 (2) Source Destination Frame 1 (1) Clock 2 (1) DATA 16 (8) Control 4 (2) Destination Source Frame 1 (1) Clock 2 (1) HIPPI-6400 PH Connections: Full Duplex with Copper or Fibre Cable Data Lines 16+4 8+2 Frame Signal 1 1 Clock 2 1 Principles: Data Transfer in Small Micropackets 4 Multiplexed channels called Virtual channel VC0 -VC3 To avoid Congestion Operating System Bypass with Scheduled Transfer To avoid OS Latency Full 48 bit Addressing Mode for Switching and Routing following IEEE 802.nn To convert easily to other network standards

  3. HIPPI-6400 PH HIPPI-6400 Standards Document: Description: Status: HIPPI-6400 PH: Physical Layer 6400 Mbit/s ANSI NCITS 323-1998 or 800 MByte/s network ISO ISO/IEC 11518-10 HIPPI-6400 SC Switch Standard NCITS 324-1999 HIPPI-6400 OP Optical Connection NCITS Submitted ST Scheduled Transfer NCITS submitted SCSI over ST SCSI commands over ST NCITS Standard SCSI T10 R-00 Sub-standards: GSN & ST conversions to Fibre-Channel, HIPPI, Gigabit Ethernet, Sonet, ATM

  4. VCO Buffer VCO Buffer VC1 Buffer VC1 Buffer VC2 Buffer VC2 Buffer VC3 Buffer VC3 Buffer Virtual Channels HIPPI-6400 PH SOURCE DESTINATION CABLE OR FIBER Output Buffer Output Buffer Virtual Channels VC0 = Maximum Size: Messages of 68 Data Micropackets ( 2176 bytes ) + header micropackets Admin Micropackets VC1 and VC2 = Maximum Size: Messages of 4100 Data Micropackets ( 128 KBytes ) + header micropackets VC3 = Maximum Size: Messages of 4 GBytes Data Micropackets + header micropackets

  5. VCO Buffer VCO Buffer VC1 Buffer VC1 Buffer VC2 Buffer VC2 Buffer VC3 Buffer VC3 Buffer Flow Control HIPPI-6400 PH Credits are consumed as a packet moves from the VCn Buffer to the Output Buffer. SOURCE DESTINATION CABLE OR FIBER Input Buffer Output Buffer TSEQ ACK(seq) RSEQ ACKs are generated independent of the VC number, and sent to the Source in the reverse direction micropacket control information. credit ( VCn,amount) Credits are generated on a VC basis when data exits from the VC buffer, and sent to the Source in reverse direction microcontrol information.

  6. DB00 DB01 d00.7 d00.7 d00.0 d00.0 DB30 DB31 d00.7 d00.7 d00.0 d00.0 c63 c55 c47 c15 c00 c07 1 Header Information c63-00 2 1 st 32 Bytes of Message data c63-00 3 2 nd 32 Bytes of Message data c63-00 N Last Bytes of Message data c63-00 Micropacket & Message Format HIPPI-6400 PH 32 DATA BYTES ( 256 Bits ) 64 CONTROL BITS MESSAGE FORMAT

  7. C3 C2 C1 C0 D15 D14 D13 D12 D11 D10 D9 D8 D7 D6 D5 D4 D3 D2 D1 D0 bit a b c d 12 13 14 15 08 09 10 11 04 05 06 07 00 01 02 03 07.4 07.5 07.6 07.7 07.0 07.1 07.2 07.3 06.4 06.5 06.6 06.7 06.0 06.1 06.2 06.3 05.4 05.5 05.6 05.7 05.0 05.1 05.2 05.3 04.4 04.5 04.6 04.7 04.0 04.1 04.2 04.3 03.4 03.5 03.6 03.7 03.0 03.1 03.2 03.3 02.4 02.5 02.6 02.7 02.0 02.1 02.2 02.3 01.4 01.5 01.6 01.7 01.0 01.1 01.2 01.3 00.4 00.5 00.6 00.7 00.0 00.1 00.2 00.3 a b c d 28 29 30 31 24 25 26 27 20 21 22 23 16 17 18 19 15.4 15.5 15.6 15.7 15.0 15.1 15.2 15.3 14.4 14.5 14.6 14.7 14.0 14.1 14.2 14.3 13.4 13.5 13.6 13.7 13.0 13.1 13.2 13.3 12.4 12.5 12.6 12.7 12.0 12.1 12.2 12.3 11.4 11.5 11.6 11.7 11.0 11.1 11.2 11.3 10.4 10.5 10.6 10.7 10.0 10.1 10.2 10.3 09.4 09.5 09.6 09.7 09.0 09.1 09.2 09.3 08.4 08.5 08.6 08.7 08.0 08.1 08.2 08.3 a b c d 60 61 62 63 56 57 58 59 52 53 54 55 48 49 50 51 07.4 07.5 07.6 07.7 07.0 07.1 07.2 07.3 06.4 06.5 06.6 06.7 06.0 06.1 06.2 06.3 05.4 05.5 05.6 05.7 05.0 05.1 05.2 05.3 04.4 04.5 04.6 04.7 04.0 04.1 04.2 04.3 03.4 03.5 03.6 03.7 03.0 03.1 03.2 03.3 02.4 02.5 02.6 02.7 02.0 02.1 02.2 02.3 01.4 01.5 01.6 01.7 01.0 01.1 01.2 01.3 00.4 00.5 00.6 00.7 00.0 00.1 00.2 00.3 c15 c00 CR VCR E T TYPE VC c31 c16 TSEQ RSEQ c47 c32 ECRC c63 c48 LCRC Micropacket Structure HIPPI-6400 PH FIELD BITS CONTROL FONCTION VC 2 VC Selector TYPE 4 Information Type T (AIL) 1 Last Micropacket E (ERROR) 1 ERROR VCR 2 Virtual Channel for credit addition CR 6 Number of Credits RSEQ 8 ACK. Sequence number TSEQ 8 Transm. Sequence number ECRC 16 End to End Checksum LCRC 16 Link Level Checksum

  8. Control Micro Packets HIPPI-6400 PH Reset / Initialize Credit Only Null Header Data Admin Data Byte Contents 0 0 0 32 Byte Header 32 Bytes Data Admin Information VC 0 0 0 any any TYPE ( hex ) 2,3,4,5 7 A 9 8 F Tail 1 0 0 1 ERROR 0 0 0 =1 if Erro =1 if Error =1 if Error TSEQ xFF xFF Increments Increments Increments Increments RSEQ 1 ACK ACK ACK ACK ACK VCR 0 0 any any any any CR 0 0 any any any any LCRC single single single single single single ECRC single single single accumulating accumulating accumulating Request on VC1 Request on VC2 =1 on last Micropacket =1 on last Micropacket

  9. DATA IN c63 c62 c61 c60 c59 c58 c57 c56 c55 c54 c53 c52 c51 c50 c49 c48 LCRC covers all of the Data Bytes and Control Bits ( except for itself). X + X + X + 1 16 12 5 Error Correction HIPPI-6400 PH LINK CRC ( LCRC ) DATA IN c47 c46 c45 c44 c43 c42 c41 c40 c39 c38 c37 c36 c35 c34 c33 c32 End to End CRC ( ECRC ) • ECRC includes only Micropacket Data Bytes including the Header Micropacket Data Bytes. • X + X + X + X + 1 16 12 3 XOR Function

  10. CLOCK CLOCK 2 2 FRAME FRAME CNTL CNTL 4 4 DATA 16 DATA 16 CLOCK CLOCK 2 2 FRAME FRAME CNTL 4 CNTL 4 DATA 16 DATA 16 HIPPI 6400 END1 CONNECTOR HIPPI 6400 END 2 CONNECTOR T T T T T T T T W X Y Z W X Y Z W X Y Z W X Y Z W X Y Z W X Y Z W X Y Z W X Z Y Cable and Timing Sequences HIPPI-6400 PH 50 METER CONNECTORS BERG MICROPAX 100 CLOCK _2 CLOCK Dnn or Cn For 16 bit Macropacket d00.0 - d00.3 d08.0 - d08.3 d16.0 - d16.3 d24.0 - d24.3 For 8 bit Macropacket d00.0 - d00.3 d00.4 - d00.7 d08.0 - d08.3 d08.4 - d08.7 d16.0 - d160.3 d16.4 - d16.7 d24.0 - d24.3 d24.4 - d24.7 FRAME 16 bit 40 nsec FRAME 8 bit 40 nsec

  11. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fiber Connection HIPPI-6400 OPT 2 CABLES 12 Fibers Each Connectors: MT Connectors: MT Transmitter Launch Power -6 to -12 dBm Wavelength: 850 nm Receiver Sensitivity: -18 dBm Berr = 10 -12 Distance: Max 1 Km Compliant to: CDRH 1040.1 Class1 IEC 825-1 Class 3a Long Distance Connection Single Mode Fibers 2 Parallel Fibers Wavelength 1300 NM Distance 10 Km A working Group is just formed

  12. D D S S D D S S D S D S S D S D HIPPI 6400 SOURCE S HIPPI 800 DESTINATION D S D S D HIPPI 6400 HIPPI 800 Switches HIPPI-6400 SC GSN - 64 SWITCH GSN - 64 NODE GSN - 64 NODE Full Crossbar Non Blocking Switch with Translation boxes for HIPPI-800 and other Data Transporting Media 1 x 1 x HIPPI 6400 DESTINATION D Translation Function BRIDGE 4 x 8 X HIPPI 800 SOURCE S HIPPI 800 Fibre Channel Gigabit Ethernet SONET OC48c HIPPI - 800 NODE

  13. Specifies small control messages to pre-arrange Data movement Virtual Connection Set-up and Tear Down. Determine the number of operations the other side can accept Determine the Buffer Size of the other side Exchange Key, Port, Identifiers, Buffer Size Values Determine Maximum Transmission Unit to avoid overrun of the buffers at the Receiving End Acknowledge Partial Transfers Easy Transfers over Different type of High Performance Networks If used with prestored Tables an extension of a Scatter Gather System Scheduled Transfer ST

  14. Scheduled Transfer ST remote Endlocal end Port Port Key Key Max. Slots Max. Slots Bufsize Bufsize Max. STU Size Max STU Size Max. Block Size Max. Block Size Out_of_order cap. Ethertype Remote Slots local Slots local Sync # Op_time Max_retry remote-id1 local-id1 remote-id2 local-id2 remote-idj local-idj local-Port local-Key remote-Port Virtual Connection Descriptors Selection and Validation Criteria Transfer Descriptor Transfer Descriptor Buffers Buffer Descriptor Table Block Descriptor Buff 0 Bufx 1 Bufx 2 Bufx n Address 0 Address 1 Address 2 Address n . . . . . . . .

  15. Bufx n n+1 ST S T WRITE Responder Request Memory Region(RMR) Memory Region Available (MRA) Mx=m length=l offset=x Bufx=n Responder Buffer Put STU Mx=m Get/FetchOp Mx=p, length=k ioffset=y, roffset=z iBufx=o, rBufx=n+1 Initiator Buffer STU Mx=p FetchOp Complete

  16. ST Storage Commands Commands are sent in ST’s RTS or RTR optional payload: Byte Content 0 type = storage command 1 length = 32 2-3 target number 4-31 SCSI Fibre Channel(FCP) Command payload, bytes 0-27

  17. Cannot use payload for SCSI status (can be >30 bytes) Use ST’s Persistent Memory Region feature: SCSI - ST Target RTS I-id=3 opt=cmd Target Buffer CTS STU(data) STU(data) STU(data) Status PMR 0 1 2 3 4 5 Status Put (STU)

  18. S T The Protocol Picture NFS FTP SCSI Software ... UDP TCP Other protocols ... IP ST DE/MUX General Delivery Hardware Hardware ST Hardware Media

  19. Products as of January 2000 INTERFACES: Silicon Graphics Origin Series Available PCI Interface 64/66 Compaq Alpha, SUN 1 Q 2000 SWITCHES: ODS - Essential 32 X 32 Available ODS - Essential 8 X 8 Available Genroco 8 X 8 Available PMR 8 X 8 Available BRIDGES: ODS-Essential Translation Function HIPPI-800 Available Genroco Storage Bridge Fibre Channel Available Genroco Network Bridge HIPPI Available Fibre Channel Available Gigabit Ethernet Available OC48c 3 Q 2000 CABLES: Berg Copper cables and Connectors Available COMPONENTS for OPTICAL CONNECTIONS: Infineon Paroli AC Modules and Fibres Available Gore Noptical Modules and Fibres 1 Q 2000

  20. LHC FREEZING SELECTION LHC OPERATIONAL 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 01 02 03 04 05 Standards & Popularity( 1995 ) Ethernet T base 100 Gigabit Ethernet Fibre Channel ATM HIPPI HIPPI-Serial Gigabyte System Network GSN PCI S-LINK

  21. SERVERS GSN GSN Network Bridge F.C. SCSI/FC HIPPI SCSI/ST /HIPPI ETHER TYPE NETWORK Fibre Channel HIPPI Gigabit Ethernet GSN GSN GSN SWITCHES GSN http://www.cern.ch/HSI/hippi/hug/demo99/Demo99.html SCSI/ST/GSN GSN Storage Bridge SCSI/FC F.C. SCSI/ST/GigE SCSI/ST/ETH FC TAPES FC DISK ARRAYS

  22. Networking Networking FC DISK ARAYS FC DISK ARAYS FC TAPES FC TAPES Cluster File Systems Data Storage Networks GFS Minnesota University NT McIntosh Solaris etc. LINUX IRIX Central File Meta Data on a central workstation Proprietary Software, e.g. DataDirect CDNA Mercury SANergy, etc.. Symmetrical System with Distributed file system Meta Data and Locks.

  23. FIFO GSN to S-LINK ?? FPGA Version with 6 S-LINK CANNELS Added Bandwidth 100 Mbyte/s per Interface Total Added bandwidth 800 Mbyte/s per Bridge FIFO FIFO 16 bit 66 MHz 132 MB/s PROCESSOR BUS FIFO FIFO FIFO FIFO reset Register SET-UP Control etc Glue Logic Synchronization etc. PROCESSOR FPGA FPGA FPGA 64 bit 40 MHz 320 MB/s 64 bit 40 MHz 320 MB/s TO BRIDGE RJ 12 RS232 FROM BRIDGE

  24. Event Building with a Switch 1 0-100 TByte/s. DETECTOR DATA 100- 1000 Bytes/s. VMEbus Read Out Buffers ( ROB ) CONNECTIONS 768 (4) S-Link or 1152 (6) S-Link or 192 HIPPI-800 10 - 100 MBytes/s 24 GSN Bridges BRIDGE BRIDGE 24 GSN Connections 32 X 32 GSN - 64 Switch Fabric 8 GSN Connections to Workstation Farm FC DISK ARRAYS To Central Data Storage or Data Analyzes Long Distance Connection

  25. Giga Router Giga Router SW Glabs 8X8 Switch 01 Gigalabs 16X16 NA48 Experiment 1 From Detector HIPPI raw data CODING & CONVERTING and RECODING & RECONVERTING 2 Data Distribution to Processor farm HIPPI 3 processor output in FDDI 4 Gigarouter converts data to FDDI over HIPPI 5 FDDI over HIPPI to computer centre 8 Km L.W. 6 Gigarouter converts to FDDI for processing 7 Via Gigarouter back to HIPPI for Storage 8 HIPPI to SCSI conversion for Storage STK Tape Units

  26. A FAT ST PIPE GSN = FC / ST / SCSI Physics Data from Experiment ? ? ? No Protocol or Format Change from DETECTOR to Storage & Analysis Long Distance Transfer Connection ( Wavelength Multiplexing ? or Sonet or ?? ) ? ? ? GSN / Gigabit Ethernet GSN / HIPPI Bridge GSN / FC Central Storage in the Computer Center FC

  27. 10 Km Physics Data Transportfor LHC LHC Experiments: Each experiment Transmits up to 100 MBytes/s How to get this data to the computer center ? OC 48c does 250 MByte/s Atlas Alice LHCB CMS

  28. GSN against Ethernet Who makes a bet ? Sorry Compaq that I mis-used your picture

More Related