360 likes | 431 Views
Network Power Users The Case for Jumbo Packets with WestGrid Examples. What are Jumbo Frames & What is 9k MTU?. 2 yrs. 0.5 yrs. 4 yrs. 9k DDS Project. 9k MTU Project. 9k APP Project. Core R&E Router Troubleshooting Large Xfer Measurements International Links
E N D
Network Power UsersThe Case forJumbo Packetswith WestGrid Examples
2 yrs 0.5 yrs 4 yrs 9k DDS Project 9k MTU Project 9k APP Project Core R&E Router Troubleshooting Large Xfer Measurements International Links Internet 2 Sponsored Physics Participants Manual Lightpath TRIUMF to CERN Test Few 9k Taps onto Core Main Tap SDSC Lightpath R&E Non-Routed… UCLP Build Dedicated 9k Links ZX GBICs/quad gigE/campus MM CANARIE NETERA BCNET 3 Universities Only 9k Link on Campus Physics & Biochem Participants HEPnet - WestGrid - TRIUMF 9k Node - HPC - 9k Node Viz HPC 9k Tests NFS HPC 9k Tests Lightpath R&E Clone-Tune 9k APP… Handoff Provide 9k Switch Biochem UVic - SFU - UofA Prelim 9k SOA Framework Drug Discovery System Flow Tune 9k SOA Framework Integrate 9k SOA Framework Demo 9k SOA-DDS Based Collab 9k SOA-DDS Software Dev Examples Complete Handoff … ;-) 9kMTU - APP- DDSProject Evolution…
9000 MTU 8192 MTU 7168 MTU 6144 MTU 5120 MTU 4096 MTU 3072 MTU 2048 MTU 512 MTU 9K MTU Project - Results
0.7 * Max Segment Size (MTU) • e2e throughput < • Round Trip Time (latency) sqrt[loss] • M. Mathis, et.al. How do Jumbo Packets Affect Bandwidth? • If TCP window size and network capacity are not rate limiting factors then (roughly): • Double the MSS, double the throughput • Effect of slow start? • Effect of irregular flow?
9kAPP– Approaching Application Performance • Need metrics for defining network impact on dependent applications • Best current example – MOS as indication of VoIP performance • Models of network dependence required • Applicable to QoS/SLA
9kAPP– User Experience • User expectations of applications • Examples: • Interaction with 3D models • Collaboration with multiple models/data/voice/video • Massive data set manipulation • Collaborative HPC simulations
9kAPP- General Models of Network Dependency • Near Real Time (nrt) • Congestion, Drop, MTU & Transit time sensitive • Transactional (tr) • Congestion, MTU & Drop sensitive • Data Transfer (dt) • Congestion, Drop, MTU & Transit time sensitive • Best Effort (be) • Not sensitive
9kAPP – Real-Time Applications • Examples – IPTV and Voice-over-IP • Requirement – support human interaction through highly subjective perceptive processes • Nature - asynchronous, constant, low-rate, non-TCP, streams • Dependencies • Highly sensitive to bursty loss • Sensitive to latency, particular in conversation context • Robust to jitter, up to some limit
9kAPP – Synchronous/Transactional • Examples - interactive collaborative systems; distributed file systems • Requirement - maintain some form of state at two or more remote locations; • Nature - intensive, bursty, synchronous traffic; varies from very small amounts of data to huge exchanges • Dependencies – requires high transfer rates, irregular; highly sensitive to latencies; intolerant of slow-start
9kAPP - Data Transfer • Example – FTP; data backup; emergency recovery • Requirement – transfer massive amounts of data, as quickly as possible • Nature - sustained one-way flows at maximum rates of transfer • Dependencies – sensitive to the characteristics of the end-host transmission protocols (i.e. TCP); high capacity; high impact on other flows
9kAPP - Best Effort • Examples - e-mail, Web browsing, and remote login • Requirement – sufficient resource to maintain minimal state or connection • Nature - largely stateless; low rates of data transfer required; not gated by human response • Dependencies – no critical requirements for network responsiveness
9kAPP - Choice of Application Performance • Key performance factor – jumbo packets • Previous 9k MTU testing applicable to simple data transfer use case • Make the case for a more demanding 9k application performance category • High performance: • Interactive/collaborative visualization • WestGrid visualization as exemplar • Distributed file systems • WestGrid Gridstore as exemplar
9kAPPProject Components • Three networks: Netera, CA*net4, BCNET
9kAPPProject Sites & Nodes • Three sites: UofA, SFU, UVic • Four nodes • TRIUMF • HEPnet • Vizserver • Gridstore • Why these? • Physics expertise • 9k lightpath ready • WestGrid applications • vizserver • gridstore
9kAPPProject - Define 9K paths • Two phases: Viz & Grid • Phase 1 Viz • TRIUMF • HEPnet • Vizserver • Phase 2 Grid • TRIUMF • HEPnet • Gridstore
9kAPPProject - Phase 1 - Viz • Vizserver session – optional compression • IRIX OS – render pipe • TCP: vary MTU 68 – 9000 at server • Measure performance vs. MTU • Vizclient session – wraps local openGL • Fully interactive local X session • refresh requests in render pipe • VMD: Visual Molecular Dynamics • Xserver & Graphics calls to render pipe • Collaborative VMD sessions
9kAPPProject - Network Build • 3 Lightpaths - UCLP - SONET • Each path STM-24c dropped on gigE • New quad gigE blade BCNET ONS • 2 new ZX GBICs • SM fibre run to SFU lit • SFU Campus Network • ZX GBIC installed in Enterasys ER16 • Phase 1 - Viz • MM fibre run direct to Cisco 6509 • Phase 2 - Grid • MM fibre run via Enterasys N7
9kAPPProject – Testing Procedure - pMTU Accumulated data from 9k MTU and 9k APP Projects
9kAPPProject – pMTU Issues • As MTU increases and increasing varies between hops • determine optimal pMTU before and changes during application use • locate problem hops with unusual behavior • Is larger effective lower layer pMTU actually better from the application perspective? • Are packets actually sized appropriately, by the packetization layer, given a larger pMTU? • What are some effects of larger pMTU under congestion conditions?
9kAPPProject - Phase 2 - Grid • Distributed file system • Possible candidates • CXFS, GPFS, NFSv4 • Possible clients/servers • SGI IRIX, IBM AIX, Linux • Preliminary model • NFSv4 on Linux & IBM AIX(gridstore) • Primary use cases • File sharing – massive data sets • Physics • Bioinformatics
9kAPPProject - Grid NFS Application • NFS session • AIX NFSv4 server • Linux NFSv4 clients • over TCP not UDP • NFS server – wraps local HPC file system • Fully interactive local file system session • Fast metadata updates for directory browsing
9kAPPProject – Grid Network & Testing • Network reconfigured with intermediate link • Viz performance rechecked • Gridstore VLAN setup • Fractional quad gigE • Test via NFSTest suite (opensource) • TCP: vary MTU 68 – 9000 at server • Probe network after MTU alteration • Measure performance vs. MTU • Linux NFSv4 to AIX NFSv4 • Probe while test in progress
9kAPPProject - Grid - NFS - Tuning • Objectives derived from WestGrid gridstore • Baseline performance – simple NFS client • Session • TCP - NFSv4 • NFStest suite • Time to complete vs. MTU • Individual test performance vs. MTU • Blocksize and other tuning considerations • NFS filesystem mount options • Block size = 8192 bytes • Fragmentation factors • Native filesystem block size
9kAPPProject • Bill Rutherford (Rutherford Research/RRX – Project Coordinator) • Loki Jorgenson (Apparent Networks/SFU – Project Coordinator) • Thomas Tam (CANARIE/CA*net4 – CANARIE/UCLP Coordinator) • Bryan Caron (TRIUMF/UofAlberta – TRIUMF/UCLP Coordinator) • Randy Sobie (HEPnet/UVic – HEPnet President/Grid Integration) • Brian Corrie (WestGrid/IRMACS/SFU - IRMACS Coordinator) • Rob Ballantyne (IRMACS/SFU - IRMACS Network Coordinator) • Martin Siegert (WestGrid/SFU – WestGrid/GridStore Coordinator) • Dave Bickle(HEPnet/UVic – HEPnet Coordinator/Grid Integration) • Ken Howard(Network Services/UVic – Network Coordinator) • Peter van Epp(Network Services/SFU – Network Coordinator)
9kDDSProject – Drug Discovery System • Based on 9k APP Project • Combined Physics, Grid, Bioinformatics • Joint development of network & software • Share network expertise • Help develop preliminary software • SOA approach • Collaborative viz • Distributed file systems • Instrument interfaces • Grid integration • Lightpath integration
Future – Performance Profiles by Application • APP Network Performance Profile • Build up statistical APP profiles • Use APP profiles to optimize context • Next Generation Router Design • Use APP Profiles • Allocate resources • Design microflow queues • Identify MTU issues • Dynamically configure path mechanics
End of Presentation Note: 9k APP Project Meeting in Room 1535 at 3:00
Outline • Short overview of previous 9K work • What is 9k? • 9k XXX Project snapshots? • Bandwidth value … example data? • How effect bandwidth? – equations? • Application performance – definition • User experience … limits? • Near Real-time (nrt) • Transactional (tr) • Bulk transfer (bt) • Best-effort (be) • Value of MOS… expand? to the VoIP industry • Project objectives • Isolate a single simple performance factor – packet size? • Identify prospective applications • Interactive collaborative visualization (nrt) • Distributed file system (tr + bt + nrt[metadata]) • Characterize application performance in context … • stack mechanics … grid integration?
Outline – cont. • Project components • Three sites: UofA, SFU, Uvic • Why these sites … only 9k available + RTT factor? • Three networks: BCNET, Netera, CA*net4 • WestGrid applications • Brian’s Westgrid vizserver… IRMACS • Martin’s Westgrid grid storage • Define 9K paths • Application profiling • Visualization server – phase 1 • Identify system • Define primary use case … why vmd … why collab? • Define network profile … describe nw build? • Identify testing procedure … pMTU tests … issues
Outline – cont. • Distributed file system – phase 2 • Identify candidates – CXFS, GPFS, NFSv3-4 • Identify possible clients/servers • Define primary use cases … physics …DDS file sharing • Define network profile • Identify testing procedure (NFStest) • Basic types of test … ? • Futures • 9k DDS … trend to separate purpose nw… key role of UCLP • Performance profiles by application … ??? • Ng rtr … mtu issues … app microflow queues? • Credits • Meeting Reminder
BCNET ANC 9k APP Project Meeting Agenda – April 26 3-4pm • Review of current status • Lightpaths • IRMACS • Gridstore • HEPnet • TRIUMF • Viz test • Network • Probes & schedule • Tests • Demos (incl special medicine collab demo UVic – UofA) • Gridstore test • Network • Probes & schedule • Tests • Demos • Follow on • 9k DDS preliminary • UCLP integration … ? • SOA ideas… ?