1 / 138

Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter

Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter. Cisco Systems Matt Slavin mslavin@cisco.com. Jan 2005. Agenda. CIGESM Introduction and Features CIGESM Management Default Operation and Best Practices Configuration Example Serial over LAN

barton
Download Presentation

Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cisco Systems Intelligent Gigabit Ethernet Switch Module for the IBM eServer BladeCenter Cisco Systems Matt Slavin mslavin@cisco.com Jan 2005

  2. Agenda • CIGESM Introduction and Features • CIGESM Management • Default Operation and Best Practices • Configuration Example • Serial over LAN • Hints and Tips • Important links

  3. Cisco Systems Intelligent Gigabit Ethernet Switch ModuleCo-Branded IBM and Cisco Product for the IBM eServer BladeCenter Cisco Systems IGESM • Developed and Manufactured by Cisco for IBM • Available from IBM and IBM Resellers • CIGESM GA by IBM on Jun 11th ’04 • Layer 2-Plus Gigabit Ethernet switch • Does more than pure L2 switching • Switching decisions based on L2/3/4 • Runs Cisco IOS Software • Advanced Management Features • 4 External 10/100/1000 RJ45 Copper Uplinks • 14 Internal GigE Ports for BladeServers • 1 Service port (console) sealed with cap • Up to 4 CIGESMs supported per BladeCenter

  4. Chassis View (Rear) Serial Port (cap plate removed) • 8 MB of Flash Memory • 32 MB of DRAM • 8K MAC Address Support • 64 Instances of Spanning-Tree • Mini-Jumbo Support (1530 Bytes) • Up to 250 Active VLANs 4 RJ45 Copper Data Ports 10/100/1000 Mbps

  5. I2C - B I2C - A Supervisor CPU +1.25V PwrOn DC-DC Converters +12V +2.5V I2C Logic (Control/ Status Interface) PwrOn 32MB SDRAM +3.3V 8MB Flash PCI Serial 10 Gig 12xGigE Switch ASIC 12xGigE Switch ASIC service 4x 10/100/1000 Copper RJ-45 QUAD PHY 10/100/1000 PHY 10/100/1000 PHY 14x SerDes GigE To Processor Blades 2x FastE To Mgmt Modules HW Block Diagram

  6. CIGESM Port Assignments • All ports tasked for a specific roll and can not be reallocated

  7. Internal ESM L2 Traffic Flow Hard Filter that prevents traffic flow between the External ports and Mgmt Module ports CIGESM 4 External Ports Primary Mgmt Module 2 Mgmt Module Ports CPU / IOS Redundant Mgmt Module 14 Internal Blade ports

  8. Blade Chassis With Ethernet Modules Management Management Module Copper gigabit uplinks Ethernet Switch 1 Ethernet Switch 2 P2p connection NIC2 NIC 2 NIC 1 NIC 1 Blade Server Modules …….. I2c Bus Management Traffic Only Management Ethernet ports

  9. Example Architecture Data Center Core/Distribution • Cisco Catalyst 6500 Switch Core or • Cisco Catalyst 7600 Router Gateway • Layer 4-7 Service Modules (FWSM, SSL, VPN) • Load Balancing (EWLM) • Network Analysis (NAM-1 & NAM-2) • Intrusion Detection (IDS Module) Core or Gateway router Mgmt Network Distribution Layer Standard BladeCenter Redundancy Deployment • 2 L2+ Switch Modules in the chassis • 2 upstream Switches aggregating the BladeCenter systems • 2-wire channels to alternate switches • A core router at the top • Separate mgmt network SM 1 Access Layer SM 2 Mgmt M. 14 Blade Server Blades BladeCenter Chassis

  10. Availability / Resiliency Optimized Delivery Advanced STP Protocols for faster Spanning tree Convergence (Rapid PVST, PVST+, Uplink Fast, Root Guard, Port Fast, STP Backbone Fast) 802.1d 802.1s/w 802.3ad (LACP) and PAgP VTP 802.1q VLAN Trunking Broadcast control / Storm control IGMP Snooping (Multicast) IGMP Filtering (Multicast) L2-L4 QoS with CoS/DSCP Shaped Round Robin Strict Priority Queuing Rate Limiting Dynamic VLANs Integrated Security Enhanced Manageability Cisco IOS CLI Console Port (IBM Service Port) Embedded CMS SNMP MIBs for Enterprise Management Systems Port Mirroring (SPAN, RSPAN) Cisco Discovery Protocol (CDP) Network Time Protocol (NTP) Show interface capabilities Extensive Debugging and Trouble Shooting Capabilities CiscoWorks support • SSH • SNMPv3 (Crypto) • ACPs based on Layer 2-4 • Port Security w/aging • ACLs based on DSCP filtering • Unicast MAC filtering • Private VLAN Edge • MAC Address Notification • RADIUS / TACACS+ • Spanning Tree Root Guard • Trusted Boundary • Aggressive UDLD • 802.1x authentication CIGESM IOS Feature Set

  11. Agenda • CIGESM Introduction and Features • CIGESM Management • Default Operation and Best Practices • Configuration Example • Serial over LAN • Hints and Tips • Important links

  12. Intro to Managing the CIGESM • Ways to access the switch • Management Module • Launch telnet java applet • Launch http to CMS • Telnet • Direct launch • Through Management Module • Through IBM Director • CMS (Cluster Management Suite) • Direct launch • Through Management Module • Through IBM Director • Service port (console connection) • Default - 9600, N, 8 1 • Default user name and password (all caps): • USERID • PASSW0RD (0=zero)

  13. Management connection flows • Paths 1 and 2 might be classified as Out-of-band or In-band management paths to the CIGESM • Paths 3 and 4 are classified as traditional In-band management paths to the CIGESM • Ether/Or Decision: • Can use 1 and 2, OR 3 and 4, but not both at the same time • For example, you can not configure for path 2 and path 3 at the same time! • Path 5 is sometimes classified as a form of Out-of-band and is always available.

  14. Management Path Consideration - MM • VLAN selection when using MM path to manage CIGESM • Can set VLAN different on CIGESM, then that on MM upstream switch and still operate • Confusing at a minimum • Recommend to set as same VLAN to prevent confusion

  15. Management Path Consideration - Uplink • VLAN selection when using CIGESM path to manage CIGESM • Even though using CIGESM uplinks, VLAN still carried over to MM • Confusing at a minimum • Can result in unexpected interaction • Need to be aware of this • End Result: • VLAN to be used should be unique from all other connections to BladeCenter

  16. Management Path Consideration • Selection of CIGESM management VLAN when multiple CIGESMs in BladeCenter • MM connection exists between CIGESMs • If different management VLANs configured, error messages generated • Native VLAN Mismatch • Best Practice: • Use same VLAN for each CIGESM in a given BladeCenter

  17. Scenario 1 – RecommendedManagement Module provides path to the CIGESM Four Basic Rules • Disable management over CIGESM uplinks • Should block CIGESM management VLAN from CIGESM uplinks • CIGESM VLAN not used by any BladeServers • Same IP subnet in use on both MM and CGIESM management interface

  18. Scenario 2 – RecommendedManagement Module provides path to the CIGESM Same rules as Scenario 1 except: MUST block CIGESM management VLAN from CIGESM uplinks

  19. Scenario 3 – RecommendedCIGESM Uplinks used for management Five Basic Rules CIGESM management over CIGESM uplinks • Enable management over CIGESM uplinks • CIGESM mgmt VLAN not used by any BladeServers • Management VLAN must be carried on CIGESM uplinks • CIGESM mgmt VLAN not to be used as MM upstream VLAN • Different IP subnets in use on CIGESM and MM

  20. Scenario 4 – Alternative CIGESM Uplinks used for management Same rules as Scenario 3 IP subnet on ETH0 and ETH1 of MM must be different than that used by the CIGESM Possible drawbacks: Mixes traffic types Possible IP proxy issue

  21. Scenario 5 – NOT Recommended Broken Design CIGESM management over CIGESM uplinks, shared VLANs between CIGESM and MM • MM may attempt to proxy ARP for BladeServers • ARP war between MM and CIGESM on upstream network • Could result in BladeServers reporting duplicate IP address and not getting on to network

  22. Scenario 6 – NOT Recommended Broken Design CIGESM management over CIGESM uplinks, shared VLANs between CIGESM and MM and BladeServers • MM will proxy ARP for CIGESM • Will result in CIGESM and MM ARP wars • MM will proxy ARP for BladeServers • Will result in BladeServers reporting duplicate IP address and not getting on to network • Mixes all traffic

  23. Scenario 7 – Eval environment only Single VLAN design for EVAL CIGESM management over MM uplinks • Allows for a single VLAN design but has caveats so not recommended for production environment • If BladeServer ports trunked, MUST block CIGESM management VLAN • Mixes all traffic types

  24. CIGESM Management Tools • CiscoWorks Integration • Full CiscoWorks support now available! • CiscoView provides IGESM front panel view, switch status, port state, and device configuration • Cisco Cluster Management Suite (CMS) support • Express Setup for device initialization • Single IP address to manage switch cluster

  25. Example of Cisco View Front Panel View Port Status

  26. Example of Cluster Manager Suite

  27. Example of Management Module Config

  28. Example of Management Module Config External Ports MUST be enabled from MM

  29. Agenda • CIGESM Introduction and Features • CIGESM Management • Default Operation and Best Practices • Configuration Example • Serial over LAN • Hints and Tips • Important links

  30. Default Operation of the CIGESM • When CIGEMS is in factory default state: • All Bladeserver facing ports are in 802.1Q trunking mode • If BladeServer is not trunking, will appear in VLAN 2 of CIGESM by default • All Up-link ports will default to try to match other side • May become 802.1Q trunk or Access port • If Trunk, native VLAN will be 2 (other side must match) • If Access, defaults to VLAN 1 • Links to Management Modules are in VLAN 1 • Default IP addresses for CIGESM: • Bay 1 192.168.70.127 • Bay 2 192.168.70.128 • Bay 3 192.168.70.129 • Bay 4 192.168.70.130

  31. Default Operation of the CIGESM • Uplink ports disabled in default condition • Must enable prior to use • Must be done initially via MM • Management over uplinks disabled by default • Must be enabled via MM if in-band management is desired • Default port configurations are rarely optimal for production! • May be suitable for early deployment lab testing • Just plugging in to upstream switches without proper config may cause upstream ports to go into disable state

  32. Production Best Practices - Critical • Rule Number 1:Pay attention to management path configuration • See management path recommendation slides • Rule Number 2: Upgrade code on the CIGESM to latest release • Latest release 12.1(14)AY4 • Check readme for fixes • See summary section for link to latest code

  33. Production Best Practices - Suggestions • Disable uplinks prior to initial configuration • Prevents unexpected STP loops while configuring both sides • Can be done with shutdown or disconnecting cables • Once both sides configured, re-enable interfaces • Use Management Module uplink to manage the CIGESM when possible • Default management configuration is this path • Save CIGESM (to text file or other) prior to making changes • important in production environments to provide roll-back if required • Enable Root Guard on upstream switches connections to CIGESM’s • CIGESM is not a desired Spanning Tree Root switch • Would almost certainly result in suboptimal flows if CGIESM became the root of the spanning tree

  34. switch(config-if)# switchport trunk allowed vlan 2,3 switch(config-if)# switchport mode trunk Production Best Practices - Suggestions • Configure uplink facing ports exactly as required: • If using trunking, hard code port to trunk • If using trunking, limit it to carry only required VLANs • Example: If only VLANs 2 and 3 required on port: • If port is a Trunk port, Native VLAN MUST be allowed • Leave BladeServer facing ports as Trunks (default), and if Access port is desired, accomplish by changing the Native VLAN • Particularly important if using Serial Over LAN (SoL) • If no SoL, using “access” setting is fine description blade1 switchport access vlan 2 switchport trunk native vlan 2 switchport trunk allowed vlan 2-4094 switchport mode trunk spanning-tree portfast trunk spanning-tree bpdufilter enable

  35. Production Best Practices - Suggestions • Leave BladeServer NICs speed and duplex set for auto-negotiate • BladeServer ports on CIGESM are hard coded to auto-negotiate • Some newer BladeServer drivers also hard coded and can not be changed • Some confusion exists that the Native VLAN must be the same throughout a common L2 network. • This assumption is not correct • On any given link, both sides must use the same Native VLAN • But different links in the same L2 network can use a different Native VLAN

  36. Production Best Practices - Suggestions • Unless otherwise instructed by network administrators: • Leave VTP mode set for Transparent • Leave STP enabled • DO NOT change STP settings on ports 15 and 16 Incorrect setting could unexpectedly block uplinks • Use cross-over cables between CIGESM and upstream switch • If link speed/duplex is set to auto – does not matter • If link speed/duplex is hard coded – may matter • Makes use of the CIGESM Design Guide • Make use of CIGESM Deployment Redpaper • If you read nothing else, read Section 7.1, 7.2, 7.4.1 and Appendix A

  37. Agenda • CIGESM Introduction and Features • CIGESM Management • Default Operation and Best Practices • Configuration Example • Serial over LAN • Hints and Tips • Important links

  38. Example Topology Configuration • Assumptions • Example uses Topology 2 from the Redpaper • Port numbers on upstream switches may vary • IOS is being used in upstream switches (CatOS is possible but not covered here) • Trunking, LACP aggregation and access connections to BladeServers and uplinks may vary in your customers environment • For this topology to function correctly – VLANs carried on uplink ports to Aggregation Layer switches must also be carried between the Aggregation Layer switches

  39. Example Topology Configuration • More assumptions for this example • Assumes using out-of-band management via Management Module • Assumes use of VLANs 1, 10, 15, 20, 25, 30, 35, 40, 45 and 50 (per Redpaper Topology 2 configuration) • Assumes using dual-uplinks on dual CIGESMs • Only provided here as one possible solution • Your environment will almost certainly be different!

  40. Example Topology Configuration • Basic steps for deployment • Investigate requirements andplanthe design • Decide Management path information • Path for management (In-band or Out-of-band) • Agree upon IP addressing and VLAN utilization for management • Decide Data path information (up-link and BladeServer requirements) • Trunking, access, aggregation, VLANs, IP addressing • Implement the design • Configure Management Module • Configure up-stream devices • Configure CIGESMs • Configure BladeServers • Confirm expected operation • Should be more than just a ping test

  41. Investigate and Plan • Designing the connectivity and networking for the BladeCenter • Should be a concerted effort between the network administrators and the server administrators • Management path discussions • IP address, mask and default gateway for the Management Module in the BladeCenter • Need two IP addresses for the MM (1 external and 1 internal) • If redundant MM, only need to configure the primary • Internal and external IP addresses must be on the same IP subnet • Default gateway must be on the same subnet as the IP address of the MM • IP address, mask and default gateway for the CIGESM(s) in the BladeCenter • Need to agree upon management path (via MM or via CIGESM uplinks) • If using in-band (via the uplinks of the CIGESM) need to agree on VLAN for management

  42. Investigate and Plan • Data path discussions • Come up with an agreed upon network diagram/design • Topology 2 from Redpaper in our example • Needs to show uplink ports on CIGESM and their usage • Agreed upon IP addressing and VLAN usage to carry data traffic between the BladeServers and the production network • Port speed and duplex (default uplinks for CIGESM are auto-negotiate) • Type of aggregation (if any) between CIGESM and upstream switches - LACP in our example • CIGESM supports LACP, PAgP and static aggregation • Some older Cisco code may not support LACP • Type of trunking or access ports - 802.1Q in our example • CIGESM only supports 802.1Q trunking - Some older Cisco devices may not support 802.1Q • If trunking, native VLAN to be used (default for CIGESM is VLAN 2) • If trunking, VLANs to be carried

  43. Implementation - Configure the MM CIGESM

  44. Upstream Switch Config • Some possible areas of concern • Concern: Link aggregation desired but upstream switch does not support LACP • Some older Cisco code does not support LACP • Switch to using PAgP for aggregation • Upgrade code on upstream switch to support LACP • Some other vendors do not support LACP or PAgP • Switch to a vendor that supports desired protocol • Use static aggregation • Concern: VLAN trunking desired but upstream switch does not support 802.1Q • Some older Cisco switches support ISL only • Upgrade/replace upstream switch to support 802.1Q • Put a switch between the CIGESM and the upstream switch that supports 802.1Q and ISL • Use Access ports rather than trunk ports

  45. Configure The CIGESMs • Must Perform on ALL CIGESMs in a BladeCenter • Recommend uplink cables be left disconnected (or ports put in shutdown state) until CIGESMs fully configured • Add and Configure VLANs • Configure Management VLAN • Defaults to VLAN 1 • Defaults to IP address 192.168.70.X • Configure Upstream connections • Configure aggregation • Configure trunking • Add VLANs to be carried on trunks • Configure Connections to BladeServers • Configure access or trunking • Add VLAN(s) to be carried on ports • Save Configuration • Save it or loose it!

  46. Configure Top CIGESM (CIGESM 1)

  47. Configure Top CIGESM (CIGESM 1)

  48. Configure Bottom CIGESM (CIGESM 2)

  49. Configure Bottom CIGESM (CIGESM 2)

  50. Configure the BladeServers • Operating System dependent • Install Broadcom NIC drivers (or Intel if HS40) • For simple “Access” port configuration • Configure physical interfaces with desired IP information • For advanced port configuration (redundancy or trunking) • Install BASP (teaming) software (or Intel software for HS40) • Required if using NIC Teaming • Teaming used for Active/Active or Active/Passive • Teaming used to configure 802.1Q • With teaming software, create “teams” • Required for trunking (802.1Q) to CIGESM • Required for NIC redundancy (A/A or A/P) • With teaming software, create VLAN logical interfaces as required • Only required if trunking to CIGESM • Configure logical interfaces with desired IP information • See Redpaper for more details on configuring BladeServers for this example

More Related