1 / 46

Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues

Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues. Blaine A. Anaya XenServer Escalation Engineer 05/24/2011. Agenda. Overview . XenServer Networking Architecture / vSwitch Architecture. Troubleshooting the Network. Agenda. Overview .

freya-hale
Download Presentation

Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues Blaine A. Anaya XenServer Escalation Engineer 05/24/2011

  2. Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network

  3. Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network Performance Testing

  4. Networking Terminology Bond Trunk Bridge Switch Router

  5. XenServer Networking Terminology PIF- Physical Interface Object – directly correlates to a physical interface VIF- Virtual Interface Object- directly correlates to a virtual interface in a VM Bridge- Represents a network and is where PIFs and VIFs are plugged in Trunk – a switch port designated to carry traffic for more than one VLAN Bond- is the association of two network interface cards to make them appear as one Dom0- Short form of Domain 0 the control domain in XenServer that manages network and storage connections for virtual machines

  6. XenServer Networking Dom0 DomU DomU Toolstack App App App App Guest OS Guest OS netback/2 netback/1 netfront/ VIF netback/3 netback/0 netfront/VIF Native Driver / PIF Bridge Xen Hypervisor Host Machine (Hardware)

  7. XenServer Networking Configurations- Linux Stack Command Line Linux Config Files XenServer Pool DB Linux NIC Drivers XAPI xsconsole XenCenter Network Card

  8. XenServer Network Terminology Private (xapi1) VIF Internal Switches Virtual Machine Network 0 (xenbr0) VIF PIF (eth0) Virtual Machine VIF Network Card

  9. XenServer Network Terminology Network 0 (xenbr0) PIF (eth0) VIF Internal Switches Virtual Machine Network 1 (xenbr1) VIF PIF (eth1) Virtual Machine VIF Network Card Network Card

  10. XenServer Network Terminology Bond 0+1 (xapi2) PIF (eth0) PIF VIF Virtual Machine PIF (bond0) VIF VIF PIF (eth1) Virtual Machine Network Card Network Card

  11. Bonding Type (Balance SLB) 0:10 SEC 0:30 SEC 0:20 SEC 0:00 SEC Bond Stacked Switches Virtual Machine Virtual Machine Virtual Machine Network Card Network Card

  12. Distributed vSwitch

  13. Open Virtual Switch for XenServer Visibility· Resource control · Isolation · Security Hypervisor Hypervisor Hypervisor VM VM VM VM VM VM VM VM VM VM VM • Open Source Virtual Switch maintained at www.openvswitch.org • Rich layer 2 feature set (in contrast to others on the market) • Ships with XenServer 5.6 FP1 as a post-install configuration option

  14. Distributed Virtual Switch Controller Hypervisor Hypervisor Hypervisor Hypervisor VM VM VM VM VM VM VM VM VM VM VM DVS DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches

  15. Distributed Virtual Switch Hypervisor Hypervisor Hypervisor Built-in policy-based ACLs move with VMs VM VM VM VM VM VM VM VM VM VM VM Virtual Interface (VIF) {MAC, IP} ACLs permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123 Virtual Interface (VIF) {MAC, IP} ACLs permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123 DVS

  16. Distributed Virtual Switch Enabling the vSwitch • #xe-switch-network-backend openvswitch (Command must be ran on each individual host) • [root@vswitch1-baa-r222 ~]# xe-switch-network-backend openvswitch • Cleaning up old ifcfg files • Remove... ifcfg-bond0 • Remove... ifcfg-bond1 • Remove... ifcfg-eth0 • Remove... ifcfg-eth1 • Remove... ifcfg-eth2 • Remove... ifcfg-eth3 • Remove... ifcfg-eth4 • Remove... ifcfg-eth5 • Remove... ifcfg-xapi2 • Remove... ifcfg-xapi4 • Remove... ifcfg-xenbr0 • Remove... ifcfg-xenbr3 • Enabling openvswitch daemon • Configure system for openvswitch networking • You *MUST* now reboot your system

  17. Distributed Virtual Switch DVS Controller vSwitch Architecture – Process Level View JSON-RPC OpenFlow ovsdb-server vswitchd OVS Flow Table Flow Table VIF VIF Flow Table Cache Flow Table Cache vSwitch Network B VIF VIF vSwitch Network A PIF PIF

  18. XenServer Networking Configurations- vSwitch Command Line vSwitchConfig XenServer Pool DB Linux NIC Drivers XAPI xsconsole XenCenter Network Card

  19. DVSCWeb Interface

  20. Agenda Overview XenServer Networking Architecture / vSwitch Architecture Troubleshooting the Network

  21. Troubleshooting The Network Symptoms Issue Intermittent Packet Loss/ Dropped Connections • Physical Connection/Switch Configuration, Bonding • Physical Connection/Switch Configuration, Change in Hardware, Configuration Conflict. • Network Appears Disconnected • Bond Fails To Pass Traffic When One Leg is Disconnected • Physical Connection/Switch Configuration, Bond Mode

  22. Troubleshooting The Network • Using Command Line Interface (CLI) • Off-line using a system status report • BareGrep Pro • Xenoscope

  23. Troubleshooting The Network 1.Check switch port configuration – Physical – Layers1-3 (Cables,NICs,Switch/Router connections) 2.Verify enabled network backend (Linux Bridge/vSwitch) 3.Use ifconfig –a to see bonds, physical interface statistics, bridges. 4.Use “brctl show” to see bridge/bond association. 5.Verify bonding configuration 6.Use ethtool for NIC settings, driver and firmware versions. 7.Use xe network-list, xepif-list, to check XAPI configuration.

  24. Troubleshooting the Network

  25. Troubleshooting The Network Common Configuration Items to Check /etc/xensource/network.conf /etc/sysconfig/network-scripts /proc/net/bonding/bond0 /etc/sysconfig/iptables

  26. Troubleshooting The Network Linux Bridge/vSwitch Enabled # brctl show # Shows the bridges and the interfaces plugged into them [root@vswitch1-baa-r222 ~]# brctl show bridge name bridge id STP enabled interfaces xapi2 0000.001517868b8f no bond1 eth5 eth4 xapi4 0000.001d09699d86 no bond0 eth1 eth2 vif5.0 vif6.0 xenbr0 0000.001d09699d84 no eth0 xenbr3 0000.001517868b8c no eth3

  27. Troubleshooting The Network Linux Bridge Enabled #brctl showmacs <brname> #Shows a list of learned MAC addresses for this bridge. [root@localhost ~]# brctl showmacs xenbr0 port no mac addr is local? ageing timer 1 00:00:0c:07:ac:3c no 1.83 1 00:0c:29:3a:12:79 no 120.59 1 00:0c:29:fa:8e:e8 no 26.52

  28. Troubleshooting The Network Linux Bridge/vSwitch Enabled # netstat -np # Provides information on connections and processes. [root@vswitch1-baa-r222 ~]# netstat -np Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:37259 127.0.0.1:443 ESTABLISHED 2645/stunnel tcp 0 0 127.0.0.1:36806 127.0.0.1:80 ESTABLISHED 6280/stunnel tcp 0 52 10.12.45.209:22 10.54.75.163:63296 ESTABLISHED 31145/5 tcp 0 0 127.0.0.1:443 127.0.0.1:37259 ESTABLISHED 6280/stunnel tcp 0 0 10.12.45.209:443 10.12.45.114:39105 ESTABLISHED 6280/stunnel tcp 0 0 10.12.45.209:34969 10.12.45.194:6633 ESTABLISHED 5304/ovs-vswitchd

  29. Troubleshooting The Network Linux Bridge/vSwitch Enabled # netstat -s # Provides summary statistics for each protocol. [root@vswitch1-baa-r222 ~]# netstat -s Ip: 17340461 total packets received 9190 with invalid addresses 0 forwarded 0 incoming packets discarded 12463755 incoming packets delivered 14230986 requests sent out 8 dropped because of missing route Tcp: 69504 active connections openings 126760 passive connection openings 0 failed connection attempts 229 connection resets received 17 connections established 12462000 segments received 13220998 segments send out 3144 segments retransmited 0 bad segments received. 416 resets sent

  30. Troubleshooting The Network Linux Bridge/vSwitch Enabled #ethtool –k <interface> #Provides information on current offload settings [root@vswitch1-baa-r222 ~]# ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: off large-receive-offload: off

  31. Troubleshooting The Network Linux Bridge/vSwitch Enabled #ethtool –i <interface> #Provides information on driver/firmware versions for network cards [root@vswitch1-baa-r222 ~]# ethtool -i eth0 driver: bnx2 version: 2.0.8e firmware-version: bc 2.9.1 bus-info: 0000:04:00.0

  32. Troubleshooting The Network vSwitch Enabled #ovs-appctl bond/list #Shows Bridge, Bond, Slave Association [root@vswitch1-baa-r222 ~]# ovs-appctl bond/list bridge bond slaves Xapi2 bond1 eth4, eth5 Xapi4 bond0 eth2, eth1 Disclaimer: Using OVS command line options for configuration purposes is not supported. The vSwitch should only be configured using XenCenter, xe CLI, xsconsole, and the Distributed vSwitch Controller.The commands shared here are for data collection and diagnostic purposes only.

  33. Troubleshooting The Network vSwitch Enabled #ovs-appctl bond/show bond0 #Shows bond members, up/down delay, and next rebalance time. [root@vswitch1-baa-r222 ~]# ovs-appctl bond/show bond0 updelay: 31000 ms downdelay: 200 ms next rebalance: 4314 ms slave eth2: enabled active slave hash 123: 1 kB load 86:43:b2:1a:f2:d0 slave eth1: enabled

  34. Troubleshooting The Network vSwitch Enabled #ovs-appctl fdb/show <bridge_name> #Shows MAC Table/VLAN information for the bridge [root@vswitch1-baa-r222 ~]# ovs-appctl fdb/show xapi4 port VLAN MAC Age 3 0 00:1d:09:2c:c4:c9 58 3 0 0a:34:ee:08:53:06 47 3 0 6a:e8:14:89:5c:af 42 3 0 ba:89:bf:f5:b8:ab 35 3 0 00:16:c8:d8:f1:11 27

  35. Troubleshooting The Network vSwitch Enabled #ovs-ofctl dump-flows <bridge_name> #Shows FlowTable – (ACLs applied from controller) [root@vswitch1-baa-r222 ~]# ovs-ofctl dump-flows xapi4 | grep drop May 02 15:49:07|00001|ofctl|INFO|connecting to unix:/var/run/openvswitch/xapi4.mgmt cookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=32763, n_packets=0, n_bytes=0, tcp,dl_dst=86:43:b2:1a:f2:d0,nw_dst=10.12.45.151,tp_src=80,actions=drop cookie=0x0, duration_sec=171s, duration_nsec=25000000ns, table_id=1, priority=65529, n_packets=15, n_bytes=930, tcp,in_port=4,dl_src=86:43:b2:1a:f2:d0,nw_src=10.12.45.78,tp_dst=80,actions=drop

  36. Troubleshooting The Network vSwitch Enabled #ovs-dpctl dump-flows <bridge_name> #Shows FlowCache – (ACLs applied from controller) [root@vswitch1-baa-r222 ~]# ovs-dpctl dump-flows xapi4 | grep mac86:43 tunnel00000000:in_port0004:vlan65535:pcp0 mac86:43:b2:1a:f2:d0->00:00:0c:07:ac:3c type0800 proto6 tos0 ip10.12.45.78->69.147.112.160 port4284->80, packets:1, bytes:62, used:2.160s, actions:drop

  37. Troubleshooting The Network vSwitch Enabled #ovs-appctl vlog/list #Show current logging levels [root@vswitch1-baa-r222 ~]# ovs-appctl vlog/list console syslog file ------- ------ ------ bridge EMER ERR INFO vswitchd EMER ERR INFO xenserver EMER ERR INFO ofproto EMER ERR INFO sflow EMER ERR INFO jsonrpc EMER ERR INFO fail_open EMER ERR INFO netflow EMER ERR INFO ovsdb_error EMER ERR INFO

  38. Troubleshooting The Network vSwitch Enabled • #vlog/set module[:facility[:level]] • #Modify vswitch logging level • Sets the logging level for module in facility to level: • Module may be any valid module name (as displayed by the --list action on ovs-appctl(8)), or the special name ANY to set the logging levels for all modules. • Facility may be syslog, console, or file to set the levels for logging to the system log, the console, or a file respectively, or ANY to set the logging levels for both facilities. If it is omitted, facility defaults to ANY. • Note: The log level for the file facility has no effect unless ovs-vswitchd was invoked with the --log-file option. • Level must be one of emer, err, warn, info, or dbg, designating the minimum severity of a message for it to be logged. If it is omitted, level defaults to dbg.

  39. Troubleshooting The Network – Off-Line Status Report and BareGrepPro

  40. Troubleshooting The Network – Off-Line Status Report and Xenoscope

  41. Troubleshooting The Network – Off-Line Status Report and Xenoscope

  42. Troubleshooting The Network – Off-Line Status Report and Xenoscope

  43. Useful Networking CTX Articles • CTX127885 - Introduction to XenServer Networking • CTX123489 - XenServer VLAN Networking • CTX124421 - Understanding Network Interface Card Bonds in XenServer • CTX127970 - Distributing Guest Traffic Over Physical CPUs in XenServer • CTX127065- XenServer Virtual Machine Performance Utility • CTX123477 - How to Move a XenServer Pool to a Different IP Subnet • CTX125358 - How to Identify the Network Adapters on XenServer • CTX101810 - Communication Ports Used By Citrix Technologies

  44. Q & A

  45. Recommended related breakout sessions: Session: YN203: Managing VM networking across the datacenter with XenServer distributed virtual switching Date: Wednesday May 25th Time: 4:30-5:15 Room: Moscone2003-2005 Session surveys are available online at www.citrixsummit.com starting Thursday, May 26 Provide your feedback and pick up a complimentary gift at the registration desk Download presentations starting Friday, June 3, from your My Organizer Tool located in your My Synergy Microsite event account Before you leave…

More Related