Top Support issues and how to solve them – Part II

1. Darren Burnett Senior Technical Support Engineer Top Support issues and how to solve them – Part II

2. Unable to connect to the Service Console

3. Rebuild Networking Network Connection Problem. Deleting the vSwitch that vSwif0 is connected Connecting the wrong NICs to vSwitch0 Upgrade issues Incorrect IP Address External Network Changes

4. Rebuild Networking At this stage you can no longer connect to your ESX server using VI Client or SSH. You can connect to the Service Console remotely if you have ILO, DRAC an IP KVM or something similar. Otherwise, it’s time to use some shoe leather and walk to the server room.

5. Rebuild Networking The following procedure will work. However it is a quick and inelegant way to get your VI client connected. Other options include, Crossover cable connected to a laptop Adding or removing NICs to a vSwitch

6. Rebuild Networking Use the esxcfg-vswitch –l command to list all of you vSwitches

7. Rebuild Networking [root@newross root]# esxcfg-vswitch -l Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch0 32 4 32 vmnic0 PortGroup Name Internal ID VLAN ID Used Ports Uplinks VM Network portgroup1 0 0 vmnic0 Service Console portgroup0 0 1 vmnic0 VMkernel portgroup7 0 1 vmnic0 Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch1 64 2 64 vmnic1 PortGroup Name Internal ID VLAN ID Used Ports Uplinks vlan100 portgroup10 100 0 vmnic1 install VLAN 310 portgroup6 0 0 vmnic1 Switch Name Num Ports Used Ports Configured Ports Uplinks vSwitch2 64 2 64 vmnic6 PortGroup Name Internal ID VLAN ID Used Ports Uplinks crossovercable portgroup9 0 0 vmnic6

8. Rebuild Networking Delete all vSwitches esxcfg-vswitch –d vSwitch0 esxcfg-vswitch –d vSwitch1 esxcfg-vswitch –d vSwitch2

9. Rebuild Networking Create a vSwitch esxcfg-vswitch -a vSwitch0 Create the Service Console portgroup esxcfg-vswitch -p "Service Console" vSwitch0 Add a NIC to the vSwitch esxcfg-vswitch –L vmnic0 vSwitch0 Add a vswif interface and configure esxcfg-vswif –a vswif0 -p "Service Console" i 10.10.10.3 –n 255.0.0.0

10. Rebuild Networking Check if you can connect. Use PING both to and from the ESX server Try SSH Try VI Client

11. Rebuild Networking If you still can’t connect. Use “esxcfg-nics –l” to list available NICs.

12. Rebuild Networking [root@newross root]# esxcfg-nics -l Name PCI Driver Link Speed Duplex Description vmnic0 03:0c.00 e1000 Up 1000Mbps Full Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) vmnic1 03:0c.01 e1000 Down 0Mbps Half Intel Corporation 82546EB Gigabit Ethernet Controller (Copper) vmnic8 07:00.00 tg3 Up 1000Mbps Full Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet vmnic6 08:00.00 tg3 Down 0Mbps Half Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet

13. Rebuild Networking Remove and add NICs to vSwitch0 Remove esxcfg-vswitch –U vmnic0 vSwitch0 Add esxcfg-vswitch –L vmnic8 vSwitch0

14. Rebuild Networking It might also need a VLAN ID esxcfg-vswitch –v 101 –p “Service Console” vSwitch0

15. Rebuild Networking To avoid this issue, be careful when configuring the Service Console virtual NIC or its parent virtual switch property that can affect the Service Console virtual NIC connectivity, for example, the uplink. If possible, before updating the Service Console virtual NIC, create another independent working Service Console NIC so that, in the event the configuration brings the console NIC down, the second Service Console NIC is still available to repair the configuration.

16. Rebuild Networking As stated, depending on your environment it may be easier not to delete all your configurations. It may just require reassigning a NIC or changing VLAN ID. Check the following guide. http://www.rtfm-ed.eu/docs/vmwdocs/esx3.x-vc2.x-serviceconsole-guide.pdf

17. Network Bond NICs in a Bond not on the same Broadcast Domain

18. Network Bond NICs in a Bond should be in the same broadcast domain. Determine what NICs are in each Bond.

19. Network Bond [root@cork root]# esxcfg-info|grep -i -A 2 VirtualSwitchImpl \==+VirtualSwitchImpl : |----Name.........................................vSwitch0 |----Uplinks......................................vmnic0 -- \==+VirtualSwitchImpl : |----Name.........................................vSwitch1 |----Uplinks......................................vmnic4 -- \==+VirtualSwitchImpl : |----Name.........................................vSwitch2 |----Uplinks......................................vmnic1,vmnic3 --

20. Next we need to determine which networks each NIC is connected.

21. [root@cork root]# esxcfg-info |grep -i -B 5 hint \==+PnicImpl : |----_name..............................................vmnic3 |----_bus...............................................6 |----_slot..............................................3 |----_function..........................................1 |----Network Hint.......................................0 10.16.157.00/255.255.255.192 -- \==+PnicImpl : |----_name..............................................vmnic0 |----_bus...............................................11 |----_slot..............................................7 |----_function..........................................0 |----Network Hint.......................................0 10.16.156.00/255.255.255.00 -- \==+PnicImpl : |----_name..............................................vmnic1 |----_bus...............................................12 |----_slot..............................................8 |----_function..........................................0 |----Network Hint.......................................0 10.16.156.00/255.255.255.00

22. HA and STP

23. HA and STP Spanning Tree Protocol When there is a network change STP can cause a temporary outage on your Network

24. HA An ESX Server will determine that it is isolated after 15 Seconds Depending on the “Isolation Response” that you have set, all your VMs may power down

25. It is therefore worth checking your network to determine if STP can be configured to reduce the temporary network outage

26. Expanding the size of a VMDK with an existing Snapshot

27. Expanding VM with a Snapshot You can NOT expand a VM’s VMDK file while it still has snapshots. e.g. #ls * important.vmdk important-000001-delta.vmdk #vmkfstools –X 20G important.vmdk

28. Expanding VM with a Snapshot If you do, you will now have a VM that won’t boot

29. Expanding VM with a Snapshot Tricking ESX into seeing the expanded VMDK as the original size. In this example we have a test.vmdk that we expand from 5GB to 6GB #vmkfstools -X 6G test.vmdk

30. Expanding VM with a Snapshot If we check test.vmdk we see # Disk DescriptorFile version=1 CID=3f24a1b3 parentCID=ffffffff createType="vmfs" # Extent description RW 12582912 VMFS "test-flat.vmdk" # The Disk Data Base #DDB ddb.virtualHWVersion = "4" ddb.geometry.cylinders = "783" ddb.geometry.heads = "255" ddb.geometry.sectors = "63" ddb.adapterType = "buslogic"

31. Expanding VM with a Snapshot Original - RW 10485760 VMFS "test-flat.vmdk“ New - RW 12582912 VMFS "test-flat.vmdk“

32. Expanding VM with a Snapshot If we have no “BACKUPS” how do we get the original value? #grep -i rw test-000001.vmdk RW 10485760 VMFSSPARSE “test-000001-delta.vmdk"

33. Expanding VM with a Snapshot We change test.vmdk RW value. # Disk DescriptorFile version=1 CID=3f24a1b3 parentCID=ffffffff createType="vmfs" # Extent description RW 10485760 VMFS "test-flat.vmdk" # The Disk Data Base #DDB ddb.virtualHWVersion = "4" ddb.geometry.cylinders = "783" ddb.geometry.heads = "255" ddb.geometry.sectors = "63" ddb.adapterType = "buslogic"

34. Expanding VM with a Snapshot Commit The snapshot(s) #vmware-cmd /pathtovmx/test.vmx removesnapshots

35. Expanding VM with a Snapshot Grow the VMDK file #vmware-cmd –X 6GB test.vmdk

36. Expanding VM with a Snapshot If needed add a snapshot #vmware-cmd pathtovmx/test.vmx createsnapshot <name> <description>

37. Corrupted .VMSD file

38. Corrupted Snapshots In this example we will deal with a corrupt .VMSD file. Let’s first look at a working .VMSD file, separated into 3 slides for the people at the back

39. Corrupted Snapshots snapshot.lastUID = "4" snapshot.numSnapshots = "3" snapshot.current = "4" snapshot0.uid = "2" snapshot0.filename = "VC1.3to201_standard-Snapshot2.vmsn" snapshot0.displayName = "myfirst" snapshot0.description = "My first test snapshot" snapshot0.createTimeHigh = "273684" snapshot0.createTimeLow = "942632403" snapshot0.numDisks = "1" snapshot0.disk0.fileName = "VC1.3to201_standard.vmdk" snapshot0.disk0.node = "scsi0:0"

40. Corrupted Snapshots snapshot.needConsolidate = "FALSE" snapshot1.uid = "3" snapshot1.filename = "VC1.3to201_standard-Snapshot3.vmsn" snapshot1.parent = "2" snapshot1.displayName = "second" snapshot1.description = "My second test snapshot" snapshot1.createTimeHigh = "273684" snapshot1.createTimeLow = "980947483" snapshot1.numDisks = "1" snapshot1.disk0.fileName = "VC1.3to201_standard-000001.vmdk" snapshot1.disk0.node = "scsi0:0"

41. Corrupted Snapshots snapshot2.uid = "4" snapshot2.filename = "VC1.3to201_standard-Snapshot4.vmsn" snapshot2.parent = "3" snapshot2.displayName = "third" snapshot2.description = "My third test snapshot" snapshot2.createTimeHigh = "273684" snapshot2.createTimeLow = "1088942286" snapshot2.numDisks = "1" snapshot2.disk0.fileName = "VC1.3to201_standard-000002.vmdk" snapshot2.disk0.node = "scsi0:0"

42. Corrupted Snapshots After corruption of the .VMSD file the file now looks like this.

43. Corrupted Snapshots

44. Corrupted Snapshots However we see that the snapshots still exist [root@newross VC1.3to201_standard]# ls VC1.3to201_standard-000001-delta.vmdk VC1.3to201_standard-Snapshot4.vmsn VC1.3to201_standard-000001.vmdk VC1.3to201_standard.vmdk VC1.3to201_standard-000002-delta.vmdk VC1.3to201_standard.vmsd VC1.3to201_standard-000002.vmdk VC1.3to201_standard.vmx VC1.3to201_standard-000003-delta.vmdk VC1.3to201_standard.vmxf VC1.3to201_standard-000003.vmdk vmware-1.log VC1.3to201_standard-flat.vmdk vmware-2.log VC1.3to201_standard.nvram vmware-3.log VC1.3to201_standard-Snapshot2.vmsn vmware.log VC1.3to201_standard-Snapshot3.vmsn

45. Corrupted Snapshots At this stage rename the .VMSD file to .VMSD.OLD

46. Corrupted Snapshots We are going to create a new .VMSD file What kind of magic is required to build a new VMSD file?

47. Corrupted Snapshots Create another snapshot to automatically recreate a .VMSD file #vmware-cmd VC1.3to201_standard.vmx createsnapshot addedforrecovey "Hope it works"

48. Corrupted Snapshots You wont be able to selectively rollback to a particular snapshot. You will have to commit them all.

49. Corrupted Snapshots Commit the Snapshots #vmware-cmd VC1.3to201_standard.vmx removesnapshots All a bit too easy 

50. Corrupted Snapshots Corrupted Snapshot