1 / 49

XenServer Storage Management and Troubleshooting

XenServer Storage Management and Troubleshooting. Daniel Lazar Lead Escalation Engineer May 11, 2010. Agenda. XenServer Storage Overview Management and Monitoring Troubleshooting and Diagnosing Common Storage Issues Q & A. XenServer Storage Overview. XenServer Storage Overview.

clancy
Download Presentation

XenServer Storage Management and Troubleshooting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. XenServer Storage Management and Troubleshooting Daniel Lazar Lead Escalation Engineer May 11, 2010

  2. Agenda • XenServer Storage Overview • Management and Monitoring • Troubleshooting and Diagnosing Common Storage Issues • Q & A Citrix Confidential - Do Not Distribute

  3. XenServer Storage Overview

  4. XenServer Storage Overview • XenServer Storage Objects • SRs, VDIs, PBDs and VBDs • Virtual Disk Data Formats • File-based VHD, LVM and StorageLink Citrix Confidential - Do Not Distribute

  5. XenServer Storage Objects What is an SR (Shared Repository)? Describes a particular storage target in which Virtual Disk Images (VDIs) are stored. Flexible—supports a wide variety of storage types. Centralized—easier to manage, more reliable with a XenServer pool. Must be accessible to each XenServer host. Citrix Confidential - Do Not Distribute

  6. XenServer Storage Objects VDIs, PBDs, VBDs Virtual Disk Images are a storage abstraction that is presented to a VM. Physical Block Devices represent the interface between a physical server and an attached SR. Virtual Block Devices are connector objects that allow mappings between VDIs and VMs. Citrix Confidential - Do Not Distribute

  7. XenServer Storage Objects SR PBD VDI VBD XenServer Host Virtual Machine VDI VBD PBD XenServer Host Virtual Machine VDI VBD PBD XenServer Host

  8. Virtual Disk Data Formats File-based VHD VM images are stored as thin-provisioned VHD format files on either a local non-shared file system (EXT type SR) or a shared NFS target (NFS type SR). What is VHD? • A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a physical Hard Disk Drive. • Image Format Specification was created by Microsoft in June, 2005. Citrix Confidential - Do Not Distribute

  9. Virtual Disk Data Formats Logical Volume (LVM)-based VHDs The default XenServer block device-based storage inserts a Logical Volume manager on a disk. VDIs are represented as volumes within the Volume manager. Introduced LVHD in XenServer 5.5 • Enhances LVM for SRs • Hosts VHD files directly on LVM volumes • Adds Advanced Storage features like Fast Cloning and Snapshots • Fast and simple upgrade • Backwards compatible Citrix Confidential - Do Not Distribute

  10. Virtual Disk Data Formats StorageLink (LUN per VDI) LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plug-in (NetApp, Equallogic or StorageLink type SRs). The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level. Citrix Confidential - Do Not Distribute

  11. Virtual Disk Data Formats StorageLink Architecture • XenServer calls direct to Array API‘s to provision and adjust storage on demand. • Fully leverages array hardware capabilities. • Virtual disk drives are individual LUNs. • High performance storage model. • Only the server running a VM connects to the individual LUN(s) for that VM. • A special master server coordinates which servers connect to which LUNs Citrix Confidential - Do Not Distribute

  12. LVM vs. StorageLink XenServer 5.5 iSCSI / FC XenServer 5.5 iSCSI / FC + Storage Repository Storage Repository LUN LUN LUN VHD header VHD header LVM Logical Volume LVM Logical Volume LVM Volume Group VM Virtual Disk

  13. Storage Management and Monitoring

  14. Management and Monitoring Overview • Understanding how XenServer Perceives the Storage • Monitoring Storage • Protecting Your Data Citrix Confidential - Do Not Distribute

  15. Management and Monitoring Understanding the physical disk layout # fdisk –l # Lists the physical block devices on the host Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes 255 heads, 32 sectors/track, 35132 cylinders Units = cylinders of 8160 * 512 = 4177920 bytes Device Boot Start End Blocks Id System /dev/cciss/c0d0p1 * 1 981 4002464 83 Linux /dev/cciss/c0d0p2 982 1962 4002480 83 Linux /dev/cciss/c0d0p3 1963 35132 135333600 83 Linux Denotes a SCSI block device locally attached to the system (HP RAID array in this case) The first partition on the disk contains the boot information for the OS. Citrix Confidential - Do Not Distribute

  16. Management and Monitoring Understanding the physical disk layout (continued) # fdisk –l # Continued output Disk /dev/sda: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sda doesn't contain a valid partition table Implies a block device using the SCSI Generic (sg) driver. It is likely attached via a separate interface such as iSCSI or FC HBA This disk is part of a Storage Repository using an LVM file system and therefore does not require a local partition table. Citrix Confidential - Do Not Distribute

  17. Management and Monitoring # sg_map –x # Displays the mapping between Linux sg and regular SCSI devices /dev/sg0 0 0 0 0 13 /dev/sg1 0 0 0 1 0 /dev/sda /dev/sg2 0 0 0 2 0 /dev/sdb /dev/sg3 1 0 0 0 13 /dev/sg4 1 0 0 1 0 /dev/sdc /dev/sg5 1 0 0 2 0 /dev/sdd Understanding the physical disk layout (continued) Host Number Bus SCSI ID LUN SCSI Type Citrix Confidential - Do Not Distribute

  18. Management and Monitoring Understanding the physical disk layout (continued) # ll /dev/disk/by-id # List the attached block devices by SCSI ID. cciss-3600508b1001035373120202020200003 -> ../../cciss/c0d0 cciss-3600508b1001035373120202020200003-part1 -> ../../cciss/c0d0p1 cciss-3600508b1001035373120202020200003-part2 -> ../../cciss/c0d0p2 cciss-3600508b1001035373120202020200003-part3 -> ../../cciss/c0d0p3 scsi-360a98000503350642f4a553833616b57 -> ../../sda This SCSI device is mapped to /dev/sda Unique ID assigned by udev. It corresponds to individual block devices. Citrix Confidential - Do Not Distribute

  19. Management and Monitoring Understanding the physical disk layout (continued) To identify a specific SR based on the SCSI ID, compare /dev/disk/by-id with the SR in XenCenter Citrix Confidential - Do Not Distribute

  20. Management and Monitoring LVM-related commands # pvs # Lists physical volumes PV VG Fmt Attr PSize PFree /dev/sda VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 lvm2 a- 99.99G 59.88G # vgs # Lists volume groups VG #PV #LV #SN Attr VSize VFree VG_XenStorage-40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 1 4 0 wz--n- 99.99G 59.88G Linux sg device LVM Volume Group stored on the physical volume. SR UUID Citrix Confidential - Do Not Distribute

  21. Management and Monitoring LVM (continued) # lvs # Lists the logical volumes LV VG AttrLSize VHD-c67a887f-3a1a-41f4-8d40-1b21f6307c4a VG_XenStor... -wi--- 24.00G VHD-c9b919a7-b93b-49ea-abe5-00acb8240cf5 VG_XenStor... -wi-ao 8.00G VHD-f3d26dde-254f-4d80-a3bb-d993e904bd63 VG_XenStor... -wi--- 24.00G LV-e056f479-b0f3-49f3-bc5d-6c226657ae6c VG_XenStor... -wi-ao 10.00G LV-ebdcad46-66d9-4020-baa1-0d5b6ac439c7 VG_XenStor... -wi-ao 24.00G The ‘a’ and ‘o’ attributes indicate the LV is ‘active’ and ‘open’ implying it is attached to a running VM Tip: Type ‘lvm help’ for a complete list of LVM command options. Represents Logical Volume containers for individual VDIs. Citrix Confidential - Do Not Distribute

  22. Management and Monitoring Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI # xesr-list type=lvmoiscsiparams=name-label,uuid,VDIs,PBDs # Lists the SRs configured for the pool name-label ( RW) : NetApp - iSCSI uuid ( RO) : 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 VDIs (SRO) : f3d26dde-254f-4d80-a3bb-d993e904bd63; c67a887f-3a1a-41f4... PBDs (SRO) : 27d05ffc-07d3-4f02-d265-3594a2179f8f Note that the VDI UUID is the same as the logical volume ID. We will make a note of this UUID to refer back to. Using the PBD UUID from this command output we will query for its characteristics in the next slide… Citrix Confidential - Do Not Distribute

  23. Management and Monitoring Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xepbd-list uuid=27d0… params=uuid,sr-uuid,device-config,currently-attached # List PBD params uuid ( RO) : 27d05ffc-07d3-4f02-d265-3594a2179f8f sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 device-config (MRO): port: 3260; SCSIid: 360a98000503350642f4a553833616b57; target: 10.12.45.10; targetIQN: iqn.1992-08.com.netapp:sn.135027806 currently-attached ( RO): true ‘device-config’ describes all the physical characteristics of the block device attached to this PBD. Note the SCSIid as referenced earlier from /dev/disk/by-id Citrix Confidential - Do Not Distribute

  24. Management and Monitoring Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xevdi-list uuid=f3d26dde-254f-4d80-a3bb-… params=uuid,sr-uuid,vbd-uuids # List VDI params uuid ( RO) : f3d26dde-254f-4d80-a3bb-d993e904bd63 sr-uuid ( RO): 40bbf542-b9d9-ffa1-6efe-aa9c56aadd95 vbd-uuids (SRO): 69afb055-3b52-57e3-63fa-d26b82a9b01d This tells us what VBDs are attached to this VDI. We will use this UUID in the next slide to query for the VBD characteristics and determine which VM this disk is attached to. Citrix Confidential - Do Not Distribute

  25. Management and Monitoring Understanding how the physical storage is represented as virtual objects in XenServer using the XenAPI (continued) # xevbd-list uuid=69afb055-3b52-… params=uuid,vm-uuid,vm-name-label,vdi-uuid,mode # List VBD params uuid ( RO) : 69afb055-3b52-57e3-63fa-d26b82a9b01d vm-uuid ( RO): 2c3a0e82-3f96-eab8-4982-db33fdb3bd88 vm-name-label ( RO): Windows 7 Test vdi-uuid ( RO): f3d26dde-254f-4d80-a3bb-d993e904bd63 mode ( RW): RW This tells us which VM (name and UUID) this VBD is attached to, and which VDI it is providing to the VM. Tip: You can issue ‘xe help <command>’ to get syntax help for any ‘xe’ commands. Citrix Confidential - Do Not Distribute

  26. Management and Monitoring FibreChannel LUN Zoning Since Enterprise SANs consolidate data from multiple servers and operating systems, many types of traffic and data are sent through the interface, whether it is fabric or the network. With Fibre Channel, to ensure security and dedicated resources, an administrator creates zones and zone sets to restrict access to specified areas. A zone divides the fabric into groups of devices. Zone sets are groups of zones. Each zone set represents different configurations that optimize the fabric for certain functions. WWN - Each HBA has a unique World Wide Name (similar to an Ethernet MAC) node WWN (WWNN) - can be shared by some or all ports of a device port WWN (WWPN) - necessarily unique to each port

  27. Fibre Channel LUN Zoning Pool1 Pool2 Xen1 Xen3 Xen2 Zone1 Zone2 Xen2 WWN Xen1 WWN Xen3 WWN FC Switch Storage WWN Storage WWN Initiator Group Xen3 Initiator Group Xen1, Xen2 Storage LUN0 LUN1 LUN2 FC Switch example

  28. Management and Monitoring iSCSI Isolation With iSCSI type storage a similar concept of isolation as fibre-channel zoning can be achieved by using IP subnets and, if required, VLANs. IQN – Each storage interface (NIC or iSCSI HBA) has configured a unique iSCSI Qualified Name Target IQN – Typically associated with the storage provider interface Initiator IQN – Configured on the client side, i.e. the device requesting access to the storage. IQN format is standardized: iqn.yyyy-mm.{reversed domain name} (e.g. iqn.2001-04.com.acme:storage.tape.sys1.xyz)

  29. iSCSI Isolation Pool1 Pool2 Xen1 Xen3 Xen2 VLAN2 / Subnet2 VLAN1 / Subnet1 Xen1 Initiator IQN Xen2 Initiator IQN Xen3 Initiator IQN Network Switch Controller 1 Target IQN Controller 2 Target IQN Controller Interface 2 Controller Interface 1 Storage LUN0 LUN1 LUN2 iSCSI Example

  30. Management and Monitoring Monitoring XenServer Storage - Alerts XenServer will generate alerts for certain storage events: • Missing or duplicate IQNs configured • HA state file lost or inaccessible • PBD plug failure on server startup XenServer can be configured to send alert notifications via email too. See the XenServer Administrator’s Guide for more information about configuring alerts. Citrix Confidential - Do Not Distribute

  31. Management and Monitoring Monitoring XenServer Storage – CLI Commands # iostat –k # Reports basic I/O stats for devices and partitions avg-cpu: %user %nice %system %iowait %steal %idle 0.12 0.00 0.05 0.09 0.02 99.72 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn cciss/c0d0 4.05 0.52 32.11 164361 10156264 sda 0.11 1.38 1.79 437259 566151 Note: iostat is not a great performance indicator for shared storage devices because it is unaware of external bottlenecks, for example the network in the case of iSCSI. Citrix Confidential - Do Not Distribute

  32. Management and Monitoring Monitoring XenServer Storage – CLI Commands # hdparm –t /dev/<device> # Performs timed sequential reads /dev/cciss/c0d0: Timing buffered disk reads: 286 MB in 3.00 seconds = 95.19 MB/sec • Has some limitations: • Does not measure non-sequential disk reads. • Does not measure disk write speed • May not be accurate with non-local storage devices since it is unaware of underlying bus architecture (iSCSI, FC, etc.) • Must be sampled repeatedly over time to get an accurate picture of I/O read performance. Citrix Confidential - Do Not Distribute

  33. Management and Monitoring Monitoring XenServer Storage – CLI Commands # dd if=<infile> of=<outfile> # Simple, common block device copy utility # dd if=/dev/<device> of=/dev/null 1998929+0 records in 1998929+0 records out 1023451648 bytes (1.0 GB) copied, 13.8456 seconds, 73.9 MB/s if = ‘infile’, the source dd reads from. of = ‘outfile’, the target dd writes to. WARNING: NEVER run dd specifying an active, running VHD as the outfile—it WILL destroy the VM container making it unreadable!! Citrix Confidential - Do Not Distribute

  34. Management and Monitoring Monitoring XenServer Storage – Additional Tips iSCSI storage throughput can usually be tied directly to network performance. If there is slow throughput for an iSCSI storage array, perform network diagnostics first!! Many SAN arrays have native logging and monitoring tools that can identify bottlenecks affecting storage performance. Refer to the Citrix Knowledge Base for best practices and known issues relating to storage performance. • http://support.citrix.com/article/CTX121634 • http://support.citrix.com/article/CTX122806 • http://support.citrix.com/article/CTX120737 Citrix Confidential - Do Not Distribute

  35. Management and Monitoring Protecting Your Data – Backup VM Metadata • Can use xsconsole or the CLI. • Makes the SR “portable”. • Can be used as part of a Disaster Recovery solution, or, as part of regular maintenance of the environment. • Can be scheduled within xsconsole. For more information relating to using XenServer as a Disaster Recovery solution, refer to the Citrix Knowledge Center: http://support.citrix.com/article/CTX117258 http://support.citrix.com/article/CTX121099 Citrix Confidential - Do Not Distribute

  36. Management and Monitoring Protecting Your Data – Exporting VMs Virtual machines can be exported directly out of XenServer into XVA files that contain a complete clone of the VM and all of its attached VDIs. Can be initiated via XenCenter or from the XenServer CLI. VM must be offline (shutdown) during export process. Since it backs up all the VM data it can take a very long time depending on the size of the VM! Citrix Confidential - Do Not Distribute

  37. Management and Monitoring Protecting Your Data – Creating VM Snapshots Snapshots create VDI clones of a VM that can be used for backup or quickly provisioned into new VMs or templates. XenServer supports two types in version 5.5 • Regular – Supports all guest environments, including Linux • Quiesced – Takes advantage of Windows Volume Shadow Copy Service (VSS). It requires the manual installation of in-guest components to enable. Citrix Confidential - Do Not Distribute

  38. Management and Monitoring Protecting Your Data – Creating VM Snapshots (continued) New in XenServer 5.6! • Introduces snapshot “Revert”, a.k.a. “Checkpoint”. • Introduces a new snapshot mode: “Snapshot with disk and memory” • XenCenter GUI enhanced for easier management of VM snapshots and to support Checkpoint feature. Citrix Confidential - Do Not Distribute

  39. Management and Monitoring Protecting Your Data – Third-Party Solutions There are also Third-Party backup options: • In-guest backups can be performed using any guest-supported solution (backup agents running in Windows or Linux, for example). • Volume snapshots performed directly on the storage via StorageLink plugins (for Dell and NetApp). • Backup solutions that plug into the XenAPI to capture VM data, or clone the LVM data directly. Citrix Confidential - Do Not Distribute

  40. Troubleshooting and Diagnosing Common Storage Issues

  41. Troubleshooting XenServer Storage Native Troubleshooting Tools – XenServer Logs Always check the logs first! XenServer creates several logs that are useful for diagnosing storage problems • /var/log/messages # General messages and system related stuff • /var/log/xensource.log # Logging specific to XenAPI • /var/log/SMlog # Logging specific to XenServer storage manager Often errors logged in any of these files can be searched for in the Citrix Knowledge Center for a solution. See http://support.citrix.com. Citrix Confidential - Do Not Distribute

  42. Troubleshooting XenServer Storage Native Troubleshooting Tools – XenAPI commands The XenAPI (xe) can be used to troubleshoot storage issues too • # xe sr-scan # Force XAPI to sync the database with local VDIs present in the underlying substrate. • # xe sr-probe # Using device-config parameters you can probe a block device for its characteristics, such as existing VM metadata and SR uuid. • # xe pbd-plug/unplug # Manually plug or unplug a PBD for an SR. This can be useful when repairing an SR in XenCenter fails. Citrix Confidential - Do Not Distribute

  43. Troubleshooting XenServer Storage Native Troubleshooting Tools – VHD commands See and verify mount point of VHD SR • # /var/run/sr-mount/<SR UUID> “full provision” VHD SR • vhd-util • See http://support.citrix.com/article/CTX118842 Check VHD architecture • # hexdump -vC <VDI-UUID>.vhd | less Citrix Confidential - Do Not Distribute

  44. Troubleshooting XenServer Storage Storage Multipathing Ensure that multipathing is enabled if you have multiple paths zoned to the XenServer • Use ‘sg_map –x’ and check the host and bus IDs Problems if you do not enable multipath • I/O Errors • Decrease in performance • Introduce errors with SR.create What is multipath.conf vs multipath-enabled.conf • multipath.conf is symlink to multipath-enable.conf or multipath-disabled.conf DMP vs. MPP multipathing • http://support.citrix.com/article/ctx121364 Citrix Confidential - Do Not Distribute

  45. Troubleshooting XenServer Storage SAN Debugging Always start at the hardware adapter, use the Qlogic or Emulex CLI tools to verify the LUNs known to the adapter • For QLogic, run ‘scli’ • For Emulex, run ‘hbanywhere’ Use ‘xe sr-probe type=lvmohba’ to trigger a bus refresh Citrix Confidential - Do Not Distribute

  46. Troubleshooting XenServer Storage Additional Scenarios • Unable to create SRs • Verify that XenServer can see the storage/LUN • Use fdisk and /dev/disk/xxx • Verify that HBA can see the LUN • Use the HBA CLI tools • Verify that iSCSI can login: • # iscsiadm –m node –L all # Will force iscsid service to log into the storage array. • Clearing the device mappings via CMD line • # echo 1 > /sys/class/scsi_devices/x:x:x:x/device/delete • Be extremely careful what device is being deleted! • Clean up of orphaned VDIs, XC not displaying the right amount of free storage • If a logical volume has no corresponding VDI it can be deleted. Be extremely careful with this because if you delete a parent disk, then you lost all differentiated disks.

  47. Questions? Comments?

  48. The TechEdge survey will be emailed out to end-user customers If you complete the survey, you will be entered to win a $250 Amazon gift card. The winner will be announced June 1st. View TechEdge videos & PPTs on the Knowledge Center by Monday, May17th http://support.citrix.com/techedge2010 TechEdge Survey, Video Postings & PPTs

More Related