第二章存储系统架构 Storage Systems Architecture

第二章存储系统架构Storage Systems Architecture 网络存储与系统虚拟化技术 Network Storage and System Virtualization Technology 北京大学信息科学技术学院网络与信息系统研究所系统虚拟化及空间信息技术实验室（http://gis.pku.edu.cn）

本章目标及内容 • 本章主要介绍存储架构的各个组成部分。首先介绍主机与存储的连接关系，然后从磁盘的基本概念出发，介绍了磁盘阵列和磁盘的RAID数据保护等概念。最后，本章介绍了磁盘存储系统的系统结构并深入说明了磁盘存储系统是如何高效的应用于系统环境的。 • 本章内容包括五个部分： • 2.1 主机环境（The Host Environment） • 2.2 连接（Connectivity: components and protocols） • 2.3 物理磁盘（Physical Disks） • 2.4 磁盘阵列（Disk Arrays） • 2.5 磁盘存储系统（Disk Storage Systems）

存储系统环境 Host system Connectivity Storage System 从单个机内磁盘到存储系统

Host • 物理部件： • CPU • Memory • Bus • 逻辑部件： • Software applications • Operating system • File systems • Databases

Connectivity • 物理部件 • 线路（Wiring） • 机内总线（Host internal buses） • 线缆（Cables） – Optical and Copper • 接头（Connectors and plugs） • 适配器（Adapters） • 主机总线适配器(Host Bus Adapter, HBA) - Enables devices to connect to a host’s internal bus system. • 网络接口卡（网卡）- Enables simple network attachments to a host. • 交换机（Switches/hubs） • 管理网络通讯及流量 • 逻辑部件 • 通信协议（Communication protocols） • 设备驱动程序（Device Drivers）

Storage • 物理部件 • 保存数据的设备 (如磁盘、磁带、光盘等) • 维持设备运行的辅助设备 (如电源、风扇等) • 机架 (racks) • 逻辑部件 • 协议（Protocols） • 数据流控制算法（Flow algorithms）

2.1 主机环境（The Host Environment） • 主机示例 Server Laptop Group of Servers Mainframe

CPU Bus Storage I/O Devices 主机物理部件

CPU CPU Bus ALU L1 Cache Bus Registers

… n Data n Data 3 3 2 Data 2 1 Data 1 0 Data 0 Address Content Storage Disk Memory

Fast Speed Slow Low High Cost Storage Hierarchy – Speed and Cost CPU registers L1 cache L2 cache Magnetic disk RAM Optical disk Tape

I/O Devices • 人机交互接口 • 键盘（Keyboard） • 鼠标（Mouse） • 显示器（Monitor） • 计算机-计算机交互接口 • 网卡（NIC） • 计算机-外围设备交互接口 • USB 口 • 主机总线适配器 (Host Bus Adapter, HBA)

主机逻辑部件

文件系统（File Systems） file systemblock Block Size

UNIX (UFS) File type and permissions Number of links Owner and group IDs Number of bytes in the file Last file access Last file modification 文件系统：元数据示例 • Windows (NTFS) • Time stamp and link count • File name • Access rights • File data • Index information • Volume information Directories and Files

文件系统：日志信息（Journaling and Logging） • Non-journaling file systems vs. Journaling file system • Journaling file system • 改善数据的完整性，加速系统重启时间 • Uses a separate area called a log or journal. • May hold all data to be written • May hold only metadata • Disadvantage - slower than other file systems. • Each file system update requires at least 1 extra write – to the log

Volume Management 卷管理：文件系统和物理磁盘之间的一个可选的中间层，能够合并或划分物理磁盘，形成虚拟磁盘；屏蔽物理磁盘的复杂结构，优化存储访问、简化存储资源的管理。

HBAs（Host Bus Adapters） HBA：用于连接存储设备的特殊硬件。通常是一块附加卡或主板上的一块芯片，并且有端口用于真正连接存储设备。HBA可以处理一些存储命令以减轻主机CPU的负担。

主机中用于改善数据可获得性(Availability)的部件主机中用于改善数据可获得性(Availability)的部件 • 通过冗余的手段 • Multiple HBAs • Multi-pathing software：驻留在服务器上提升数据可获得性的软件解决方案 • It utilizes the available HBAs on the server to provide redundant communication paths between host and storage devices. It provides multiple path I/O capabilities and path failover, and may also provide automatic load balancing. This assures uninterrupted data transfers even in the event of a path failure. • Clustering

Teacher Course File(s) File System Files File System Blocks Configures / Manages Reside in Mapped by file system to Disk Sectors Disk Physical Extents LVM Logical Extents Managed by Disk Storage Subsystem Consisting of Mapped by LVM to Residing in How Files are Moved to and from Storage

小结 • Hosts typically have: • Hardware: CPU, memory, buses, disks, ports, and interfaces. • Software: applications, operating systems, file systems, device drivers, volume managers • HBAsconnect hosts to storage devices. • Multi-pathing software uses redundant paths to ensure uninterrupted communication between the host and the storage • Clustering uses redundant host systems to improve data availability

2.2 连接（Connectivity） • 物理部件：Host with Internal Storage Bus CPU Port HBA Host Cable Port Disk

总线技术（Bus Technology ） Serial Serial Bi-directional Parallel

总线技术（Bus Technology ） • 系统总线（System Bus） – connects CPU to Memory • 外部总线（Local Bus or I/O Bus） – carries data to/from peripheral devices. • 总线带宽（Bus width）：measured in bits • 总线速度（Bus speed）：measured in MHz • 总线吞吐率（Throughput）：measured in MB/S

DirectlyAttachedEntities Network ConnectedEntities Tightly ConnectedEntities 连接协议（Connectivity Protocols ） • Protocol = a defined format for communication • allows the sending and receiving devices to agree on what is being communicated. System bus or I/O Bus processor to RAM Networked hosts NAS or SAN Devices connected at moderate distances host to printer host to JBOD or DAS

Host Apps Operating System PCI SCSI or IDE/ATA Device Drivers 通信协议：外部(I/O)总线 • PCI • IDE/ATA • SCSI

PCI总线 • Peripheral Component Interconnect (PCI) • defines the local bus system within a computer • It is an interconnection between microprocessor and attached devices, in which expansion slots are spaced closely for high-speed operation. • HBA for PCI：PCI expansion cards • Has Plug and Play functionality. • PCI Width is 32/64 bit • PCI Throughput is 133 MB/sec

IDE/ATA总线 • Integrated Device Electronics (IDE) / Advanced Technology Attachment (ATA) • Most popular interface used with modern hard disks • IDE盘、ATA盘 • Good performance at low cost • Desktop and laptop systems • Inexpensive storage interconnect SATA: Serial ATA / SATA盘 133MB/s vs. 150/300/600MB/s

SCSI总线 (Small Computer System Interface) • Most popular hard disk interface for servers. • Higher cost than IDE/ATA. • Supports multiple simultaneous data access. • Currently both parallel and serial forms. • Used primarily in “higher end” environments.

SCSI Model commands Target Initiator interrupts SCSI HBA Disks: SCSI Controller

LUNs Target ID Initiator ID SCSI Model Every device connected to SCSI should have an ID. LUN：Logical Unit Number, identifies a specific Logical Unit in a target.

LUN Initiator ID Target ID SCSI寻址（Addressing） • Initiator ID - a number from 0 to 15 with the most common value being 7. • Target ID - a number from 0 to 15 • LUN - a number that specifies a device addressable through a target.

Host Addressing Controller Target LUN t0 LUNs PeripheralController d0 d1 d2 Target c0 - ControllerInitiator, HBA SCSI寻址：确定磁盘 c0 t0 d0

优点 Fast transfer speeds, up to 320 MB/s Reliable, durable components Can connect many devices with a single bus, more than just HDs (Printers, Scanners,…) SCSI host cards can be put in almost any system Full backwards compatibility SCSI的优缺点 • 缺点 • Configuration and setup specific to one computer • Unlike IDE, few BIOS support the standard • Overwhelming number of variations in the standard, hardware, and connectors • No common software interfaces and protocol

IDE/ATA与SCSI的比较 Feature IDE/ATA SCSI Internal and External Storage Connectivity Market Internal Storage 320 Speed (MB/sec) 100/133/150 Yes Hot Pluggable No Very good but veryexpensive to set up Expandability Easier to set up High cost/Fasttransfer speed Cost/Performance Good

2.2 连接（Connectivity） • 物理部件：Host with External Storage Port Bus CPU Host HBA Cable • SCSI • Fibre Port Disk

Host Apps DBMS Mgmt Utils File System LVM Multipathing Software Device Drivers HBA HBA HBA Fibre Channel Storage Arrays 光纤通道 (Fibre Channel) Hardware Components Storage Protocol

SCSI和FibreChannel的比较 • SCSI • Limited distance • Limited device count • Usually limited to single initiator • Single-ported drives • Fibre Channel • Greater distance • High device count in SANs • Multiple initiators • Dual-ported drives

采用FibreChannel连接存储设备 • Fibre Channel相关的物理设备 • HBA • Optical cables • Switch/Director/Bridge Hosts Storage Switches

小结 • 物理部件 • Internal/External • Bus/Cable/Port • 逻辑部件：通信协议 • PCI • IDE/ATA • SCSI • Fibre Channel

2.3 物理磁盘（Physical Disks） • 磁盘的物理组成（physical components）及其功能 • 磁盘的逻辑构成（logical constructs） • 磁盘的访问特征及其性能 • 磁盘的逻辑分区（logical partitioning）

磁盘的物理组成：盘片（Platters ） 01010100111010101010 00110100111010101010 00110100111010101010 10110101011010101010 In HAD (Head Disk Assembly)

磁盘的物理组成：轴（Spindle ） Spindle Platters The spindle rotates continuously driven by a motor.

磁盘的物理组成：读写头（Read/Write Head） • 几个概念： • flying height • landing zone • head crash Reading and writing data is a magnetic process.

磁盘的物理组成：读写头激励器（Actuator） Spindle Actuator

磁盘的结构：Actuator Arm Assembly R/W Head The read/write heads move across the platter simultaneously. R/W Head Actuator

磁盘的物理组成：控制器（Controller） • Controller, a printed circuit board • contains a microprocessor (as well as some internal memory, circuitry, and firmware) • controls power, communication, positioning, and optimization Controller Interface HDA Power Connector A bottom view of disk drive

第二章存储系统架构 Storage Systems Architecture