The HP AutoRAID Hierarchical Storage System

1. The HP AutoRAID Hierarchical Storage System John Wilkes, Richard Golding, Carl Staelin, and Tim Sullivan Hewlett-Packard Laboratories

2. File System Review

3. File System Review • UNIX File System (1974) • provides an addressable structure to store and retrieve files from disk • simple & elegant but slow (2% bandwidth)

4. File System Review • UNIX File System (1974) • provides an addressable structure to store and retrieve files from disk • simple & elegant but slow (2% bandwidth) • Berkeley Fast File System (1984) • modified the block size to allow bandwidth to reach up to 47% • created cylinder groups that spread metadata to reduce seek times • considered hardware specifics during file system parameterization

5. File System Review • UNIX File System (1974) • provides an addressable structure to store and retrieve files from disk • simple & elegant but slow (2% bandwidth) • Berkeley Fast File System (1984) • modified the block size to allow bandwidth to reach up to 47% • created cylinder groups that spread metadata to reduce seek times • considered hardware specifics during file system parameterization • Sprite Log-structured File System (1991) • relies on increasingly large file caches to handle most reads • buffers multiple writes before going to disk • buffer then gets copied entirely to disk in a single write • introduced the concept of extents (large continuous set of free blocks) • requires cleaning(garbage collection) & requires restructuring of active/non-active data • improved crash recovery with roll forward capability !!!!

6. Redundancy FS X FFS X LFS X What about Hardware Failure??

7. Redundancy FS X FFS X LFS X RAIDü What about Hardware Failure??

8. RAID • Redundant array of independent disks (early 80s) • early days of mainframes • Redundant array of inexpensive disks (1988 Patterson, et al) • for smaller computer pc ( became widely popular) • introduced the concept of partial redundancy • Virtualization • Array of Disks are viewed as a Single Virtual Disk • Requires Array Controller, SCSI connector, hardware and software support Controls Array of Disks

9. the many Levels of RAID • Patterson introduced five levels • No Standards Exist • Companies are free to invent their own versions

10. STRIPING Pros Good performance on large requests 100% storage capacity Cons Not fault tolerant Not considered raid by many enthusiasts because nothing is redundant raid0

11. MIRRORING Pros Good performance And its fault tolerant Cons 50% storage capacity Gets expensive to scale raid1

12. Parity • Parity is calculated using XOR • Controller takes a bit from each disk • if the total is even  parity = 0 • If the total is odd  parity =1 • Same protection as mirroring without all the overhead • Increased capacity to 80% (1-1/n where n=disks) • Easy to restore bits to a single failed drive • For missing data, what bit makes parity correct?

13. Combine Striping and Redundancy Pros increased storage capacity (1 - 1/N)% high throughput for large files provides partial redundancy using parity Cons parity is at the bit level Poor performance for small I/O no parallel reads or writes possible because parity is on a single disk raid3

14. Spread Parity Across All Disks Pros block level striping allows hot swappable disk replacement on failure small requests could be performed in parallel Cons small writes require reading old data, writing new data, reading corresponding old parity value, and writing new parity value(small-write problem) if workload contains too many small writes performance suffers dramatically raid5

15. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes

16. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes • How Can we combine the best of both Levels?

17. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes • How Can we combine the best of both Levels? • Use raid1 for Active data and raid5 for Inactive data

18. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes • How Can we combine the best of both Levels? • Use Raid1 for Active Data and Raid 5 for Inactive data • Create a mapping that allows migration between the two

19. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes • How Can we combine the best of both Levels? • Use Raid1 for Active Data and Raid 5 for Inactive data • Create a mapping that allows migration between the two • Assign a hierarchical preference to each level

20. All these levels, how do I choose the right one? • No Level fits for all occasions • Raid1 fast but doesn’t scale well • 50% storage capacity • Raid5 scales but can’t handle multiple small writes • How Can we combine the best of both Levels? • Use Raid1 for Active Data and Raid 5 for Inactive data • Create mapping that allows migration between the two • Assign a hierarchical preference • Provide a way to migrate data between the two hierarchies

21. Who Manages the Migration? • Not the system administrator • Error prone • Can not adapt fast enough to changing environment • Not the file system • Good idea but not a portable solution • Could use an array controller if it were smart enough • It would have to • identify active and inactive data • migrate active data to mirrored storage and inactive data to raid5 storage • provide a virtual disk to the existing file system • be easy to configure

22. HP AutoRAID • Super intelligent array controller • Uses Embedded software to manage hierarchy • Presents virtual logical units to file system • The file system is unaware of • storage hierarchy • active/inactive grouping • data migration • Have to provide a mapping to go from virtual to physical addresses!

23. Data Layout – placing the data on the disk • PEX physical extent • 1MB of disk space allocation • These are the columns of data • PEG physical extent group • group of at least three PEX’s on different disks • Spread across disks to balance data • PEG States • Can be assigned to the mirrored storage class • Can be assigned to the raid5 storage class • Can be unassigned • Segment – 128KB contiguous space • Included in a stripe or mirrored pair • RB Relocation Block - 64KB • LUN logical unit Host-visible virtual disk • STRIPE row of parity & data segments in raid5 peg

24. Mapping Structure PEG TABLES PEGn LUN/ Virtual Device Tables OS File System • LUN ptrs to PEGS • PEG tables • list of RB’s • list of PEX’s • PEX tables • 1 per disk PEG2 PEG1 Disk 1 Disk 2 Disk3 Pex1 segment table Pex2 segment table Pex3 segment table

25. HP AutoRAID What can it do? • Initially array starts out empty • Data is added to mirrored storage until it is full • Some mirrored storage is immediately reallocated to raid5 storage • Just re-map PEX’s in mirrored PEG’s to RAID5 PEG’s • As workload changes • Newly active data are promoted to mirrored storage • Data that are less active are demoted to raid5 storage • All of this is done in the background - no performance interference • Hot-pluggable disks allow for failed component to be removed while system is running • Disks can be added to the array at any time up to maximum of 12 • Controller fail-over support • Active hot spare to reduce the risk of having two drive failures • Raid5 uses Log-Structured writes for added performance

26. HP AutoRAID is very Slick! • Added redundancy • Have the ability to add disks to the array on the fly • We pushed control disk control from the File System to some fancy hardware with embedded software • As far as the file system is concerned we have solved all the problems, right? • Well, not really! • RAID5 uses log-structured writes, what about the garbage collection?

27. HP AutoRAID is very Slick! • Added redundancy • Have the ability to add disks to the array on the fly • We pushed control disk control from the File System to some fancy hardware with embedded software • As far as the file system is concerned we have solved all the problems, right? • Well, not really! • RAID5 uses log-structured writes, what about the garbage collection? • Same as layout balancing, garbage collection is done in the background • This is done by identifying periods of idleness • Cleaning requires filling the Holes left when data are promoted to the mirrored storage class

28. Compaction, cleaning, hole plugging RAID5 PEG Hole-Plugging Garbage collection • If it is nearly full • RB’s from almost empty PEG’s copied to fill holes • Minimizes data movement • If it is almost empty • Those RB’s are used to fill holes in the nearly full ones • If it is almost empty and no others holes are ready to be plugged • valid RB’s are written to the end of the log • Complete PEG is reclaimed as a unit

29. Performance • OLTP macrobenchmark results • Raid redundancy • HPAutoRaid redundancy • JBOD–LVM NO redundancy • Striping though, so geared for speed • Results are as expected • Transaction rate relative to number of disks • Working set to large for 5 drives • Write set doesn’t fit entirely in mirrored storage • Thrashing causes poor performance

30. Summary • HP AutoRAID works well to provide performance and redundancy • Extremely easy to setup and use • Works in a variety of real life environments • Provides outstanding general purpose storage

31. References • Wilkes, John. et al “The HP autoraid hierarchical storage system” Hewlett-Packard Laboratories • Patterson, David A “A case for redundant arrays of inexpensive disks (RAID)” Department of Electrical Engineering UC Berkeley • Henson, Val “A Brief History of Unix File Systems” http://www.lugod.org/presentations/filesystems.pdf • Rosenblum, Mendel. et al “The design and implementation of a log-structured file system” Department of Electrical Engineering UC Berkeley • McKusick, Marshal K. et al “A fast file system for UNIX*” Department of Electrical Engineering UC Berkeley • Raid graphics from http://www.prepressure.com/techno/raid.htm • Parity graphics from http://www.commodore.ca/windows/raid5/raid5.htm#Parity • Tanenbaum, Andrew S “Modern Operating Systems 2nd Edition” Prentice-Hall of India