Perl Loops and Adding Disks: Run Instructions and Check Conditions

1. Chapter 8 Perl Loops and Adding Disks

2. Loops • Run instructions again, and again, and again…. • Check some condition • If true, repeat • 4 types of loops (we will look at 2 of them) • for loop • while loop • First, the variable used in loop condition must be initialized, then execution of the loop begins. • A test (condition) is made at the beginning of each iteration. • The body of loop ends with a statement that modifies the value of the test (condition) variable.

3. For Loop • Syntax: for ( init; condition; increment ) { statement(s); } • Example #!/usr/bin/perl # for loop execution for( $a = 10; $a < 20; $a = $a + 1 ) { print "value of a: $a\n"; }

4. While Loop • Syntax: while(condition) { statement(s); } #!/usr/bin/perl $a = 10; # while loop execution while( $a < 20 ) { print "Value of a: $a\n"; $a = $a + 1; }

5. Foreach Loop • The foreach loop iterates over a list value and sets the control variable (var) to be each element of the list in turn: • Syntax: • The syntax of a foreach loop in Perl programming language is: • foreachvar (list) { ... } #!/usr/local/bin/perl @list = (2, 20, 30, 40, 50); # foreach loop execution foreach $a (@list) { print "value of a: $a\n"; }

6. Storage Hardware • Hard Disks • Flash Memory • SSD • Hybrids • Magnetic Tapes • Optical Media

8. Hard Disk Failure • A 2007 Google Labs study of 100,000 drives • Drives 2+ years old average annual failure rate (AFR) of more than 6%! • The overall pattern was a few months of infant mortality, a two-year honeymoon of annual failure rates of a few percent, and then a jump up to the 6%–8% AFR range. • Overall, hard disks in the Google study had less than a 75% chance of surviving a five-year tour of duty.

9. Disk Geometry • Platter - Stacked circular plates with magnetic coating. • Sector – Smallest unit of storage on the disk which the system can access, and is typically 512 bytes. • Track – Series of sectors, on the same physical platter of a disk, all of which can be accessed without moving the disk's read/write heads. • Cylinder - A cylinder is the set of tracks, each of which is on a different platter and all of which can be accessed without moving the disk's read/write heads. • logical partitions (or slices) - start cylinder of each partition plus the number of cylinders it contains.

10. Disk Geometry Platters rotate inside hard disk around spindle Disk head reads and writes data to sectors (similar to needle and record player) Faster the platter spins (rotation) the higher the bandwidth can be accessed from disk. Disk speeds 5,400 RPMs – 15,000 RPMs

11. Solid State Drives • Flash memory – capable of parallel read and write(s) • Using SATA 6Gb/s interface Typical internal laptop SSD 415MB/s (reads) 250MB/s (writes) • Long term reliability ????? - YES • $$$$ vs hard drives – prices coming down

12. Disks Interfaces • IDE/ATA (Advanced Technology Attachment) PATA – Parallel ATA • SATA - Serial ATA • SCSI – “skuzzy” • Fibre Channel 1-40 Gb/s • USB – Universal Serial Bus • Thunderbolt

13. PATA aka IDE • Parallel ATA commonly known as IDE • Traditional interface found in lower end workstations / PCs (becoming deprecated) • Transfer rates up to 133 MB/s • 18in. cable length

14. SATA • Serial ATA - SATA1,2,3 • 6 Gb/s • 1 meter cable length • SATA3 backwards compatible to SATA1 • Commonly found in today’s workstations and PCs

15. Small Computer System Interface (SCSI) • Generic data pipe used by different types of devices (similar to USB) • Daisy chain drives together • Hot swappable • SCSI-3 defines a family of standards including Firewire and Fibre Channel

16. Universal Serial Bus - USB • USB 1.0, 2.0, 3.1 • 3.1 transfer rates up to • 10 Gbps (max theoretical) • Data pipe used by different peripherals

17. Thunderbolt 3 • Formally known as LightPeak - developed by Intel • Transfer rates up to 40 Gbps

18. The Software Side of Storage • Partition – fixed sub-sized section of a storage device • RAID - a redundant array of inexpensive/independent disks – combines multiple drives into a virtual disk. (more on RAID later) • Volume groups and logical volumes - These systems aggregate physical devices to form pools of storage called volume groups. Then subdivide this pool into logical volumes.

19. Linux Filesystem • Check filesystem “df –T” command • Ext2: This is like UNIX file system. It has the concepts of blocks, inodes and directories. • Ext3: It is ext2 filesystem enhanced with journalling capabilities. Journalling allows fast file system recovery. • Ext4: Supports huge individual file size and overall file system size. Maximum individual file size can be from 16 GB to 16 TB Overall maximum ext4 file system size is 1 EB (exabyte). 1 EB = 1024 PB (petabyte). 1 PB = 1024 TB (terabyte). Directory can contain a maximum of 64,000 subdirectories (as opposed to 32,000 in ext3) • XFS - supports a maximum file system size of 8 exbibytes (10^18) bytes

20. The Software Side of Storage

21. Storage Details • Partitions – fixed size sub section of a storage device • RAID array (redundant array of inexpensive disks) – combines multiple storage devices into one virtualized device • Volume groups and logical volumes associated with logical volume managers (LVM) - physical devices for pools of storage called volume groups – volume groups can then be subdivided into logical volumes

22. The Software Side of Storage cont’ • Filesystem - mediates between the raw bag of blocks presented by a partition, RAID array, or logical volume and the standard filesystem interface expected by programs: paths such as /var/spool/mail, Linux file types, Linux permissions, etc

23. Adding a Hard Drive to your system • Physically connect the drive to the PC, system, RAID array, etc • Hardware verification • The BIOS of the PC can tell you if its connected correctly • System log files • Files located in /dev will update with the new drive information automatically • Linux block device /dev/sdaPartition /dev/sda1 • USB are /dev/sdb • Run command parted /dev/sdb print

24. Time to put on filesystem • mkfs.ext4 Formats a disk to ext4 file system • mkdir <new mountpoint> • Mount <new filesystem drives> <new mountpoint>

26. LVM • Logical Volume Management • Essentially abstract dynamic disk partitioning. • Groups individual storage devices into “volume groups.” • The blocks in a volume group can then be allocated to “logical volumes,” which are represented by block device files and act like disk partitions.

27. Linux LVM • A Linux LVM configuration proceeds in a few distinct phases: • Creating (defining, really) and initializing physical volumes • Adding the physical volumes to a volume group • Creating logical volumes on the volume group

28. LVM Example • https://www.thegeekdiary.com/redhat-centos-a-beginners-guide-to-lvm-logical-volume-manager/

29. Format & Bad Block Management • Low level format – identifies bad blocks, imperfections in the media that result in areas that cannot be reliably read or written. • New drives come preformatted. • No need to low level format new drives. • Rm –rf does not remove the contents of a drive.

30. hdparm • general way to interact with the firmware of SATA, IDE, and SAS hard disks. • Among other things, hdparm can set drive power options, enable or disable noise • hdparm [options] device

31. Disk Partitioning • There are several options for partitioning a hard disk: • fdisk • Parted (****) • gparted (GUI)

32. RAID • Redundant Array of Inexpensive Disks • Best handled in software by the kernel (although dedicated hardware is available) • RAID has two features – striping and replication • Striping – spreads data out across multiple disks • Replication: • Mirroring – byte for byte copy of the data across one or more disks • Parity – error correcting checksums of data blocks spread across multiple drives

33. RAID Levels • Linear mode - concatenates the block addresses of multiple drives to create a single, larger virtual drive. (no redundancy or performance improvement) • RAID 0 – used strictly to increase performance stripes data across multiple disks

34. RAID Level 1 • RAID 1 • Mirroring – write functions are duplicated across two or more drives simultaneously

35. RAID Level 1 + 0 & 0 + 1 • RAID levels 1+0 and 0+1 are stripes of mirror sets or mirrors of stripe sets. • The goal of both modes is to simultaneously obtain the performance of RAID 0 and the redundancy of RAID 1.

36. RAID Level 5 • RAID 5 - stripes both data and parity information, adding redundancy while simultaneously improving read performance. • RAID 5 is more efficient in its use of disk space than is RAID 1. • If there are N drives in an array (at least three are required), N–1 of them can store data.

37. RAID Levels 6 • RAID 6 is similar to RAID 5 with two parity disks. A RAID 6 array can withstand the complete failure of two drives without losing data.

38. Lets just quickly see how to add a disk • Attach hard drive and reboot • No RAID (will cover later) • One logical volume (will cover later) • Identify and format the right drive • Linux recipe • Type “sudo fdisk –l” to list the system disks and identify a new drive • Use a partitioning utility to create partition table • Put all drives spare in one partition of unspecified type.

39. Lets just quickly see how to add a disk cont’ • Sudo pvcreate /dev/sdc1 • Sudo vgcreatevgname /dev/sdc1 • Sudo lvcreate –l 100%FREE –n volnamevgname • Sudo mkfs –t ext4 /dev/vgname/volname • Sudo mkdir mountpoint • Sudo vi /etc/fstab (device to be mounted is /dev/vgname/volname) • Sudo mount mountpoint