Introduction to Data Structure

Introduction to Data Structure CHAPTER 7 SORTING 7.1 Searching and List Verification 7.2 Definitions 7.3 Insertion Sort 7.4 Quick Sort 7.5 Optimal Sorting Time 7.6 Merge Sort 7.7 Heap Sort 7.8 Radix Sort Chapter 7 Sorting

Contents Chapter 1 Basic Concepts Chapter 2 Arrays Chapter 3 Stacks and Queues Chapter 4 Linked Lists Chapter 5 Trees Chapter 6 Graph Chapter 7Sorting Chapter 8 Hashing Chapter 9 Heap Structures Chapter 10 Search Structures Chapter 7 Sorting

7.1 Searching and List Verification • Motivation of Sorting • The term list here is a collection of records. • Each record has one or more fields. • Each record has a key to distinguish one record with another. • For example, the phone directory is a list. Name, phone number, and even address can be the key, depending on the application or need. Chapter 7 Sorting

Sequential Searching • Two ways to store a collection of records • Sequential • Non-sequential • Assume a sequential list f. To retrieve a record with key f[i].key from such a list, we can do search in the following order: f[n].key, f[n-1].key, …, f[1].key => sequential search Chapter 7 Sorting

Sequential Searching (cont.) int SeqSearch (int list[], int searchnum, int n) { int i; list[n] = searchnum; for (i = 0; list[i] != searchnum; i++); return (i < n) ? i: -1); } Program 7.1, p. 321 • The average number of comparisons for a successful search is Chapter 7 Sorting

Binary Search • Basic concept Compare searchnum and list[middle].key • searchnum < list[middle].key • search list[0] ~ list[middle-1] • searchnum = list[middle].key • return TRUE • searchnum > list[middle].key • search list[middle+1] ~ list[n-1] • Program 7.2, p. 322 • Complexity • A binary search only takes O(log n) time to search a sequential list with n records. Chapter 7 Sorting

List Verification • Definition • Given two lists, verification if both are the same. list1 = list2? • Example: Tax verification • The IRS gets salary reports from employers. • IRS also gets tax filing reports from employees about their salary. • Need to verify the two numbers match for each individual employee. • Two methods • Method 1: Random verification • Method 2: Ordered verficiation Chapter 7 Sorting

Random Verification: Unsorted Lists void verify1(element list1[], element list2[], int n, int m) /* Compare two unordered lists list1 and list2 */ { int i, j; int marked [MAX_SIZE]; for (i = 0; i < m; i++) marked[i] = FALSE; for (i = 0; i< n; i++) if ((j == seqsearch(list2, m, list1[i].key()) < 0) printf(“%d is not in list 2\n”, list1[i].key); else /* check each of the other fields from list1[i] and list2[j], and print out any discrepancies */ marked[j] = TRUE; for (i = 1; i < m; i++) if (!marked[i]) printf(“%d is not in list 1\n”, list2[i].key); } Complexity: O(mn) Why? Chapter 7 Sorting

Sorted Verifying: Sorted Lists void verify2(element list1[], element list2[], int n, int m) { int i, j; sort(list1, n); sort(list2, m); i = j = 0; while (i < n && j < m) if (list1[i].key < list2[j].key) { printf(“%d is not in list 2\n”, list1[i].key); i++; } else if (list1[i].key == list2[j].key) { /* compare list1[i] and list2[j] on each of the other fields and report any discrepancies */ i++; j++; } else { printf(“%d is not in list 1\n”, list2[j].key); j++; } for (; i < n; i++) printf(“%d is not in list 2\n”, list1[i].key; for (; j < m; i++) printf(“%d is not in list 2\n”, list1[i].key; } Complexity: O(max{n log n, m log m}) Chapter 7 Sorting

7.2 Definition • Formal definition • Given a list of records (R0, R1, …, Rn-1), each with a key Ki. • The sorting problem is to find permutation, σ, such that Kσ(i)≤Kσ(i+1) , 1 ≤i≤n– 1. • The desired ordering is (Rσ(1), Rσ(2), Rσ(n)). • If a list has several key values that are identical, the permutation, σs, is not unique. • Let σs be the permutation of the following properties: (1) [sorted] Kσ(i)≤Kσ(i+1) , 1 ≤i≤n– 1 (2) [stable] If i < j and Ki == Kj in the input list, then Ri precedes Rj in the sorted list. • The above sorting method that generates σs is stable. Chapter 7 Sorting

0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 2 3 4241 5 7 2 3 4142 5 7 3 2 41 7 42 5 Key Key Key b a c e f d b a e c f d a b c d e f Record Record Record Stable Sorting • Example • Stable permutation • Unstable permutation 0 1 2 3 4 5  1 0 2 5 3 4 0 1 2 3 4 5  1 0 3 5 2 4 Chapter 7 Sorting

Category of Sorting Methods • Internal method • Methods to be used when the list to be sorted is small enough so that the entire sort list can be carried out in the main memory. • Example • Insertion sort • Quick sort • Merge sort • Heap sort • Radix sort • External method • Methods to be used on larger lists Chapter 7 Sorting

sorted a1a2a3 … ak ai sorted a1a2aia3 … ak 7.3 Insertion Sort • Basic concept insert 1 3 5 7 9 4 1 3 4 5 7 9 Chapter 7 Sorting

Insertion Sort: Program • Program 7.5, p 327 void insertion_sort(element list[], int n) { int i, j; element next; for (i = 1; i < n; i++) { next = list[i]; for (j = i – 1; j >= 0 && next.key < list[j].key; j--) list[j+1] = list[j]; list[j+1] = next; } } Chapter 7 Sorting

Insertion Sort: Example • Record Ri is left out of order (LOO) iff Ri< • Example 1 • Assume n = 5 and the input key sequence is 5, 4, 3, 2, 1 Chapter 7 Sorting

Insertion Sort: Example • Example 2 • Assume n = 5 and the input key sequence is 2, 3, 4, 5, 1 O(1) O(1) O(1) O(n) Chapter 7 Sorting

Insertion Sort • Analysis • If there are k LOO records in a list, the computing time for sorting the list via insertion sort is O((k+1)n) = O(kn) • Therefore, if k << n, then insertion sort might be a good sorting choice. Chapter 7 Sorting

Kj = p K0K1 … Kj-1 Kj+1Kj+2 … Kn-1 7.4 Quick Sort • Quick sort is developed by C. A. R. Hoare. • Quick sort has the best average behavior among the sorting methods. • Basic concept • Based on the divide and conquer paradigm • Choose an element (pivot) p, i.e., p = K0 • Place p into a proper position j such that • K0 ~ Kj-1 p • Kj+1 ~ Kn-1 > p Chapter 7 Sorting

Quick Sort (cont.) • Example 25 57 48 37 12 92 86 33 (12) 25 (57 48 37 92 86 33) 12 25 (48 37 33) 57 (92 86) 12 25 (37 33) 48 57 (92 86) 12 25 33 37 48 57 (92 86) 12 25 33 37 48 57 86 92 Chapter 7 Sorting

Quick Sort (cont.) • Key mechanism: a partition method • Algorithm quicksort(list, left, right) { if (left < right) { partition(list, left, right, j); quicksort(list, left, j-1); quicksort(list, j+1, right); } } Chapter 7 Sorting

Quick Sort: Partition • Concept of partition method • Let pivot = list[left] be the pivot • Use two pointers, i and j • i  until list[i]  pivot; • j  until list[j]  pivot; • list[i]  list[j] if i < j Chapter 7 Sorting

j j j i i i Quick Sort: Partition (cont.) • Example of partition method 25 57 48 37 12 92 86 33 25 12 48 37 57 92 86 33 (12) 25 (48 37 57 92 86 33) Chapter 7 Sorting

Quick Sort: Program Codes void quicksort(element list[], int left, int right) { int pivot, i, j; element temp; if (left < right) { i = left; j = right + 1; pivot = list[left].key; do { do i++; while (list[i].key < pivot); do j--; while (list[j].key > pivot); if (i < j) SWAP(list[i], list[j], temp); } while (i < j); SWAP(list[left], list[j], temp); quicksort(list, left, j–1); quicksort(list, j+1, right); } } partition Chapter 7 Sorting

Quick Sort: Example • Example • Input list: 10 records with keys (26, 5, 37, 1, 61, 11, 59, 15, 48, 19). Chapter 7 Sorting

Quick Sort: Analysis • Analysis of QuickSort • Worse case: O(n2) • Average case: • Assume each time a record is correctly positioned • |left sublist| = |right sublist| • Let T(n) be the time taken to sort a list of size n T(n) ≤ cn + 2T(n/2), for some constant c ≤ cn + 2(cn/2 +2T(n/4)) ≤ 2cn + 4T(n/4) : : ≤ cn log2n + T(1) = O(n logn) Chapter 7 Sorting

Quick Sort Variant • Quick sort using a median of three • Pick the median of the first, middle, and last keys in the current sublist as the pivot. • Thus, pivot = median{Kleft, K(left+right)/2, Kright}. Chapter 7 Sorting

7.5 Optimal Sorting Time • Question • How quickly can we sort a list of n objects? • Answer • If only operations permitted on keys are comparisons and interchanges, then O(n logn) is the best possible time. • Method • This is done by using a tree called decision tree that describes the sorting process. • Each vertex of the tree represents a key comparison, and the branches indicate the result. Chapter 7 Sorting

Decision Tree for Insertion Sort [0, 1, 2] K1 ≤ K2 No Yes [0, 1, 2] [1, 0, 2] K2 ≤ K3 K1 ≤ K3 No No Yes Yes [0, 1, 2] [1, 0, 2] [0, 2, 1] stop K2 ≤ K2 [1 , 2, 0] stop K1 ≤ K3 No Yes No IV Yes I [2, 0, 1] [1 , 2, 0] [2, 1 , 0] [0, 2, 1] stop stop stop stop V VI II III Chapter 7 Sorting

Decision Tree (cont.) • Theorem 7.1: Any decision tree that sorts n distinct elements has a height of at least log2(n!) + 1 • Corollary: Any algorithm that sorts only by comparisons must have a worst-case computing time of Ω(n log n) Chapter 7 Sorting

7.6 Merge Sort • Kernel operation of Merge Sort: Merging • Given two sorted list, merge them into a single sorted list • Example 25 37 48 57 12 33 86 92 => 12 25 33 37 48 57 86 92 • Methods for merging • Simple merge • O(1) space merge Chapter 7 Sorting

Simple Merge void merge(element list[], element sorted[], int i, int m, int n) /* merge list[i],…,list[m], and list[m+1],…,list[n] */ { int j, k, t; j = m+1; k = i; while (i<= m && j <= n) { if (list[i].key <= list[j].key) sorted[k++] = list[i++]; else sorted[k++] = list[j++]; } if (i > m) for (t = j; t <= n; t++) sorted[k+t-j] = list[t]; else for (t = i; t <= m; t++) sorted[k+t-i] = list[t]; } Time & space complexity: O(n-i + 1) Chapter 7 Sorting

O(1) Space Merge • A merge algorithm only requires O(1) additional space • Assumption • The total number of records n is a perfect square • The numbers of records in the left sublist and the right sublist are multiple of Chapter 7 Sorting

O(1) Space Merge  Algorithm Step 1: Identify the records with largest keys. This is done by following right to left along the two lists to be merged. Step 2: Exchange records of the second list identified in Step 1 with those just to the left of those identified from the first list. Step 3: Swap the block of largest with the leftmost block (unless it is already the leftmost block). Sort the rightmost block. Step 4: Reorder the blocks, excluding the block of largest records, into nondecreasing order of the last key in the blocks. Step 5: Perform as many merge substeps as needed to merge the blocks, other than the block with the largest keys. Step 6: Sort the block with the largest keys. Chapter 7 Sorting

O(1) Space Merge: Analysis • Steps 1 and 2 • O( ) time and O(1) space • Step 3 • Swapping: O( ) time and O(1) space • Sorting: O(n) time and O(1) space (via insertion sort) • Step 4 • O(n) time and O(1) space (via selection sort) • Selection sort sorts m records using O(m2) key comparisons and O(m) record moves. • O(n1.5) time and O(1) space (via insertion sort) • Insertion sort needs O(m2) record moves ( records per block * n record moves). Chapter 7 Sorting

O(1) Space Merge: Analysis (cont.) • Step 5 • Merge substeps: The total number of is at most . The total time for is O(n). • Step 6 • The sort of can be done in O(n) by using either a selection sort or an insertion sort. • In total • O(n) time and O(1) space Chapter 7 Sorting

Iterative Merge Sort • Concept • Treat the input as n sorted lists, each of length 1. • Lists are merged by pairs to obtain n/2 lists, each of size 2 (if n is odd, the one list is of length 1). • The n/2 lists are then merged by pairs, and so on until we are left with only one list. Chapter 7 Sorting

Iterative Merge Sort: Example 26 5 77 1 61 11 59 15 48 19 5 26 1 77 11 61 15 59 19 48 1 5 26 77 11 15 59 61 19 48 1 5 11 15 26 59 61 77 19 48 1 5 11 15 19 26 48 59 61 77 Chapter 7 Sorting

Iterative Merge Sort: Analysis • Program code • Program 7.9 and 7.10 • Time complexity • Total of passes are made over the data • Each pass of merge sort takes O(n) time • The total of computing time is O(n log n) Chapter 7 Sorting

Recursive Merge Sort • Concept • Divide the list to be sorted into two roughly equal parts: • left sublist [left : (left+right)/2] • right sublist [(left+right)/2 +1 : right] • Sort each sublist recursively, and merge the sorted sublists • To avoid copying, the use of a linked list (integer instead of real link) for sublist is desirable. • Program code • Program 7.11 and 7.12, complexity: O(n log n) Chapter 7 Sorting

Recursive Merge Sort: Example 26 5 77 1 61 11 59 15 48 19 5 26 11 59 19 48 5 26 77 11 15 59 19 48 1 61 1 5 26 61 77 11 15 19 48 59 1 5 11 15 19 26 48 59 61 77 Chapter 7 Sorting

Natural Merge Sort • Concept • It takes advantage of the prevailing order within the list before performing merge sort • It runs an initial pass over the data to determine the sublists of records that are in order • Then it uses the sublists for the merge sort Chapter 7 Sorting

Natural Merge Sort: Example 26 5 77 1 61 11 59 15 48 19 5 26 77 1 11 59 61 15 19 48 1 5 11 26 59 61 77 15 19 48 1 5 11 15 19 26 48 59 61 77 Chapter 7 Sorting

7.7 Heap Sort • Preliminary • Merge sort needs O(n) additional storage space, even though its computing time is O(n log n) • Merge sort using O(1) merge only needs O(1) space but the sorting algorithm is much slower • Heap sort • only requires a fixed amount of additional storage • achieves worst-case and average computing time O(n log n) Chapter 7 Sorting

Heap Sort (cont.) • Concept • Adopt the max-heap structure • Consists of two phases • Phase 1: create the heap • Insert the n records into an empty heap • Phase 2: adjust the heap • Exchange the max element with the current last element and perform adjustment Chapter 7 Sorting

Heap Sort: Program Code void heapsort (element list[], int n) { int i, j; element temp; for (i = n/2; i > 0; i--) /* Phase 1 */ adjust(list, i, n); for (i = n-1; i > 0; i--) { /* Phase 2 */ SWAP(list[1], list[i+1], temp); adjust(list, 1, i); } } Complexity: O(n log n) Chapter 7 Sorting

Heap Sort: Program Code (cont.) void adjust (element list[], int root, int n) { int child, rootkey; element temp = list[root]; rootkey = list[root].key; child = 2*root; while (child <= n) { if (child < n) && (list[child].key < list[child+1].key)) child++; if (rootkey > list[child].key) break; else { list[child/2] = list[child]; child *= 2; } } list[child/2] = temp; } Chapter 7 Sorting

Heap Sort: Example 26 77 [1] [1] [2] 5 77 61 59 [2] [3] [3] 1 61 11 59 [4] 48 19 11 26 [5] [4] [5] [6] [7] [7] [6] 15 48 19 15 1 5 [8] [9] [10] [10] [8] [9] (b) Initial heap (a) Input array Chapter 7 Sorting

Heap Sort: Example (cont.) 59 61 [1] [1] 48 26 48 59 [2] [2] [3] [3] 15 19 11 1 15 19 11 26 [4] [5] [5] [7] [7] [6] [6] 5 5 1 [8] [8] [9] Heap size = 8, Sorted = [61, 77] Heap size = 9, Sorted = [77] Chapter 7 Sorting

Introduction to Data Structure