Cosequential Processing and Sorting of Large Files

1. 평균: 40.75862

2. Chap 8. Cosequential Processing and the Sorting of Large Files

3. Chapter Objectives(1) • Describe a class of frequently used processing activities known as cosequential process • Provide a general object-oriented model for implementing varieties of cosequential processes • Illustrate the use of the model to solve a number of different kinds of cosequential processing problems, including problems other than simple merges and matches • Introduce heapsortas an approach to overlapping I/O with sorting in RAM

4. Chapter Objectives(2) • Show how merging provides the basis for sorting very large files • Examine the costs of K-way merges on disk and find ways to reduce those costs • Introduce the notion of replacement selection • Examine some of the fundamental concerns associated with sorting large files using tapes rather than disks • Introduce UNIX utilities for sorting, merging, and cosequential processing

5. Contents 8.1 Cosequential operations 8.2 Application of the OO Model to a General Ledger Program 8.3 Extension of the OO Model to Include Multiway Merging 8.4 A Second Look at Sorting in Memory 8.5 Merging as a Way of Sorting Large Files on Disk 8.6 Sorting Files on Tape 8.7 Sort-Merge Packages 8.8 Sorting and Cosequential Processing in Unix

6. Cosequential operations • Coordinated processing of two or more sequential lists to produce a single list • Kinds of operations • merging, or union • matching, or intersection • combination of above

7. Matching Names in Two Lists(1) • So called “intersection operation” • Output the names common to two lists • Things that must be dealt with to make match procedure work reasonably • initializing that is to arrange things • methods that are getting and accessing the next list item • synchronizing between two lists • handling EOF conditions • recognizing errors e.g. duplicate names or names out of sequence

8. Matching Names in Two Lists(2) • In comparing two names • if Item(1) is less than Item(2), read the next from List 1 • if Item(1) is greater than Item(2), read the next name from List 2 • if the names are the same, output the name and read the next names from the two lists

9. Cosequential match procedure(1) PROGRAM: match Item(1) Item(1) < Item(2) List 1 same name use input() & initialize() procedure List 2 Item(1) > Item(2) Item(2)

10. Cosequential match procedure(2) int Match(char * List1, char List2, char *OutputList) { int MoreItems; // true if items remain in both of the lists // initialize input and output lists InitializeList(1, List1); InitializeList(2, List2); InitializeOutput(OutputList); // get first item from both lists MoreItems = NextItemInLIst(1) && NextItemInList(2); while (MoreItems) { // loop until no items in one of the lists if(Item(1) < Item(2) ) MoreItems = NextItemInList(1); else if (Item(1) == Item (2) ) { ProcessItem(1); // match found MoreItems = NextItemInList(1) && NextItemInList(2); } else MoreItems = NextItemInList(2); // Item(1) > Item(2) } FinishUp(); return 1; }

11. General Class for Cosequential Processing(1) template <class ItemType> class CosequentialProcess // base class for cosequential processing { public: // the following methods provide basic list processing // these must be defined in subclasses virtual int InitializeList (int ListNumber, char *LintName) = 0; virtual int InitializeOutput (char * OutputListName) = 0; virtual int NextItemInList (int ListNumber) = 0; // advance to next item in this list virtual ItemType Item(int ListNumber) = 0; // return current item from this list virtual int ProcessItem(int ListNumber) = 0; // process the item in this list virtual int FinishUp() = 0; // complete the processing // 2-way cosequential match method virtual int Match2Lists (char *List1, char * List2, char *OutputList); };

12. General Class for Cosequential Processing(2) • A Subclass to support lists that are files of strings, one per line class StringListProcess : public CosequentialProcess<String &> { public: StringListProcess (int NumberOfLists); // constructor // Basic list processing methods int InitializeList (int ListNumber, char * List1); int InitializeOutput(char * OutputList); int NextItemInList (int ListNumber); // get next String & Item (int ListNumber); // return current int ProcessItem (int ListNumber); // process the item int FinishUp(); // complete the processing protected: ifstream * List; // array of list files String * Items; // array of current Item from each list ofstream OutputLsit; static const char * LowValue; //used so that NextItemInList() doesn’t // have to get the first item in an special way static const char * HighValue; };

13. General Class for Cosequential Processing(3) • Appendix H: full implementation • An example of main #include “coseq.h” int main() { StringListProcess ListProcess(2); // process with 2 lists ListProces.Match2Lists (“list1.txt”, “list2.txt”, “match.txt”); }

14. Merging Two Lists(1) • Based on matching operation • Difference • must read each of the lists completely • must change MoreItems behavior • keep this flag set to true as long as there are records in either list • HighValue • the special value (we use “\xFF”) • come after all legal input values in the files to ensure both input files are read to completion

15. Merging Two Lists(2) • Cosequential merge procedure based on a single loop • This method has been added to class CosequentialProcess • No modifications are required to class StringListProcess template <class ItemType> int CosequentialProcess<ItemType> :: Merge2Lists (char * List1Name, char * List2Name, char * OutputList) { int MoreItems1, MoreItems2; // true if more items in list InitializeList (1 List1Name); InitializeList (2, List2Name); InitializeOutput (OutputListName); MoreItems1 = NextItemInLIst(1); MoreItems2 = NextItemInList(2); while(MoreItems1 || MoreItems2) { (continued … )

16. Merging Two Lists(3) while (MoreItems(1) || MoreItems(2) ) { // if either file has more if (Item(1) < Item(2)) { // list 1 has next item to be processed ProcessItem(1); MoreItem1 = NextItemInList(1); } else if (Item(1) == Item(2) ) { ProcessItem(1); MoreItems1 = NextItemInList(1); MoreItems2 = NextItemInList(2); } else // Item(1) > Item(2) { ProcessItem(2); MoreItem2 = NextItemInList(2); } } FinishUp(); return 1; }

17. (Item(1) < Item(2) )or match NAME_1 List 1 OutputList NAME_2 List 2 Item(1) > Item(2) Cosequential merge procedure(1) PROGRAM: merge

18. Summary of the CosequentialProcessing Model(1) • Assumptions • two or more input files are processed in a parallel fashion • each file is sorted • in some cases, there must exist a high key value or a low key • records are processed in a logical sorted order • for each file, there is only one current record • records should be manipulated only in internal memory

19. Summary of the Cosequential Processing Model(2) • Essential Components • initialization -reads from first logical records • one main synchronization loop • - continues as long as relevant records remain • selection in main synchronization loop • Input files & Output files are sequence checked by comparing the previous item value with new one if (Item(1) > Item(2) then .......... else if ( Item(1) < Item(2)) then ......... else ........... /* current keys equal */ endif

20. Summary of the Cosequential Processing Model(3) • Essential components (cont’d) • substitute high values for actual key when EOF • main loop terminates when high values have occurred for • all relevant input files • no special code to deal with EOF • I/O or error detection are to be relegated to supporting method so the details of these activities do not obscure the principal processing logic

21. 8.2 The General Ledger Program (1) • Figure 8.7 Sample journal entries • Acct.No. Check No. Date Description Debit/credit • 101 1271 04/02/97 Auto expense -78.70 • 1271 04/02/97 Tune-up and minor repair 78.70 • 1272 04/02/97 Rent -500.00 • 1272 04/02/97 Rent for April 500.00 • 1273 04/04/97 Advertising -87.50 • 1273 04/04/97 Newspaper ad re:new product 87.50 • 670 04/02/97 Office expense -32.78 • 670 04/02/97 Printer cartridge 32.78 • 1274 04/02/97 Auto expense -31.83 • 510 1274 04/09/97 Oil change 31.83

22. 8.2 The General Ledger Program (2) • Figure 8.8 Sample ledger printout showing the effect of posting from the journal • Checking account #1 • 1271 04/02/97 Auto expense -78.70 • 1272 04/02/97 Rent -500.00 • 1273 04/04/97 Advertising -87.50 • 1274 04/02/97 Auto expense -31.83 Prev.bal:5219.23 New bal: 4521.20 • Checking account #2670 04/02/97 Office expense -32.78 Prev.bal:1321.20 New bal: 1288.42 • Advertising expense1273 04/04/97 Newspaper ad re:new product 87.50 Prev.bal:25.00 New bal: 112.50 • Auto expenses • 04/02/97 Tune-up and minor repair 78.70 • 1274 04/09/97 Oil change 31.83 Prev.bal:501.12 New bal: 611.65 이전 잔액 현재 잔액

23. 8.2 The General Ledger Program (3) • Figure 8.9 List of journal transactions sorted by account number • Acct.No. Check No. Date Description Debit/credit • 1271 04/02/97 Auto expense -78.70 • 101 1272 04/02/97 Rent -500.00 • 101 1273 04/04/97 Advertising -87.50 • 101 1274 04/02/97 Auto expense -31.83 • 102 670 04/02/97 Office expense -32.78 • 505 1273 04/04/97 Newspaper ad re:new product 87.50 • 510 1271 04/02/97 Tune-up and minor repair 78.70 • 510 1274 04/09/97 Oil change 31.83 • 540 670 04/02/97 Printer cartridge 32.78 • 550 1272 04/02/97 Rent for April 500.00

24. 8.2 The General Ledger Program (4) • Figure 8.10 Conceptual view of cosequential matching of the ledger and journal files • Ledger List Journal List • Checking account #1 101 1271 Auto expense 101 1272 Rent 101 1273 Advertising 101 1274 Auto expense • Checking account #2 102 670 Office expense • Advertising expense 505 1273 Newspaper ad re: new product • 510 Auto expenses 510 1271 Tune-up and minor repair 510 1274 Oil change

25. 8.2 The General Ledger Program (5) • Figure 8.11 Sample ledger printout for the first six accounts • Checking account #1 • 1271 04/02/97 Auto expense -78.70 • 04/02/97 Rent -500.00 • 1274 04/02/97 Auto expense -31.83 • 1273 04/04/97 Advertising -87.50 • Prev.bal:5219.23 New bal: 4521.20 • Checking account #2670 04/02/97 Office expense -32.78 Prev.bal:1321.20 New bal: 1288.42 • Advertising expense1273 04/04/97 Newspaper ad re:new product 87.50 Prev.bal:25.00 New bal: 112.50 • Auto expenses • 04/02/97 Tune-up and minor repair 78.70 • 1274 04/09/97 Oil change 31.83 Prev.bal:501.12 New bal: 611.65 • Bank charges Prev.bal: 0.00 New bal: 0.00 • 520 Books and publications Prev.bal: 87.40 New bal: 87.40

26. 8.2 The General Ledger Program(6) • The ledger (master) account number • The journal (transaction) account number • Class MasterTransactionProcess (Fig 8.12) • Subclass LedgeProcess (Fig 8.14)

27. 8.2 The General Ledger Program (7) Template <class ItemType> class MasterTransactionProcess: Public CosequentialProcess<ItemType> // a cosequential process that supports master/transaction processing {public: MasterTransactionProcess(); // constructor Virtual int ProcessNewMaster() = 0; //processing when new master read Virtual int ProcessCurrentMaster() = 0; Virtual int ProcessEndMaster() = 0; Virtual int ProcessTransactionError()= 0; //cosequential processing of master and transaction records int PostTransactions (char * MasterFileName, char * TransactionFileName, char * OutputListName); };

28. 8.2 The General Ledger Program (8) while (MoreMasters || MoreTransactions) if (Item(1) < Item(2)) { // 이 마스터 레코드를 끝낸다 ProcessEndMaster(); MoreMasters = NextItemInList(1); if (MoreMasters) ProcessNewMaster(); } else if (Item(1) == Item(2)) { // 마스터와 일치하는 트랜잭션 ProcessCurrentMaster(); // 마스터를 위한 또다른 트랜잭션 ProcessItem(2); // 트랜잭션 레코드 출력 MoreTransactions = NextItemInList(2); } else { // Item(1) > Item(2) 마스터를 갖지 않는 트랜잭션 ProcessTransactionError(); MoreTransactions = NextItemInList(2); }

29. 8.2 The General Ledger Program (9) int LedgerProcess::ProcessNewMaster() { // 헤더를 프린트하고 마지막 달의 잔액을 설정 ledger.PrintHeader(OutputList); ledger.Balances[MonthNumber] = ledger.Balances[MonthNumber-1]; } int LedgerProcess::ProcessCurrentMaster() { // 이 달의 잔액에 트랜잭션의 양을 더한다 ledger.Balances[MonthNumber] += journal.Amount; } int LedgerProcess::ProcessEndMaster() { // 잔액을 출력한다 PrintBalances(OutputList, ledger.Balances[MonthNumber-1], ledger.Balances[MonthNumber]); }

30. 8.3 A K-way Merge Algorithm • A very general form of cosequential file processing • Merge K input lists to create a single, sequentially ordered output list • Algorithm • begin loop • determine which list has the key with the lowest value • output that key • move ahead one key in that list • in duplicate input entries, move ahead in each list • loop again

31. 8.3 Selection Tree for Merging Large Number of Lists • K-way merge • nice if K is no larger than 8 or so • if K > 8, the set of comparisons for minimum key is expensive • loop of comparison (computing) • Selection Tree (if K > 8) • time vs. space trade off • a kind of “tournament” tree • the minimum value is at root node • the depth of tree is log2 K

32. 8.3 Selection Tree 7, 10, 17....List 0 7 9, 19, 23....List 1 7 11, 13, 32....List 2 11 18, 22, 24....List 3 input 5 12, 14, 21....List 4 5 5, 6, 25....List 5 5 15, 20, 30....List 6 8 8, 16, 29....List 7

33. 8.4 A Second Look at Sorting in Memory • Read the whole file from into memory, perform sorting, write the whole file into disk • Can we improve on the time that it takes for this RAM sort? • perform some of parts in parallel • selection sort is good but cannot be used to sort entire file • Using Heap technique! • processing and I/O can occur in parallel • keep all the keys in heap • Heap building while reading a block • Heap rebuilding while writing a block

34. 8.4 Overlapping processing and I/O : Heapsort • Heap • a kind of binary tree, complete binary tree • each node has a single key, that key is less than or equal to the key at its parent node • storage for tree can be allocated sequentially • so there is no need for pointers or other dynamic overhead for maintaining the heap

35. 8.4 A heap in both its tree form and as it would be stored in an array A (1) * n, 2n, 2n+1 positions B c (2) (3) E H I D (4) (5) (6) (7) F (9) G (8) 1 2 3 4 5 6 7 8 9 A B C E H I D G F

36. 8.4 Class Heap and Method Insert(1) class Heap { public: Heap(int maxElements); int Insert (char * newKey); char * Remove(); protected: int MaxElements; int NumElements; char ** HeapArray; void Exchange (int i, int j); // exchange element i and j int Compare (int i, int j) // compare element i and j { return strcmp(Heaparray[i], HeapArray[j]); } };

37. 8.4 Class Heap and Method Insert(2) int Heap::Insert(char * newKey) { if (NumElements == MaxElements) return FALSE; NumElements++; // add the new key at the last position HeapAray[NumElements] = newKey; // re-order the heap int k = NumElements; int parent; while(k > 1) { // k has a parent parent = k/2; if (Compare(k, parent) >= 0) break; // HeapArray[k] is in the right place // else exchange k and parent Exchange(k, parent); k = parent; } return; }

38. Heap Building Algorithm(1) input key order : F D C G H I B E A New key to be inserted Heap, after insertion of the new key Selected heaps in tree form F 1 2 3 4 5 6 7 8 9 F D 1 2 3 4 5 6 7 8 9 D F C C 1 2 3 4 5 6 7 8 9 C F D D F G 1 2 3 4 5 6 7 8 9 C F D G H 1 2 3 4 5 6 7 8 9 C F D G H (continued....)

39. Heap Building Algorithm(2) input key order : F D C G H B E A New key to be inserted Heap, after insertion of the new key Selected heaps in tree form I 1 2 3 4 5 6 7 8 9 C F D G H I C F D B 1 2 3 4 5 6 7 8 9 B F C G H I D G H I E 1 2 3 4 5 6 7 8 9 B E C F H I D G B C F A 1 2 3 4 5 6 7 8 9 A B C E H I D G F H G I D (continued....)

40. Heap Building Algorithm(3) input key order : F D C G H B E A Heap, after insertion of the new key New key to be inserted Selected heaps in tree form A 1 2 3 4 5 6 7 8 9 A B C E H I D G F A C B D H I E G F

41. Illustration for overlapping input with heap building(1) (Free ride of main memory processing: heap building is faster than IO!) Total RAM area allocated for heap First input buffer. First part of heap is built here. The first record is added to the heap, then the second record is added, and so forth Second input buffer. This buffer is being filled while heap is being built in first buffer.

42. Illustration for overlapping input with heap building(2) (One Heap is growing during IO time!) Second part of heap is built here. The first record is added to the heap, then the second record, etc Third input buffer. This buffer is filled while heap is being built in second buffer Third part of heap is built here Fourth input buffer is filled while heap is being built in third buffer

43. 8.4 Sorting while Writing to the File • Heap rebuilding while writing a block (Free ride of main memory processing) • Retrieving the keys in order (Fig 8.20) • while( there is no elements) • get the smallest value • put largest value into root • decrease the # of elements • reorder the heap • Overlapping retrieve-in-order with I/O • retrieve-in-order a block of records • while writing this block, retrieve-in-order the next block

44. 8.4 Method Remove char * Heap::Remove() { // 가장 작은 원소를 제거하고, 힙을 재정렬시키고, 가장 작은 원소를 리턴한다 // 리턴하기 위해 가장 작은 값을 val에 넣는다 char * val = HeapArray[1]; HeapArray[1] = HapArray[NumElements]; // 루트에 가장 큰 값을 넣는다 NumElements--; // 원소의 수를 감소시킨다 // 교환과 하향이동에 의해 힙을 재정렬한다 int k = 1; // 가장 큰 값을 포함하는 힙의 노드 int newK; // 가장 큰 값으로 교환되는 힙의 노드 while (2*k <= NumElements) { // k는 최소 하나의 자식을 가짐 // k의 가장 작은 자식 인덱스로 newK를 설정 if (Comapre(2*k, 2*k+1) > 0) newK = 2*k; else newK = 2*k+1; // k와 newK가 순서대로 되어 있는지를 비교 if (Compare(k, newK) < 0) break; // 순서대로 되어 있는 경우 Exchange(k, newK); // k와 newK가 순서대로 되지 않은 경우 k = newK; //트리의 아래 방향으로 진행} return val; } Figure 8.20

45. 8.5 Merging as a Way of Sorting Large Files on Disk • Keysort: holding keys in memory • Two Shortcomings of Keysort • substantial cost of seeking may happen after keysort • cannot sort really large files • e.g. a file with 800,000 records, size of each record: 100 bytes, • size of key part: 10 bytes, then 800,000 X 10 => 8G bytes! • cannot even sort all the keys in RAM • Multiway merge algorithm • small overhead for maintaining pointers, temporary variables • run: sorted subfile • using heap sort for each run • split, read-in, heap sort, write-back

46. 800,000 unsorted records 80 internal sorts ............. 80runs, each containing 10,000 sorted records ............. Merge 800,000 records in sorted order Sorting through the creation of runs and subsequential merging of runs

47. Multiway merging (K-way merge-sort) • Can be extended to files of any size • Reading during run creation is sequential • no seeking due to sequential reading • Reading & writing is sequential • Sort each run: Overlapping I/O using heapsort • K-way merges with k runs • Since I/O is largely sequential, tapes can be used

48. How Much Time Does a Merge Sort Take? • Assumptions • only one seek is required for any sequential access • only one rotational delay is required per access • Four I/Os • during the sort phase • reading all records into RAM for sorting, forming runs • writing sorted runs out to disk • during the merge phase • reading sorted runs into RAM for merging • writing sorted file out to disk

49. Four Steps(1) • Step1: Reading records into RAM for sorting and forming runs • assume: 10MB input buffer, 800MB file size • seek time --> 8msec, rotational delay --> 3msec • transmission rate --> 0.0145MB/msec • Time for step1: • access 80 blocks (80 X 11)msec + transfer 80 blocks (800/0.0145)msec • Step2: Writing sorted runs out to disk • writing is reverse of reading • time that it takes for step2 equals to time of step1

50. Four Steps(2) • Step3: Reading sorted runs into RAM for merging • 10 MB of RAM is for storing runs. 80 runs • reallocate each of 80 buffers 10MB RAM as 80 input buffers • access each run 80 buffers to read all of it • Each buffer holds 1/80 of a run (0.125MB) • total seek & rotational time --> 80 runs X 80 seeks --> 6400 seeks. 6400 X 11 msec = 70 seconds • transfer time --> 60 seconds (800MB  0.0145MB/msec) • total time = total seek & rotation time + transfer time