Fundamental File Structure Concepts

Fundamental File Structure Concepts Chapter 4

Record and Field Structure • A record is a collection of fields. • A field is used to store information about some attribute. • The question: when we write records, how do we organize the fields in the records: • so that the information can be recovered • so that we save space • so that we can process efficiently • to maximize record structure flexibility

Field Structure issues • What if • Field values vary greatly • Fields are optional

Field Delineation methods • Fixed length fields • Include length with field • Separate fields with a delimiter • Include keyword expression to identify each field

Fixed length fields • Easy to implement - use language record structures (no parsing) • Fields must be declared at maximum length needed 10 10 15 15 2 9 last first address city state zip “Yeakus Bill 123 Pine Utica OH43050 “

Include length with field • Begin field with length indicator • If maximum field length <256, a byte can be used for length last first address city state zip Length bytes Yeakus Bill 123 Pine 06 59 65 61 6B 75 73 04 42 69 6C 6C 08 31 32 33 20 50 69 6E 64 . .

Optional Fields • Fixed length • Leave blank • Field length • zero length field • Delimiter • Adjacent delimiters • Keywords • Just leave out

Reading a stream of fields • Need to break record into fields • Fixed length can simply be read into record structure • Others must be “parsed” with a parse algorithm

Record Structures • How do we organize records in a file? • Records can be fixed length or variable length • Fixed length allows simple direct access lookup • Fixed may waste space • Variable - how do we find a records position?

Record Structures • Fixed Length Records • Fixed number of fields in records • Variable length • prefix each record with a length • Use a second file to keep track of record start positions • Place delimiter between records

Fixed Length Records • All records same length • Record positions can be calculated for direct access reads. • Does not imply the that the sizes or number of fields are fixed. • Variable length records would lead to unused space.

Fixed number of fields in records • Field size could be fixed or variable • Fixed • results in fixed size records • simply read directly into “struct” • Variable sized fields • delimited or field lengths • Simply count fields while parsing

Variable length Records • prefix each record with a length • Use a second file to keep track of record start positions • Place delimiter between records

Prefix records with a length • Allows true variable length records • Form of prefix: • Character number (fixed length) • Binary number (write integer without conversion) • Must consider Maximum length • No direct access (great for sequencial access)

Index of record start addresses • A second file is simply a list of offsets to successive records • Since the offsets are fixed length, this file allows direct access, thereby allow direct access to main file. • Problem • Maintaining file (adding and deleting records) • Cost of index

Place delimiter between records • Special character not used in record • Allows efficient variable size • No direct access • Bible files - use ‘\n’ as delimiter

Binary data in files • Binary reals and integers can be written, and read, from a file: • Need to know byte size of variables used. • “tsize” function returns data size

Binary data in files int rsize; char rec_buf[MAX]; ... cpystr(rec_buf,”this is a test record”); rsize = strlen(rec_buf); write(my_fd,&rsize,tsize(int)); // write the size write(my_fd,&rec_buf,rsize); // write the record ... read(my_fd, &rsize,tsize(int)); // read the size read(my_fd,&rec_buf,rsize); // read the record

Viewing Binary file data • Use the file dump utility (od - octal dump) • od -xc <filename> • x - hex output • c - character output • Useful for viewing what is actually in file

Using Classes to Manipulate Buffer • Three Classes • delimited fields • Length-based fields • Fixed length fields

Class for Delimited fields • Consider a class to manage delimited text buffers • Allows reading and writing of delimited records • Allows packing and unpacking

Class for Delimited fields class Person { public: // fields char LastName [11]; char FirstName [11]; char Address [16]; char City [16]; char State [3]; char ZipCode [10]; // Methods next ... }

Class for Delimited fields class DelimTextBuffer { public: DelimTextBuffer (char Delim = '|', int maxBytes = 1000); int Read (istream &); int Write (ostream &) const; int Pack (const char *, int size = -1); int Unpack (char *); private: char Delim; char DelimStr[2]; // zero terminated string for Delim char * Buffer; // character array to hold field values int BufferSize; // size of packed fields int MaxBytes; // maximum number of characters in the buffer int NextByte; // packing/unpacking position in buffer };

Class for Delimited fields • Packing a buffer Person Bill_Yeakus DelimitedTextBuffer buffer; buffer.pack(Bill_Yeakus.LastName); buffer.pack(Bill_Yeakus.FastName); … buffer.pack(Bill_Yeakus.ZipCode); buffer.Write(stream);

Class for Delimited fields int DelimTextBuffer :: Pack (const char * str, int size) // set the value of the next field of the buffer; // if size = -1 (default) use strlen(str) as Delim of field { short len; // length of string to be packed if (size >= 0) len = size; else len = strlen (str); if (len > strlen(str)) // str is too short! return FALSE; int start = NextByte; // first character to be packed NextByte += len + 1; if (NextByte > MaxBytes) return FALSE; memcpy (&Buffer[start], str, len); Buffer [start+len] = Delim; // add delimeter BufferSize = NextByte; return TRUE; }

Class for Delimited fields int DelimTextBuffer :: Write (ostream & stream) const { stream . write ((char*)&BufferSize, sizeof(BufferSize)); stream . write (Buffer, BufferSize); return stream . good (); }

Class for Delimited fields int DelimTextBuffer :: Read (istream & stream) { Clear (); stream . read ((char*)&BufferSize, sizeof(BufferSize)); if (stream.fail()) return FALSE; if (BufferSize > MaxBytes) return FALSE; // buffer overflow stream . read (Buffer, BufferSize); return stream . good (); }

Class for Delimited fields int DelimTextBuffer :: Unpack (char * str) // extract the value of the next field of the buffer { int len = -1; // length of packed string int start = NextByte; // first character to be unpacked for (int i = start; i < BufferSize; i++) if (Buffer[i] == Delim) {len = i - start; break;} if (len == -1) return FALSE; // delimeter not found NextByte += len + 1; if (NextByte > BufferSize) return FALSE; strncpy (str, &Buffer[start], len); str [len] = 0; // zero termination for string return TRUE; }

Class for Delimited fields • Class Person can be extended to provide specialized packing functions

Class for Delimited fields int Person::Pack (DelimTextBuffer & Buffer) const {// pack the fields into a FixedTextBuffer, return TRUE if all succeed, FALSE o/w int result; Buffer . Clear (); result = Buffer . Pack (LastName); result = result && Buffer . Pack (FirstName); result = result && Buffer . Pack (Address); result = result && Buffer . Pack (City); result = result && Buffer . Pack (State); result = result && Buffer . Pack (ZipCode); return result; }

Class for Delimited fields int Person::Unpack (DelimTextBuffer & Buffer) { int result; result = Buffer . Unpack (LastName); result = result && Buffer . Unpack (FirstName); result = result && Buffer . Unpack (Address); result = result && Buffer . Unpack (City); result = result && Buffer . Unpack (State); result = result && Buffer . Unpack (ZipCode); return result; }

Class for Fixed Length fields int FixedTextBuffer :: AddField (int fieldSize) { if (NumFields == MaxFields) return FALSE; if (BufferSize + fieldSize > MaxChars) return FALSE; FieldSize[NumFields] = fieldSize; NumFields ++; BufferSize += fieldSize; return TRUE; }

Class for Fixed Length fields int FixedTextBuffer :: Read (istream & stream) { stream . read (Buffer, BufferSize); return stream . good (); }

Class for Fixed Length fields int FixedTextBuffer :: Write (ostream & stream) { stream . write (Buffer, BufferSize); return stream . good (); }

Class for Fixed Length fields int FixedTextBuffer :: Pack (const char * str) // set the value of the next field of the buffer; { if (NextField == NumFields || !Packing) // buffer is full or not packing mode return FALSE; int len = strlen (str); int start = NextCharacter; // first byte to be packed int packSize = FieldSize[NextField]; // number bytes to be packed strncpy (&Buffer[start], str, packSize); NextCharacter += packSize; NextField ++; // if len < packSize, pad with blanks for (int i = start + packSize; i < NextCharacter; i ++) Buffer[start] = ' '; Buffer [NextCharacter] = 0; // make buffer look like a string if (NextField == NumFields) // buffer is full { Packing = FALSE; NextField = NextCharacter = 0; } return TRUE; }

Class for Fixed Length fields int FixedTextBuffer :: Unpack (char * str) // extract the value of the next field of the buffer { if (NextField == NumFields || Packing) // buffer is full or not unpacking mode return FALSE; int start = NextCharacter; // first byte to be unpacked int packSize = FieldSize[NextField]; // number bytes to be unpacked strncpy (str, &Buffer[start], packSize); str [packSize] = 0; // terminate string with zero NextCharacter += packSize; NextField ++; if (NextField == NumFields) Clear (); // all fields unpacked return TRUE; }

Class for Fixed Length fields void FixedTextBuffer :: Print (ostream & stream) { stream << "Buffer has max fields "<<MaxFields<<" and actual "<<NumFields<<endl <<"max bytes "<<MaxChars<<" and Buffer Size "<<BufferSize<<endl; for (int i = 0; i < NumFields; i++) stream <<"\tfield "<<i<<" size "<<FieldSize[i]<<endl; if (Packing) stream <<"\tPacking\n"; else stream <<"\tnot Packing\n"; stream <<"Contents: '"<<Buffer<<"'"<<endl; }

Class for Fixed Length fields class FixedTextBuffer { public: FixedTextBuffer (int maxFields, int maxChars = 1000); int AddField (int fieldSize); int Read (istream &); int Write (ostream &); int Pack (const char *); int Unpack (char *); private: char * Buffer; // character array to hold field values int BufferSize; // sum of the sizes of declared fields int * FieldSize; // array to hold field sizes int MaxChars; // maximum number of characters in the buffer int NextCharacter; // packing/unpacking position in buffer };

Class for Fixed Length fields int Person::Pack (FixedTextBuffer & Buffer) const {// pack the fields into a FixedTextBuffer, return TRUE if all succeed, FALSE o/w int result; Buffer . Clear (); result = Buffer . Pack (LastName); result = result && Buffer . Pack (FirstName); result = result && Buffer . Pack (Address); result = result && Buffer . Pack (City); result = result && Buffer . Pack (State); result = result && Buffer . Pack (ZipCode); return result; }

Class for Fixed Length fields int Person::Unpack (FixedTextBuffer & Buffer) { Clear (); int result; result = Buffer . Unpack (LastName); result = result && Buffer . Unpack (FirstName); result = result && Buffer . Unpack (Address); result = result && Buffer . Unpack (City); result = result && Buffer . Unpack (State); result = result && Buffer . Unpack (ZipCode); return result; }

Record Access - Keys • Attribute used to identify records • Often used to find records • Standard or canonical form • rules which keys must conform to • prevents missing record because key in different form • Example: • all capitals • Phone in form (nnn) nnn-nnnn

Record Access - Keys • Keys can distinct - uniquely identify records • Primary keys • one-to-one relationship between key value and possible entities represented • SSN, Student ID • Keys can identify a collection of records • Secondary keys • one-to-many relationship • City, position, department

Record Access - Keys • Primary key desired characteristics • unique among collection of entities • dataless - what if some entities have not value of this type (e.g. SSN) • unchanging

Record access • Performance of access method • how do we compare techniques? • Must be careful what events we count. • “big-oh” notation gives us a way to factor out all but the most significant factors

Record Access - timing • Sequential searching • Consider file of 4000 records • What if no blocking done, and one record per block? (500 bytes records, 512 byte blocks) • What if cluster size set to 8? • always requires O(n), but search is faster by a constant factor

Sequential searching • Usually NOT the best method • Sometimes it is best: • Searching for some ASCII pattern (grep) • Small files • Files rarely searched • Searching on secondary key, and a large percentage of records match (say 25%)

Unix Tools for sequential file processing • cat - display a file • wc - count lines, words, and characters • grep - find lines in file(s) which match regular expression.

Direct Access • Move “directly” to record without scanning preceding data • Different languages/OS’s support different models: • Byte offset model • Programmer must specify offset to record, and record size to read. • Supports variable size records, skip sequential processing • Relative Record Number (RRN) model • File has a fixed record size (declared at creation time) • Records are specified by a record number • File modeled as a collection of components • Higher level of abstraction

Fundamental File Structure Concepts