Software Engineering: Analysis and Design (CSE3308)

Software Engineering: Analysis and Design (CSE3308) Lecture 11b: Software Development in Industry CSE3308/CSC3080/DMS/2000/26

Lecture 11b • Software Development Process Topics • Software Releases • Revision Control • Software Testing • Memory Usage Testing • GUI Testing Introduction

Software Releases • Release levels • Tree of revisions • Release Checklists • Release Checklist Example • Release Criteria • Defect Severity • Support Objectives Software Releases

Release levels • Four levels of controlled releases at OSA: • Engineering Build (build) • result of a product/system build • regularly built from ‘leaf code’ in the trunk and/or CPE branch • tested with automated test suite • Alpha Release (alpha) • shipped only to OSA internal staff • tested with automated test suite, and against Alpha Release Checklist • Beta Release (beta) • shipped only to nominated customer(s) and their OSA support staff. • tested with automated test suite, and against Beta Release Checklist • Manufacturing Release (MR) • shipped with new product and/or upgrade sales, and to all customers with a support contract. • tested with automated test suite, and against MR Release Checklist Software Releases

Release levels • Variations of an MR level Release: • Patch • a update to a Manufacturing Release to correct defects • built from code in a CPE (Customer Product Engineering) branch of the ‘revision tree’. • shipped with new product and/or upgrade sales, and to all customers with a support contract • tested as per an MR, with additional testing to ensure that it can be applied over a particular MR • General Availability Release (GA) • aka: Gold Release • concurrent availability of MR level releases for a product on all of its supported platforms. • tested as per an MR, for all supported platforms. • shipped with new product and/or upgrade sales, and to all customers with a support contract Software Releases

Tree of revisions Product: OpenUI Development File: windows1.cxx • trunk revisions are on central line • branch revisions are to the sides • trunk revisions, branches and branch revisions can be labeled symbolically. 4.36.1 4.36.1.7 3.48.1 3.48.1.1 CPE branch CPE branch d420p6d420p7 d350p1 d300 d350 d400 d410 d420 trunk ... rev. 3.0 3.48 4.0 4.36 4.12 branch 4.12.1 4.12.1.1 Software Releases

Checklists used to guide the release process document the release procedure confirm that all release criteria have been met met record notes from release process record sign-offs with dates are accessible on-line through CGI based forms Checklists identify: Product Name Package Name and Build Number Release Platform/s Product Distribution Details Product Changes & Defect Fixes ExampleRelease Checklist Sections Release Procedure Product Configuration Distribution Details Product Change Details LPE Checklist Product Development Checklist Quality Checklist Support Checklist Manufacturing Checklist Release Sign-Off Post-Signoff Processing Announcement Approval Notes Release Checklists Software Releases

Release Checklist Example • Ref: Release Sign-Off Form Software Releases

Release Criteria • Test Coverage, Reliability, & DefectsThe following test coverage, minimum reliability, and defect rate criteria will be applied. NB: These criteria apply to both customer reported and OSA reported defects. • For new or modified code • Release Quality ImprovementFor MR releases, quality as measured against defect, reliability, and test coverage criteria will be on a par with, or better than, the prior release of the same product. Software Releases

Defect Severity • CriticalThe product can't be used. No work-around exists. • Data corrupted or lost • System failure or lockup occurs • Incorrect answers when necessary for useful operation • Program aborts when used correctly • Corruption of O/S or system due to incorrect use of product • SeriousThe product can't be used but a work-around exists. • Much harder to use than reasonably expected • Defect with awkward or inefficient work-around • Defect with misleading output • Documentation functionally incorrect • Program aborts when used incorrectly Software Releases

Defect Severity • Medium / ModerateThe product is useable with moderate workaround that can be provided. • Confusing user interface • Program aborts in malicious use • Defect is not serious to the operations of the user • Inefficient algorithm leads to substantial waste of time/space • Defect causes a preferred feature not to be used • LowThe product is useable with a simple workaround or fix. • Grammatical error in output • Ugly screen layout • User misunderstanding because of somewhat unclear documentation • Cosmetic documentation error where correct use of the product is obvious Software Releases

Support Objectives • Verification and Grading Period for DefectsThe Verification and Grading Period is the time from receipt of a Problem Report, until it is verified, analysed, and graded. • Rectification Period for DefectsThe Rectification Period includes the Verification and Grading Period. i.e. it is the total time from the first customer report of the defect, until a defect repair is delivered to the customer. Software Releases

Revision Control • Revision Control • Cron brings it all together • Daily RCS logs • Review processes • Formal review guidelines Revision Control

Revision control • Functions of a revision control system: • Maintenance of a tree of revisions. • RCS is used to maintain a complete history of changes (revisions) to a ‘source’ file. • In addition to the text of each revision, RCS stores the author, date and time of check in, and a log message summarizing the change. • RCS can maintain separate lines of development for each file. It stores a tree structure that represents the ancestral relationships among revisions. • Resolution of access conflicts. • When two or more people try to modify the same revision of a file, RCS alerts them and prevents one modification from corrupting the other. • Merging of revisions and resolution of conflicts. • Two separate lines of development of a file can be coalesced by merging. If the revisions to be merged affect the same lines of a file, RCS flags the overlapping changes. • Release and configuration control. • Revisions can be assigned symbolic names or labels. Labels can be used to identify a file revision in relation to Product Releases, and development milestones. When systematically applied, revision labeling can enable system configurations to be simply accessed and managed. Revision Control

Cron brings it all together • At OSA, every night, several machines do clean full build of the product, picking up all changes made during the day. This requires a large set of recursive makefiles • Every code change must be reviewed by at least one other engineer. The name of the reviewer and the DTS (work unit) number that is fixed by the change must be included in the RCS log for the change. • Every night, a report is generated by collating the RCS logs of all changes made that day. This report is checked by a manager the following day. Revision Control

Cron brings it all together • When each work unit is resolved, the engineer must include a labnotes file, describing the rationale behind the change, and a documentation file, if a documentation change is made. • Every night, a report is generated of all work units that have changed state, including the additional labnotes and documentation files. This report is checked by a manager the following day. Revision Control

Daily RCS logs • *** /src/forms/master/src/eqlib/common/fxform.cxx,v ***revision 3.2date: 1996/04/12 04:25:39; author: shaun; state: Exp; lines: +4 -4DTS 9001. Previous change wasn't quite correct. Reviewers: pca, agw.===============================================*** /src/forms/master/src/eqlib/common/text1.cxx,v ***revision 3.33date: 1996/04/12 06:07:20; author: joeca; state: Exp; lines: +4 -3DTS9001: added member initialization for copy constructor to stop purifyUMR's. Rev michael===============================================*** /src/forms/master/src/eqlib/common/browser.cxx,v ***revision 1.11.3date: 1996/04/12 06:09:17; author: michael; state: Exp; lines: +38 -29DTS14306:Fixed msie i/f to work with dde calls. rev joeca.=============================================== Revision Control

Review processes • Reviews can be conducted at all stages in the development process: • The purpose of holding a review includes: • to increase familiarity of code and designs within projects • to clarify and understand interfaces • to gain exposure to examples of code / design (both good and bad) • to provide feedback to the author • to find problems • to reduce testing phases • to stabilize design before coding • to increase exposure to the review process itself • Reviews can be formal or informal • OSA has a practice that all code changes are at least informally reviewed. Revision Control

Formal review guidelines • In a formal review, each review attendee has a specific role to play: • Author Organizes the review • Reader Conducts the review • Facilitator Controls the review • Inspector Attends the review to find problems • Observer Attends the review for experience • Reviews are intended to identify problems not to solve them • Preparation by the Reader and Inspector is very important and may take several hours. • The reader "reads" code by describing what it is doing. The Inspector identifies problem areas as the Reader reaches them. The Author responds to questions about problems areas. When there is agreement on the identification of a problem, the Facilitator notes the problem, and the review continues on. Revision Control

Formal review guidelines • Formal reviews can be quite intimidating, but well run they are an excellent mechanism for improving the quality of the software and team effectiveness. Revision Control

Software Testing • Testing approaches and strategies • Software testing environment • Regression suites • Unit Test guidelines Software Testing

Testing approaches and strategies • Their are two main approaches: • white-box testing: done with an understanding of the internals of the implementation • black-box testing: only concerned with the system’s external behavior • With black-box testing, two main strategies: • limits testing: push the limits of the system to see where and when it fails • equivalence partitioning: functions and parameters within the system can be divided into equivalence classes, one sample from each class is used for testing. Software Testing

Regression suites • A Regression Test Suite is used to identify regressed, or unexpected charges in system behavior and performance, with respect to an established baseline. • A regression suite is a group of tests designed to thoroughly test all required functionality of a system. • Typically, a regression suite is run against each new build of a product to see if any of its functionality has regressed with respect to the established baseline. • Three important rules to follow when creating regression tests: • Build the capability to recover from errors into every logical component of the regression • Structure the tests so that no component depends on the success of a previous component • Start each component from a known base state Software Testing

Unit Test guidelines • Some Guidelines: • a programmer should avoid testing their own code • thoroughly inspect the results of each test • test cases must be written for the invalid and unexpected (as well as the valid and expected) • make test cases and data repeatable • do not plan a testing effort on the tacit assumption that no errors will be found • testing is an extremely creative and intellectually challenging task Software Testing

Software testing environment • Some commercial software testing tools are particularly useful: • Memory test tools, e.g. Purify and TestCenter • GUI Test Tools, e.g. QA Partner and X Runner/WinRunner • Memory Test Tools • Memory errors in languages such as C and C++ are easy to make and can be quite difficult to find. • Memory Test Tools provide a form of internal limits testing, often detecting problems which become critical in extreme situations. • GUI Test Tools • File-based programs can be tested by comparing actual output files with expected output files. • For GUI programs, however, the "output" is to the screen. This output is not fixed but depends on screen size, resolution, and positioning. • Commercial tools provide specialized GUI toolkit drivers which accommodate this flexibility. Software Testing

end of lecture 11b

Memory Usage Testing • Memory usage testing • Memory usage errors • Memory usage error examples • UMR: Uninitialised Memory Read • ABR/W: Array Bounds Read/Write • FMR/W: Freed Memory Read/Write • FUM: Freeing Unallocated Memory • How Purify Works • Memory leaks Memory Usage Testing

Memory usage testing • Recall, on UNIX, memory is available in the following areas: • Data segment: initialized static memory • BSS segment: zeroed static memory • Stack: parameters and local variables • Heap: malloc'd memory Memory Usage Testing

Memory usage errors • A number of memory errors can easily occur: • UMR: Uninitialised memory read (stack, heap) • ABR/W: Array bounds read/write (heap) • FMR/W: Freed memory read/write (heap) • FUM: Freeing Unallocated Memory (heap) • NPR/W: Null Pointer Read/Write • Error detection • In C and C++, these errors are only detected when they cause a segmentation violation, often long after a detectable error has occurred. • Other languages provide checks which may detect errors earlier, but for debugging purposes the stack backtrace of who allocated the current or related chunks of memory is most useful • Code should not leave home without being Purified. Memory Usage Testing

UMR: Uninitialised Memory Read • A UMR is a warning that uninitialised memory is about to be read. For example: • main(){ int num1, num2; num1 = num2 + 1; /* num2 is used */ exit (0)} • will yield the following report: • UMR: Uninitialised memory read:* This is occurring while in: main [line 5, test.c]* Reading 4 bytes from 0xf7fff348 on stack.* This is local variable "num2" in function main. Memory Usage Testing

UMR: Uninitialised Memory Read • Often, uninitialised memory will be zero, especially during unit testing. Your program will seem to perform correctly but the UMR will eventually cause incorrect behavior. • If the memory about to be read has been allocated on the heap, then you get a stack backtrace of who allocated the memory: • typedef struct{ char *data_p; int length;} string;main(){ string *sptr; int slen; sptr = (string *) malloc(sizeof(string)); slen = sptr->length + 1; exit(0);} Memory Usage Testing

UMR: Uninitialised Memory Read • will yield the following report: • UMR: Uninitialized memory read:* This is occurring while in: main [line 13, test.c]* Reading 4 bytes from 0x5f524 in the heap.* Address 0x5f524 is 4 bytes into a malloc'd block at 0x5f520 of 8bytes.* This block was allocated from: malloc [rtlib.o] main [line 12, test.c] Memory Usage Testing

ABR/W: Array Bounds Read/Write • An ABR/W indicates that the program is about to read or write a value before or after an allocated block. ABR indicates a warning, ABW indicates a corrupting error. For example: • main(){ char *orig = "test"; char * copy; copy = (char *) malloc(strlen(orig)); strcpy(copy, orig); free(copy); exit(0);} Memory Usage Testing

ABR/W: Array Bounds Read/Write • will yield the following report: • ABW: Array bounds write:* This is occurring while in: strcpy [rtlib.o] main [line 7, test.c]* Writing 5 bytes to 0x5f558 in the heap (1 byte at 0x5f55c illegal).* Address 0x5f558 is at the beginning of a malloc'd block of 4 bytes.* This block was allocated from: malloc [rtlib.o] main [line 6, test.c] Memory Usage Testing

FMR/W: Freed Memory Read/Write • An FMR/W indicates that the program is about to read from or write to heap memory that has already been freed. FMR indicates a warning, FMW indicates a corrupting error. For example: • main(){ int * num_p; int other_num; num_p = (int *) malloc(sizeof(int)); free(num_p); other_num = *num_p + 1; exit(0);} Memory Usage Testing

FMR/W: Freed Memory Read/Write • yields the following report: • FMR: Free memory read:* This is occurring while in: main [line 8, test.c]* Reading 4 bytes from 0x5f530 in the heap.* Address 0x5f530 is at the beginning of a freed block of 4bytes.* This block was allocated from: malloc [rtlib.o] main [line 6, test.c]* There have been 0 frees since this block was freed from: free [rtlib.o] main [line 7, test.c] Memory Usage Testing

FUM: Freeing Unallocated Memory • An FUM indicates that the program is trying to free unallocated memory (duplicate free or free of bad heap pointer). FUM indicates a warning. For example: • main(){ char *mem_p; mem_p = (char *) malloc(10); free(mem_p); free(mem_p); exit(0);} Memory Usage Testing

FUM: Freeing Unallocated Memory • yields the following report: • FUM: Freeing unallocated memory:* This is occurring while in: free [rtlib.o] main [line 7, test.c]* Attempting to free block at 0x5f520 already freed.* This block was allocated from: malloc [rtlib.o] main [line 5, test.c]* There have been 1 frees since this block was freed from: free [rtlib.o] main [line 6, test.c] • This error often occurs due to confusion about pointer ownership. Memory Usage Testing

How Purify Works • Purify checks every read, write, allocation and free of memory, and verifies whether these operations have been done legally. • Conceptually, Purify color codes the memory. It is coded into three colors at the byte level: • Red: unallocated and uninitialised memory. It cannot be read, written or freed. The heap is initially red. • Yellow: allocated but not initialised. It can be written or freed, but cannot be read. Memory returned by malloc, and stack memory on function entry are set yellow. • Green: allocated and initialised. It can be read written or freed. The data and BSS segments are automatically set to green. • Purify also maintains stack backtraces to specific depth for all memory operations. Memory Usage Testing

How Purify Works • At link time, Purify inserts checking code into the program to maintain the color-coded map of memory and detect errors. • Since it works at the object code level, it is able to detect errors in the entire program, including files for which you do not have the source. Memory Usage Testing

Memory leaks • Memory leaks are memory chunks that cannot be accessed or freed. They slow down a program by fragmenting or using memory, causing increased paging, or worse causing a program to run out of memory. • Purify gives a memory leakage report at the end of each run, indicating how much memory was leaked from each distinct memory allocation stack backtrace. Memory Usage Testing

GUI Testing • Graphical user interface (GUI) applications consist of a set of windows and graphical controls, such as buttons, menus, edit texts lists, and combo-boxes • GUI testing presents some special difficulties: • System input/output is not fixed but depends on fonts, colors, screen size, screen resolution and positioning - record/playback is very sensitive to these changes • Screen snapshots generate too much low level data to be useful as expected output - one small change may invalidate all expected output • Tests may need to be portable across Microsoft Windows, OSF/Motif, Apple Macintosh, OS/2 Presentation Manager • These problems are overcome with special GUI testing tools such as QA Partner and WinRunner. GUI Testing

How QA Partner works • The QAP Recorder gets all the information the GUI has about each window and control on the screen, so the test scripts can refer to them by name, not by location and appearance. • Test scripts are written in 4Test, a C++ like language • Each test script has the general form: • initialize GUI • perform test • check result • handle exceptions GUI Program QAPRecorder QAPAgent QA Partner (+ script) GUI Testing

How QA Partner works • The QAP Recorder gets all the information the GUI has about each window and control on the screen, so the test scripts can refer to them by name, not by location and appearance. • Test scripts are written in 4Test, a C++ like language • Each test script has the general form: • initialize GUI • perform test • check result • handle exceptions GUI Program QAPRecorder QA Partner (+ script) GUI Testing

How QA Partner works • QA Partner runs the test script to perform test cases. • The QAP Agent executes GUI actions and queries against the program under test. • Many Agents can simultaneously execute a single QAP script to enable distributed or multi-host testing. • To allow a suite of Test Cases to be run, QA Partner implements an on-error recovery system that can restore the application back to a pre-defined state. GUI Program QAPRecorder QAPAgent QA Partner (+ script) GUI Testing

Referring to GUI Objects • Windows and controls in a GUI are organized as a hierarchy • Each window and control is an instance of a class • All windows or controls have associated text, such as the title of a window, the label of a button, the label of a menu • GUI objects are identified by a sequence of components corresponding to the object hierarchy. • Components can be separated by ”." • Each component can consist of: • the object's class, denoted by "[class]" • the object's name can be specified as follows: • its associated text • internal GUI ID • ordinal number in relation to other objects at that level in hierarchy, • prior static text associated with object • For example: • OurApp.Open.Files is a listbox on a dialog that is identified by “Files”, “#1”, “$4046”, “(56, 94)” • OurApp.Open.OK is a button on a dialog GUI Testing

Properties and operations on objects • Each class has a built-in set of attributes and methods • For example the built-in window class Menu includes these attributes: • lsContents The contents of the menu • bEnabled Whether the menu is enabled • It also includes these methods: • GetContents Gets the contents of the menu (an enquiry method) • IsEnabled Checks if the menu is enabled (a state query method) • Pick Selects an item from the menu (a GUI action method) • VerifyContents Verifies that the menu has the expected contents (a verification method) GUI Testing

Questions

Software Engineering: Analysis and Design (CSE3308)