AUSTIN: A tool for Search based software testing

Kiran Lakhotia, Mark Harman and Hamilton Gross in AUSTIN: A tool for Search based software testing And its Evaluation on Deployed Automotive Systems

Publication Trend in SBSE

Spread of Activities in SBSE

Yet Another Search Based Testing Tool?

Outline

AUgmented Search based TestINg c source files preprocessed c source files gcc -P -E CIL AST, CFG instrumented c source runAUSTIN gcc -o sut.exe instrument

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • }

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 87, key2 = 33, p = 0, q = 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 80, key2 = 33, p = 0, q = 0 Fitness Function: approach_level + norm(branch_distance) Search Method: Alternating Variable Method

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0, q = 0 Symbolic Path Condition: key1 == key2 && p == 0 Simplified Path Condition: p == 0 Solve: p ≠ 0 p 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0 Symbolic Path Condition: key1 == key2 && p ≠ 0 && p ≠ q Simplified Path Condition: p ≠ 0 && p ≠ q Solve: p ≠ 0 && p == q p 0 p,q 0 q

AUgmented Search based TestINg • void testme(key1, key2, p, q) • int key1; • int key2; • int* p; • int* q; • { • if(key1 == key2) • if(p != 0) • if(p == q) • //target • } Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0x...

rZSlJ3hb Evolutionary Testing Framework (ETF) • ETF was developed as part of the EvoTest project • Integrated into the Eclipse IDE • Supports white & black box testing • Uses Evolutionary Algorithms (through the GUIDE, EO evolutionary library interface) to generate test data Send Fitness Values ETF GUIDE + EO Send Individuals

rZSlJ3hb ETF cont. Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009

Fitness Function and Algorithm • Both ETF and AUSTIN use the standard fitness function for branch coverage in Search Based Testing • apporach_level + norm(branch_distance) • normalization function: 1 - 1.001-branch_distance • ETF uses a Genetic Algorithm • AUSTIN uses the Alternating Variable Method

Pointers in ETF and AUSTIN • AUSTIN • Construct an equivalence of memory locations based on = and ≠ operators in symbolic path condition • Initialize (pointer) input variables based on the graph • Assign new memory location via malloc or use existing memory location • On-demand pointer handling • ETF • Generate an index and auxiliary variable for each pointer input • Assign all input variables of the same type (including globals) to a pool • Optimize indices and values of auxiliary variables along with the inputs. Indices are used to select variables from pools and assign their address to the pointer input

Pointers in ETF struct cell{ int value; struct cell* next; }; intglobX; double globDbl; struct cell* pCell; void testme(int* p, int* q) { ...} p = &globX; q = &val_p;

Hypothesis

Empirical Study • 3 case studies (comprising 8 functions) • Embedded Software Modules from Automotive Industry

Coverage ETF vs AUSTIN

Efficiency ETF vs AUSTIN

Coverage Random vs AUSTIN

Efficiency Random vs AUSTIN

ETF or AUSTIN? • Coverage is about the same • AUSTIN is overall a little bit more efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup) • AUSTIN does not construct random memory graphs • ETF is not able generate inputs such as struct cell{ intmem; }; void testme(int* p, struct cell* c) { if( p == &c->mem ) //target }

Summary

Summary http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm

AUSTIN: A tool for Search based software testing