By: Drew Wicke Jointly with: Sri Hari Krishna Narayanan and Paul Hovland

Identifying Active Variables to Improve the Performance of Operator Overloading Automatic Differentiation By: Drew Wicke Jointly with: Sri Hari Krishna Narayanan and Paul Hovland

What is Automatic Differentiation? • A means of computing the derivative of a function within a computer program • Operator Overloading AD – Utilizes features of the programming language to alter the meaning of mathematical operators to compute the derivative

What is Sacado? • Operator overloaded version of AD • Made by Sandia National Laboratories • A package in the Trilinos framework • http://trilinos.sandia.gov/packages/sacado/index.html DERIV_TYPE operator* (DERIV_TYPE other) { this->val = this->val() * other.val(); this->dx = this->val() * other.dx() + other->val() * this->dx(); return *this; }

Example Code //Function where x is the independent f // is the dependent voidfoo(double *x, double *f){ doubletheConst = 3.14 * 2 – 6; double div = ((*x) / 2); doublesquarePoly = (*x) * (*x) + 45; if (squarePoly >= 100) { (*f) = div * theConst; } else{ doublealtInput = (*x) * theConst; (*f) = div * altInput + theConst; } } int main() { doublex, f; foo(&x, &f); return0; }

Sacado Example typedefSacado::Fad::DFad<double> DERIV_TYPE_double; voidfoo(DERIV_TYPE_double x, DERIV_TYPE_double f); int main() { DERIV_TYPE_double x, f;// independent and dependent vars // init value of x and f is set to zero and deriv // value is set to zero x= 2.0; // set the initial x.diff(0, 1); // set that x is independent foo(&x, &f); // call foo with x and f as args // print the value of x and the value of f // by calling the method val which returns // the non-derivative value printf("f(%f) = %f\n", x.val(), f.val()); // print the derivative by calling the dx method // and pass zero as the argtodxto specify what // independent variable we want the derivative // with respect to printf("df/dx = %f\n", f.dx(0)); return0; }

Example Continued //Function where x is the input f is the output voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ DERIV_TYPE_doubletheConst = 3.14 * 2 – 6; DERIV_TYPE_double div = ((*x) / 2); DERIV_TYPE_doublesquarePoly = (*x) * (*x) + 45; if (squarePoly >= 100) { (*f) = div * theConst; } else{ DERIV_TYPE_doublealtInput = (*x) * theConst; (*f) = div * altInput + theConst; } }

Why That Code is Not Optimal • Vary variables are those whose value is computed using an independent variable. • Useful variablesare used to compute the value of the dependent variable • Active variablesare both vary and useful. voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ DERIV_TYPE_doublesquarePoly= (*x) * (*x) + 45; DERIV_TYPE_doubletheConst= 3.14 * 2 – 6; DERIV_TYPE_double div = ((*x) / 2); if (squarePoly >= 100) (*f) = div * theConst; else{ DERIV_TYPE_doublealtInput= (*x) * theConst; (*f) = div * altInput + theConst; } }

What Does Our Tool Do? • Replaces the manual process, which is slow and overestimates the number active variables. voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ double squarePoly= (*x) * (*x) + 45; // vary double theConst= 3.14 * 2 – 6; // useful DERIV_TYPE_double div = ((*x) / 2); // active if (squarePoly >= 100) { (*f) = div * theConst; } else{ DERIV_TYPE_doublealtInput= (*x) * theConst; (*f) = div * altInput + theConst; } }

Activity Analysis Tool Flow ROSE AST OpenAnalysis ICFG Input code convert int main(){ doubleval; val = 4.3; … return0; } convert Useful Analysis Vary Analysis Activity Analysis Output code Change type of active variables int main(){ DERIV_TYPE_doubleval; val = 4.3; … return0; }

Challenges in Type Changing • Typedefs – must extract base type and all other types such as pointers to set to the derivative type. • Structs – must copy struct and change all variables with the body of the struct to the Sacado derivative type. struct mine { doublemdou; }; struct mine smin; // active (before) typedefdouble* fir; typedef fir* sec; sec test;//active (before) structmine { doublemdou; }; structDERIV_CLASS_mine { DERIV_TYPE_doublemdou; } typedefstructDERIV_CLASS_mineDERIV_TYPE_mine; DERIV_TYPE_minesmin;// (after) // after: DERIV_TYPE_double **test;

Initial Code //Function where x is the independent f is the dependent voidfoo(double *x, double *f){ # pragma $adic_indep,x # pragma $adic_dep,f// do some math doubletheConst = 3.14 * 2 – 6; double div = ((*x) / 2); doublesquarePoly = (*x) * (*x) + 45; if (squarePoly >= 100) { (*f) = div * theConst; } else{ doublealtInput = (*x) * theConst; (*f) = div * altInput + theConst; } }

Resulting Code //Function where x is the input f is the output voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ # pragma $adic_indep,x # pragma $adic_dep,f// do math doubletheConst = 3.14 * 2 – 6; // .28 DERIV_TYPE_double div = ((*x) / 2); doublesquarePoly = (*x) * (*x) + 45;// ERROR must downcast! if (squarePoly >= 100) { (*f) = div * theConst; } else{ DERIV_TYPE_doublealtInput = (*x) * theConst; (*f) = div * altInput + theConst; } } Add a downcast if the left hand side is vary and the right hand side has an active variable.

Resulting Code With Downcast //Function where x is the input f is the output voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ # pragma $adic_indep,x # pragma $adic_dep,f// do math doubletheConst = 3.14 * 2 – 6; // .28 DERIV_TYPE_double div = ((*x) / 2); doublesquarePoly = ADValue((*x) * (*x) + 45); // Problem solved if (squarePoly >= 100) { (*f) = div * theConst; } else{ DERIV_TYPE_doublealtInput = (*x) * theConst; (*f) = div * altInput + theConst; } }

Tool Limitations • Typedefed structs: typedef struct{…} myStruct; • C++ structs, classes, templates • Function pointers: double (*func)(double, double) • Array parameters due to not being made active: void foo(double myParam[]);

Conclusion and Future Work • Can be used with any operator overloading AD package • Benchmarking the tool • Add support for C++ Thank You

How Does the Tool Work? • Creates Abstract Syntax Tree (AST) of code using ROSE • int main(){ • double val; • val = 4.3; • return0; • } • Converts AST to an OpenAnalysis form definition

Creates Interprocedural Control Flow Graph (ICFG) Images from: M. J. Harrold, G. Rothermel, and S. Sinha, “Computation of Interprocedural Control Dependence”, Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ‘98), pp. 11-20, March 1998

What Analyses are Used? • Vary Analysis – Identifies variables that vary with the independent variable, but are not used in the output • Useful Analysis – Identifies variables that are not vary but contribute to the output • Activity Analysis – Identifies active variables, those that depend on the value of an input variable to compute the output variable. The intersection of vary and useful variables.

What Does Our Tool Do? • Replaces the manual process that is slow and overestimates, with our tool. • Identifies active variables using the source code analysis toolkit OpenAnalysis • Changes the type of active variables to a typedef type using the source transformation tool ROSE voidfoo(DERIV_TYPE_double *x, DERIV_TYPE_double *f){ double theConst= 3.14 * 2 – 6; DERIV_TYPE_double div = ((*x) / 2); double squarePoly= (*x) * (*x) + 45; if (squarePoly >= 100) { (*f) = div * theConst; } else{ DERIV_TYPE_doublealtInput= (*x) * theConst; (*f) = div * altInput + theConst; } }

By: Drew Wicke Jointly with: Sri Hari Krishna Narayanan and Paul Hovland