Slycat Ensemble Analysis

1. input meta-features output meta-features … c1 ck i1 . . . inputs ik features o1 o2 outputs . . . om CCA components Patricia J. Crossno, Timothy M. Shead, Milosz A. Sielicki, Warren L. Hunt, Shawn Martin, and Ming-Yu Hsieh Sandia National Laboratories Slycat Ensemble Analysis Problem: Electrical Circuit Simulation Sensitivity Analysis Finding Most Significant Inputs Model Confidence Input parameters Slycat Sensitivity Analysis Simple Regression (1-to-1) Multiple Regression (Many-to-1) Finding Anomalous Simulations Bar chart: Ensemble-wide Relationships • Scatterplot: Each Simulation Relative to Ensemble How About Many-to-Many Correlations? Viewing 2nd CCA component in both bar chart & scatterplot Viewing 1st CCA component in both views Simulation Ensemble Distance off diagonal shows difference from ensemble as a whole, plus potential anomalies. 250 simulations, each color-coded by its x23 input value 266 scrollable Inputs Inputs x25 & x14 have the most impact on both outputs y1 and y2 Positive many-to-many correlation (bar color the same) between X25 & X14 and Y2 & Y1 X23 selected for scatterplot color-coding (dark green row highlight) CCA1 Green = Inputs Objectives: • Map Output Variability Back to Inputs • Reduce Number of Input Parameters • Reduce Number of Simulations to Run • Identify Anomalous Runs • Increase Model Confidence 250 simulations, each color-coded by its y1 output value Input x1 has the least impact on outputs y1 and y2 Inputs and outputs sorted by correlation strength within CCA2 component Three distinct groups of input values All to all analysis 4 anomalous runs in y4 values sn s4 Selected simulation Purple = Outputs s3 s2 Structure Correlations Click CCA column header to select CCA component in views s1 250 simulations, each color-coded by its y4 output value AnalysisTasks: • Find strongest input/output correlations • Find inputs with least impact on outputs • Find anomalous simulation runs • Rerun CCA analysis between all inputs and y4 to find strongest correlations (all-to-1) 2641 simulations, each color-coded by its y4 output value Inverse correlation (red vs. blue) between x23 & y4; CCA3 captures relationship between x8 & y3 Approach: Canonical Correlation Analysis (CCA) Scatterplot color-coding changed by clicking on y4 row (darker purple highlight) Three output value groups map to the 3 input groups All to y4 analysis simulations 4 anomalous runs share common x248 values … s1 s2 s3 s4 sn i1 . . . inputs Viewing 3rd CCA component in both bar chart & scatterplot 250 simulations, each color-coded by its x8 input value ik features CCA o1 o2 outputs . . . om 2641 simulations, each color-coded by its x248 input value (strongest) X8 selected for scatterplot color-coding (dark green row highlight) CCAVisual Representations Inverse correlation between x8 & y3; CCA2 captures relationship between x23 & y4 X8 inputs range from low (blue) to high (red) Note R2 is increasing & P is decreasing with each CCA component All to y4 analysis 4 anomalous runs share common x255 values 250 simulations, each color-coded by its y3 output value Reduce Inputs & Simulations 2641 simulations, each color-coded by its x255 input value (2nd strongest) Click header triangle to sort variables (toggles from decreasing to increasing) • In the 2641 run ensemble above, analysis allowed input parameters to be reduced from 266 to 21, decreasing simulation time ten-fold. Available Open Source • https://github.com/sandialabs/slycat Corresponding y3 outputs inversely range from high (red) to low (blue) Scatterplot color-coding changed by clicking on y3 row (darker purple highlight) Patricia J. Crossno: pjcross@sandia.gov Timothy M. Shead: tshead@sandia.gov Milosz A. Sielicki: masieli@sandia.govWarren L. Hunt: wlhunt@sandia.gov Shawn Martin: smartin@sandia.gov Ming-Yu Hsieh: myhsieh@sandia.gov Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND 2014-1399P