Impact of Sales Force Structure Change on Products Performance Pilot Study

1. Impact of Sales Force Structure Change on Products PerformancePilot Study Business Intelligence Solutions June, 2015

2. Objectives/Business Questions • Does 2-up promotion of Product A have a positive impact on its sales relative to 1-up promotion? • The hypothesis behind 2-up promotion: Engaging a 2nd representative in the promotion will accelerate product adoption and have a positive impact on product performance of relative to 1-up promotion • Testing 1-up versus 2-up promotion will allow to assess the impact and relative value of a 2nd representative engaged in active promotion of Product A within selected customer segment • Does the incremental revenue associated with the 2nd sale representative actively promoting Product A provide an acceptable return on the investment? • Is promoting Product A the best short-term use of the SF1 sales force capacity?

3. Findings / Conclusions • There is no statistically significant or practically important difference in Product A sales between Test 1 and Control 1 groups • Promotion cost for Control 1 group is two times higher than for Test 1 group • The 2-up structure does not produce desired/expected outcome for Product A

4. Structure of Test – Control Groups • Test 1: Product B and Product C • Test 2: Product B, Product C, and Product A • Control: Product B, Product C • Control groups are formed on the basis of the last 2014 quarter sales data • The Test and Control groups were selected to allow for a sufficient number of matched customers across the two groups to account for other variables that may impact Product A sales • By matching locations with respect to other variables (DTC, business size, geography, etc.) we can effectively isolate the number of representatives actively promoting Product A as the differentiating factor between the groups Test 1 Test 2 Control SF2 SF1 SF2 SF2 SF1 SF1 Product B Product A Product B Product A Product B Product A Product B Product A Product B Product A Product C Product B Product C

5. Methodology • Form Test1- Control1 and Test2 - Control2 groups, using the data of the last quarter of 2014 and propensity score technique with: • nonparametric nonlinear logistic model • greedy one-to-one matching technique • Develop Stochastic Gradient Boosting regression models for the first quarter of 2015 for each pair of Test – Control groups, using the following dependent variables: • Product B sales • Product A sales • Product C Sales controlling for all • “User demographics” variables (sales potential, milestone, state, business size, etc.) • promotion variables in last quarter of 2014 • Estimate the difference in sales for different sales team

6. One-to-one Matching on Propensity ScorePropensity Score Basics • Propensity score • is the predicted probability of receiving the treatment (probability of belonging to a test group) • is a function of several differently scaled covariates • Propensity_Score = f (Product_B_Sales_Pre, Product_A_Sales_Pre, Product B_Sales_Potential, State , Product A_Sales_Potential, Product B_Potential_Decile, Promotion variables, etc.) where fis a non-parametric non-linear multivariate function, unique for each pair of Test – Control study • If State in ('MA', 'MI', 'MN', 'IL', 'FL', 'NJ') then DTC_Indicator = 1; else DTC_Indicator=0; • If State in ('NC', 'CA', 'NY', 'GA', 'VA') then Paper_Indicator = 1; else Paper_Indicator = 0; • A sample matched on propensity score will be similar across all covariates used to calculate propensity score

7. Control Groups • Control groups are formed on the base of propensity score methodology, using only the last 2014 quarter data • Control1 (for Test1 group with 547 Users): • Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MI, MN, NC, NJ, WI • Total Unmatched Number of Users: 4,244 • Matched Number of Users: 543 • Control2 (for Test2 group with 717 Users): • Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MA, MN, NC, NJ, TN, WI • Total Unmatched Number of Users: 6,784 • Matched Number of Users: 717

8. Propensity Scores Calculation • Approaches/software on non-parametric logistic regression: • SAS SEMMA (Sample, Explore, Modify, Model, Assess) methodology within SAS Enterprise Miner • SPSS CRISP (Cross Industry Standard Process for Data Mining) • Salford Systems CART, MARS, TreeNet, and Random Forest • Approach selected: SAS SEMMA within SAS Enterprise Miner and Stochastic Garadient Boosting of Salford Systems • Test1 – Control1: (543 Product Users per group) • Best model: Funnel architecture of Neural Net • Test2 – Control2: (717 Product Users per group) • Best model: Cascade Correlation architecture of Neural Net

9. Propensity Score: Selection the Best Modeling Paradigm Neural Net was the best modeling paradigm

10. Propensity Score for Test1 – Control1 Groups: Selection the Best Modeling Method Neural Net with Funnel architecture was the best modeling method Misclassification Rate: Train Validation 0.11 0.12

11. Propensity Score for Test2 – Control2 Groups: Selection the Best Modeling Method Neural Net with Cascade architecture was the best modeling method Misclassification Rate: Train Validation 0.09 0.10

12. Matched-Pair Samples Comparison • Non-parametric tests: • For interval variables: • Kolmogorov-Smirnov Two-Sample Test • For nominal variables: • Chi-square test • Before matching there was a significant difference in predictor distribution across all variables for • Test1 – Control1 • Test2 – Control2 • After matching there was no significant difference in predictor distribution across all variables for • Test1 – Control1 • Test2 – Control2

13. Sales Analysis by GroupTreeNet/Stochastic Gradient Boosting Modeling • Total number of predictors: 42 • Non-parametric model structure: Dep_var_Post = f(Dep_var_Pre, Promo_vars_Pre, … User_demographics_vars)

14. Dependent Variable: Product B Sales Post Product B Sales Post Product B Sales Post Control2 Test2 Control1 Test1 Difference is staistically significant but practically not important

15. Dependent Variable: Product B Sales Post for Test1 – Control1 Product B Sales Post The most important 5 predictors of Product B Sales Post: Product_B_Sales_Pre Product_B_Sales_Potential State Product_B_Visits_Pre Product _A_Sales_Potential Control1 Test1 Difference is practically not important

16. Dependent Variable: Product C Sales Post for Test1 – Control1 Product C Sales Post Test1 Control1 The most important 5 predictors of Product C Sales Post: Product_C_Sales_Pre Product_A_Sales_Potential Product_B_Sales_Potential State Product_B_Visits_Pre Difference is practically not important

17. Dependent Variable: Product A Sales Post for Test1 – Control1 The most important 5 predictors of Product A Sales Post: Product_A_Sales_Pre Product_A_Sales_Potential State Product_B_Sales_Pre Product _C_Sales_Pre Conclusions There is no statistically significant or practically important difference in Product A sales between Test 1 and Control 1 groups, but promotion cost for Control 1 group is two times higher than for Test 1 group. In other words, 2-up structure does not produce desired/expected outcome for Product A