Counting what will count Does your dashboard predict ?

1. Counting what will countDoes your dashboard predict ? Koen Pauwels Amit Joshi

2. Marketing dashboards • Marketing accountability & accelerating change • Limits to human processing capacity still there: MSI: ‘separate signal from noise’, ‘dashboards’ • Top management interest: ‘substantial effort’ by 40% of US and UK companies (Clark ea 2006) • But current applications often fail to impress: Marketing Scientists can help raise the bar !

3. Academic research focus Reibstein ea (2005) discuss 5 development stages: • Identify key metrics (align with firm goals) • Populate the dashboard with data • Establish relation between dashboard items • Forecasting and ‘what if’ analysis • Connect to financial consequences Most dashboards yet to move beyond stage 2 ! Need research to select best metrics, relate to performance

4. You’ve got 33 x 3 = 99 variables • Market variables: price national brand, store brand, ∆ • Awareness: top-of-mind, aided, unaided, ad awareness • Trial/usage: ever, last week, last 4 weeks, 3 months • Liking/Satisfaction: given aware and given tried • Preference: favorite brand, will it satisfy future needs? • Purchase intent: given aware and given tried • Attribute ratings: taste, quality, trust, value, fun, feel • Usage occasion: home, on the go, afternoon, entertain Need to reduce 99 to 6-10 metrics (US) or 10-20 (UK)

5. Metric deletion rules (Ambler 2003) • Does the metric rarely change? • Is the metric too volatile to be reliable?  Univariate tests on time series properties • Is metric leading indicator of market outcome?  Pairwise tests of metric with performance • Does the metric add sufficient explanatory power to existing metrics?  Econometric models to explain performance

6. This research • Univariate: st. dev., coef. of variation, evolution • Pairwise: Granger Causality test with performance • Explanatory power: regression model comparison • Stepwise regression (Hocking 1976, Meiri ea 2005) • Reduced Rank Regression (Reinsel and Velu 1998) • Forecast error variance decomposition, based on Vector Autoregressive Model (Hanssens 1998) • Assessment: forecasting accuracy hold-out sample • Managerial control: impact size and lead time

7. Stepwise regression • Automatic selection based on statistical criteria Objective: select set of metrics with highest R2 • Forward: add variables with lowest p-value Backward: delete variables with highest p-value • Unidirectional: considers one variable at a time Stepwise: checks all included against criterion Combinatorial: evaluates every combination

8. Reduced Rank Regression • Uses correlation of key metrics and performance Yi = Xi’C + εi with Yi (m x 1) and Xi (n x 1) C (m x n) has rank r ≤ min (m, n) Restriction: m – r linear restrictions on C Maximize explained variance under restriction • Originally shrinkage regression (Aldrin 2002), now for selecting best combination variables

9. Forecast  variance decomposition • Based on Vector Autoregressive Model A ‘dynamic R2’, FEVD calculates the percentage of variation in performance that can can be attributed to changes in each of the endogenous variables (Hanssens 1998, Nijs ea 2006) • Measures the relative performance impact over time of shocks initiated by each endogenous var • We consider the FEVD at 26 weeks

10. Methods share 4/10 metrics

11. Stepwise scores within sample

12. But sucks out-of-sample

13. Sales Impact Size and Timing

14. Value (price-quality) matters right now, Awareness and Trial soon ! PRICE Unaided Awareness Tried last month QUALITY Entertain friends Like if tried Satisfying Afternoon Lift TRUST

15. Summary dashboard intuition 1) To increases sales immediately (0-1 weeks) a) promote on price and on afternoon lift usage b) communication focus on quality, affect, trust 2) To increase sales soon (2-3 weeks) a) provide free samples (to up ‘tried last month’) b) focus on satisfying and entertainment use c) advertise for unaided awareness

16. Conclusion: P-model for dashboards • Which metrics are leading indicators? Granger causality tests • Explain most of performance dynamics Forecast error Variance decomposition • Forecast multivariate baseline with Vector Autoregressive or Error Correction model • Displays timing and size of sales impact

17. Your Questions ?