Introduction To NLP Logix, LLC

1. Introduction To NLP Logix, LLC Ben Webster Data Scientist / Statistician, NLP Logix, LLC ben.webster@nlplogix.com

2. Business and Definitions • NLP Logix provides machine learning, predictive modeling, and statistical analysis as a service for many industries. Our primary focus has historically been in marketing, client retention / customer churn modeling, and optimization tasks. We use cutting edge technologies such as neural networks, boosted trees, and support vector machines in order to leverage historical data to predict future events with unparalleled accuracy. Definitions: Predictive Modeling: • The area of data analytics concerned with forecasting probabilities and trends Big Data : • Collections of data sets that become difficult to process using on-hand database management tools or traditional data processing applications. Machine Learning : • The science of using computers to analyze big data to develop predictive algorithms beyond the human element of time/labor and personal biases.

3. Putting the Model to Work 3. Production – We use your data to build the most accurate predictive model available, now let’s put it to work

4. Keeping the Pulse Monitor the Models – Now that your model is working hard for you, we are working hard to make sure it is performing at the optimal level. We have integrated advanced reporting tools into our platform to monitor model performance and look for “regime change” Example of why monitoring is extremely important: If you built an automotive marketing predictive model in 2007 and never changed it, how effective would it be after 2008? NLP Logix monitoring tools would have detected the market shift caused by the financial crisis and retrained the models to account for the “regime change”

5. Model Refit Frequency • Most clients require a model refit weekly. • Some require nightly • Models tested using multiple criteria to determine which variables are changing and what this means • We are always hungry for new variables and new ways of expressing existing variables

6. Printed Circuit Board Manufacturer • Which customers are at risk of defection? • Which customers are potentially being underserved? (upsell potential) • Which customers should we focus on today? UK based client—Customer Churn Model

7. Phrasing the Predictive Task • Given all surrounding data points, what is the probability that a given customer’s purchase amount in the next 6 months will meet or exceed their purchase amount in the last 6 months? • What is the probability that a customer’s satisfaction with the manufacturer is declining? In a specialized industry, purchases don’t imply satisfaction.

8. Probability of Growth

9. Probability of Growth • Low Probability of Growth— • Just placed an unusually large order or • Orders have been consistently slowly declining or • They have been receiving late orders, or bad customer service

10. Probability of Growth • High Probability of Growth— • Haven’t ordered in 6 months or • Consistently increasing order amount and frequency or • Increased contact for quotes, special requests, prototypes

11. Actionable Insights

12. Looking Forward • Salesrep Scorecard • Sentiment analysis on email and salesrep notes data • Industry analysis for pursuing new business

13. Wireless Zone—Ponte Vedra • Verizon Wireless Sales / Service • Facilitate multi-level marketing campaign • Develop product lifecycle analysis • Create upsell prediction engine for POS system

14. Phrasing the Predictive Task • How many phones of specified type will be sold in a certain interval? • Who is most likely to purchase, given they receive a mailer? • Which customers are most valuable? (Generate the most revenue per year) • What is the most likely upsell opportunity?

15. Product Lifecycle

16. Identify Most Profitable Customers

17. Reduce the Mailing Footprint/ Costs St. Johns County: Previous Mailers went to most of these zip codes Targets of first new customer acquisition campaign Customer Base across 3 most saturated zip codes

18. Targeted Mailers

19. Results of First Campaign

20. Looking Forward • Use the results from marketing campaigns to drive upsell / cross sell opportunities at the point of sale • Begin analysis of results from email outreach to strengthen product suggestion engine

21. Virtual Banking—Bank for Banks • Client Satisfaction Model—Currently Very Low Churn • Which clients are less satisfied than others? • Which clients would be receptive to upsells? • Which clients should we use for referrals?

22. Phrasing the Predictive Task • Note: Customer Satisfaction scores are provided quarterly by all client banks (Red, Yellow, Green) • What is the expected use frequency for certain products for a bank this size? • What is the expected technical issue frequency for a bank this size? • What can we do to increase customer satisfaction levels for each client?

23. Four “Pillars” of Data These data can be separated into 4 distinct categories • Institutional Information • User Behavior and Entanglement • Project Frequency and Technical Issues • Client Support

24. Institutional Information • Revenue • Bauer Score • Annual Survey • Conference Attendance These are fixed values which a Relationship Manager (RM) can not necessarily address, but which contribute significantly to the satisfaction rating of a client, and may be indirectly influenced by other RM actions

25. User Behavior and Entanglement • The number of users by category. • How many users are utilizing Online Banking? Mobile Banking? How does this compare to clients of comparable size and revenue? • Is the number of users for any specific service significantly increasing / decreasing? As with Institutional Information, these values are fixed to an extent. An RM can not increase the number of users, but could suggest projects that are more successful in comparable banks.

26. Combining Fixed Variables—Generating an Expected Value • By combining the attributes of the bank which can not be changed by RM intervention we generate an expected satisfaction level by assessing all banks with similar attributes. We then consider all attributes including the client specific case and survey data to generate actual satisfaction scores.

27. Project and Development Frequency • Project Frequency: Number of new projects per client per year. The average amount of time between new projects. Is this client more reluctant to begin new projects than similar clients? • Proportional Number of Issues: The number of project issues with respect to number of projects. Does a given client have more or less project issues than average client of comparable size? • Resolution Time: The average time taken to resolve a project issue. Is the average case age for the client similar to what would be expected for a client of comparable size?

28. Technical Support / Client Support • Questions versus Problems—How often does the client call in with basic questions relative to the number of services they subscribe to, and how often do they require service? • Is the client getting the expected turn around time for problems, and are the frequency and type of problems to be expected? • Number of Fraud Reports—Time to resolve • Client Tech Support Satisfaction Surveys

29. Model Output for each Client

30. What we look for in a Client • Maintain an enterprise transaction system (point of sale, accounting, enterprise resource program etc.) • Allows for easy access to data to build predictive models • Allows for easy ability to deploy models into the customers work-flow • Single owner or closely held • Active and focused leadership benefit the most from predictive modeling • Provide consistent and iterative feedback to make the output more relevant to business making the model better • Limited IT resources • Provides for greater dependency on NLP Logix to deliver information • Lessens barriers to sales and product delivery • Operate in a competitive market place • Predictive modeling is a powerful competitive edge that is measurable. • Once a company has adopted predictive modeling into their work-flow, it is very difficult to leave • Have a need for advanced analytics • Data scientists are in very high demand and low supply at this time.  This trend appears to be accelerating, as the need for advanced analytics grows.

31. Levels of the Enterprise Use of Data Improved Decision Making Optimization Predictive Modeling Statistical Analysis Alerts Increased Business Value Current enterprise data analytics systems Query Drill Down Standard/ Adhoc Reports

32. Things to Remember • In most cases this becomes an iterative process. No models currently in production are as they were described in Phase 1 of the journey. • The end user will see the results then add additional variables that they hadn’t deemed pertinent until they saw the model in action. • Insight from one model always leads to inquiries about the next

33. Thanks!! Questions??