1 / 27

Analytics and Witch Doctoring: A Cure for the Black Box Mentality

Analytics and Witch Doctoring: A Cure for the Black Box Mentality. September 21, 2011 O’Reilly Strata Conference J.C. Herz Jnhq@yahoo.com. Analytics: Occult Phenomenon. Very powerful Don’t understand it Practitioners possess arcane knowledge. Secret Algorithms.

colorado-morse
Download Presentation

Analytics and Witch Doctoring: A Cure for the Black Box Mentality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Analytics and Witch Doctoring: A Cure for the Black Box Mentality September 21, 2011 O’Reilly Strata Conference J.C. Herz Jnhq@yahoo.com

  2. Analytics: Occult Phenomenon Very powerful Don’t understand it Practitioners possess arcane knowledge

  3. SecretAlgorithms

  4. Greek Letters Are Your Kryptonite

  5. High Status Helplessness • If you understood the technology, you’d be one of those people whose job it is to make technology work. • You know, underlings

  6. OODA Loop Which of these does an analytic tool/technology do?

  7. Data Due Diligence & Auditing

  8. Critical Question: Data Quality • How complete is it? • Ex: 600 custom fields, only two have more than 50% coverage • How accurate is it? How do you know? • How consistent is it? • Good test: make three calls to different parts of the company, to get an answer to a factual question that doesn’t require calculation.

  9. Precision vs. Accuracy When precision exceeds accuracy, you’re setting yourself up for analytic failure

  10. Critical Question: Half-Life of Data • How long is the data accurate? • How long is the data useful?

  11. Critical Question: Tempo • How fast will a decision be made on the basis of analytic insight? • Quarterly? • Daily? • Within seconds? • Milliseconds? • Never? • Realtime vs. Continuous vs. Batch

  12. Critical Question: Real World Context • Without real world data, “behavioral” metrics are misleading • Where is the transactional data that validates insights from non-transactional data? • How would you prove the magic analytics WRONG?

  13. Critical Question: What is the Validation Test? • Formulating the validation test keeps both the customer and the developer focused - and honest • Suggest pay for performance, and see if the developer or vendor freaks out. • Make sure validation is ongoing - in case the ground is shifting

  14. Critical Questions: Workflow • What workflow changes will this proposed capability require? • People hate changing their workflow, even if it’s an improvement • Never attribute to stupidity what can be attributed to laziness • What is your plan for changing workflow? How do you enforce it?

  15. What are you prepared to do?

  16. Critical Question: Consequences • What actions are you willing to take on the basis of validated analytic insight? • Change your product? • Change your marketing budget? • Change people’s job descriptions? • Re-allocate R&D budgets? • What actions are you not willing to take?

  17. Questions? • J.C. Herz jnhq@yahoo.com (202) 213-3151

  18. Technical Reality vs. Leadership Attention Span

  19. Business Payoff vs. Intellectual Appeal Market Segmentation 360º Lead Scoring Operations Research Competitive Intelligence Validate Marketing Effectiveness Social Network Analysis Pilots to test new analyst tools with tiny amounts of generic data

  20. The perfect application that no-one uses is still worthless

  21. Shame Shame

  22. Politics

  23. Before You Rip ‘n’ Replace:What is the exit cost of this technology?Does “turnkey” mean monoculture?

  24. Ugly Babies & Pretty Babies

  25. A Ballad of Spectacular Information Display • Time Magazine 1976 • Telex text routing: information off the wire goes to terminals, properly foldered • Z8 terminal display awes executives • Pneumatic system not eliminated

More Related