1 / 35

Tuning The Money Funnel

Tuning The Money Funnel. Measure. Change. Measure. OR. Guess. Change. Guess. Background : Larry Mellon 20+ years of engineering @ scale. Research era. Project Jade & ARC : distributed system research Distributed computing, 1983+ Distributed workspaces & knowledge acquisition

step
Download Presentation

Tuning The Money Funnel

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tuning The Money Funnel Measure Change Measure OR Guess Change Guess

  2. Background: Larry Mellon20+ years of engineering @ scale Research era • Project Jade & ARC: distributed system research • Distributed computing, 1983+ • Distributed workspaces & knowledge acquisition • Jade Simulations: parallel simulation • Optimistic computation OS for supercomputers • Fault-tolerant cluster computing • Synthetic Theatre of War: training virtual worlds • DARPA: 50,000+ entities, real-time virtual worlds • ADS, ASTT, HLA & RTI 2.0, interest management & cluster computing Wife era • EA (Maxis): The Sims Online, The Sims 2.0 • Scalable simulation architectures • Automated testing to scale the development process, player & performance metrics • Emergent Game Technologies (CTO) • Architect for scalable, flexible MMO development

  3. Darius Kazemi Orbus Gameworks darius@orbusgameworks.com

  4. Panelists

  5. Why aren’t we using more metrics in game design, production and operations? • TRADITION: we’ve never used many metrics • It used to be hard to get data

  6. Why do we need to switch to Metrics DrivenDevelopment? • MMOs are a dollars in, dollars out business over ~10 year lifecycle: tune your money funnel! • Use CPM to reduce cost & increase revenue • Need to stay nimble in the marketplace • Observe and respond to player behavior

  7. Business models are changing: How can metrics affect MMO models? • MMOs are a service industry, not a boxed goods business • That means multiple recurring revenue streams • Customer retention is king • Content refresh rate needs to be high, and reflect current player interest

  8. ~500K SLOC & ~1Gig Content (1 CPU & 1 GPU) ~5M SLOC & ~10Gig Content (multi-core CPU & GPU) The problem: small & simple have become big & clumsy Team Size ~5 to ~50 (tightly knit) people ~50 to ~300 (loosely coupled) people Implementation Complexity

  9. Next Gen Games • Increased Complexity • Increased Complexity of Analysis Art from “Fun Meters for Games”, Nicole Lazzaro and Larry Mellon

  10. Next Gen Games

  11. How will metrics affect game play and game designin the future?

  12. ~ $10 per customer Tuning the funnel: Lower dev & ops costs Profit… New Content Regression Customer Support Operations

  13. ~ $10 per customer Tuning the funnel: Lower dev & ops costs Profit… Lower New Content Cost Regression Customer Support Operations

  14. ~ $10 per customer Tuning the funnel: Lower dev & ops costs Profit… Lower New Content Cost Lower Testing Cost Customer Support Operations

  15. ~ $10 per customer Tuning the funnel: Lower dev & ops costs Profit… Lower New Content Cost Lower Testing Cost Happy Customers Don’t Call Operations

  16. ~ $10 per customer Tuning the funnel: Lower recurring costsWhat tuning factors are useful? Profit… Lower New Content Cost Lower Testing Cost Happy Customers Don’t Call Operations Lower bandwidth & CPU

  17. Tuning the Incoming side of the Money Funnel • How do metrics help • Optimize income per user • Increase the number of users? • Retain users over time? • Other industries report: • MUCH easier to retain a customer than recruit • ~80% of Harrah’s Casinos revenue comes from ~20% of players

  18. The “3P's”: of game metrics Player Performance Process

  19. Metrics-Driven Development:each group needs different metrics Production • Time on task • Fun zone • Dead zone • … Metrics Designers Operations Engineers

  20. Metrics-Driven Development • CPU load per event • Lag time under load • … Metrics Engineers

  21. Engineering Metrics:Aggregated Instrumentation Flags Trouble Spots Server Crash

  22. Metrics-Driven Development • Number of each type of packet, over time • Client failure rate • Number of players per CPU • … Metrics Operations

  23. Metrics-Driven Development Production • Percent of world terrain completed each month • Number of animations per month • Number of automated tests that pass each month • Broken build time wastage • Number of supportable clients each month • … • MUCH more valuable if you share these metrics team-wide! Metrics

  24. Tuning imbalances or exploits can throw your entire economy out of kilter, but remember to triangulate!

  25. Tuning imbalances or exploits can throw your entire economy out of kilter, but remember to triangulate!

  26. First sample Now Measurable targets & projected trends give you actionable progress metrics, early enough to react Target Risk! Any metric (e.g. # clients) Time Any Time (e.g. Alpha)

  27. Bug introduced Feedback takes hours (or days) Unstable builds are expensive & slow down your entire team! Development Checkin Repeated cost of detection & validation Firefighting, not going forward Build Impact on others Smoke Regression Play test

  28. Rate of occurrence within a play session Name of sound Correlated data is much more valuable than single-source data Click rate Motive decay Time Time Player & game actions Minute 1 open door cook food Minute 2 fire breaks out Screenshots!

  29. Metrics change how you work! Measure Change Measure OR Guess Change Guess

  30. War stories: What metrics have saved your bacon? • Larry • Engineer efficiency: Compile / load / link times • System: Non-determinism level and location • Bug frequency per source code file • Team iteration rate: Build times & failure rate • Panel?

  31. How to boot strap metrics on your MMO • If you want to get an executive’s attention to justify a metrics project, show how it will save them money! • Server cost, Network cost, … • Cost of content, cost of customer service, cost of customer acquisition, cost of customer retention, cost of QA, ... • How did this panel get started with metrics?

  32. Where do you get your metrics from? Indirect (asking players) Direct (observing players) Data Mining (Game, CS, bugs, …)

  33. Implementation complexities • Being online lets us pull data from client side, live servers, the persistent database, customer database, etc. • We have a broader array of data that we can deploy against our problems than ever before! • But, volumes of data effects collection and analysis: how do you deal with collection, correlation and distribution? • Meta-metrics! • Flexible reporting interface!

  34. GIGO: vocal minority & metrics bugs How do you deal with the Garbage In / Garbage Out Problem? Silent Majority Measuring the wrong thing, or measuring it the wrong way, can lead to the wrong result! Calibration and Triangulation are your best friends Metrics are an attention funnel! Valuable feedback to team (“My work made a visible difference!”) Love It! Hate It!

More Related