Ken Demarest Ken@demarest Kdemarest@eidos

1. Code and data structure implications when patching without local storage… and resource management overall. Ken DemarestKen@demarest.comKdemarest@eidos.com

2. Background on this talk • This is a thought experiment – it all started with Jon asking me what hard problems I had heard about recently… • Patching consoles… Turns out I never did it, so… • Focus on high concepts, ignores numerous practical issues • Make assumptions about a future that may not occur • Goal is to drive discussion, out-of-the-box thought • Gather some current best practices relative to future improvements • And maybe… perspective for console guys who might not have the depth of online experience PC developers have, and PC guys who are unused to storage limitations

3. Initial Poll • Who has ever delivered a patch to their game? • Who uses late linking of data? • Who has edit-and-continue in their data path? • What areas cause level resets? • What areas invalidate edit-and-continue? • Whose products support mods / plugins? • Who is console only (vs PC or console+PC)?

4. Situation • Next gen consoles (PS3, XB2, PSP, DS) will provide connectivity • Sending data to console games post-sale will enable new revenue opportunities • Patching (should never happen, of course) • Content unlock (fill every DVD) • Content extension (Lode Runner maps) • Community mods (console mods built on PC) • Marketing intrusion… I mean surveys, advertising, dynamic product placement, up-sell, cross-sell • Storage may be limited • Never as massive as PC storage – MMO ‘Patch Kings’ • Current PS2 no HD; Microsoft’s HD ‘nut’; trend to mem cards

5. Data Transfer Options • Require storage • But even then you won’t be the only app to utilize it • Send data every time • Plausible, but slow • Think of online connection as a ‘remote hard drive’ • You’ve got ‘more bandwidth than storage’ (ex: 60% broadband, 40% modem on PS2 today) • Conclusion: any data sent should be as small as possible… but how?

6. Reality Check • North American PS2 install base is ~30M • 3.2M connected users • Appears unlikely to break 30% connectivity in near- to mid-term • Europe still hamstrung • Japan uptake less certain - <1M today; Korea likely uptake • But all new PS2s have network adapter built in • Still, 30% of 30M is 9M users – the size of the largest PC game penetration, and about 18x the size of the MMO audience • Assuming XB2 leverages MSN/Zone/Passport user base, we could see big numbers here as well • So still a viable profit center

7. Considered as a patch problem, how do we keep data transfer small? • Today’s code & data layout often resists efficient patch sizes • Code fixups • Data aggregation, indexes, etc • The good news: • Simple techniques can ease the problems • They (might) ripple into greater build efficiency • The bad news: • Many techniques imply slower loads • Might have to change your fundamental resource management architecture • But hey, you’ve got a while to make these systemic changes

8. Code Options Warning: Absurd ideas herein for the sake of illustration • Stop fixup problems • DLLs (or equivalent – MS TDR) • Ship the obj files – link at run time • Fixed, predictable link order • Imagine an order-retaining linker • Post ship, name everything zz1, zz2, etc • Code hooks (like a plug-in architecture… or a mod) • In main loop, around mem allocations, root ancestor constructor/destructor, DLL detection, kernel process insertion, etc. • Code intercept installation on load • All facilitated with a compact game state • Manual aspect-orientation • Crazy VMT redirect [applyDamage()]

9. Data has troubles too… • Sample data path: • Hot spots • Getting data game-friendly seldom has order guarantees • Some source data has Butterfly Effect on derived data • Derived data takes a long time to generate; often large (otherwise you wouldn’t need to pre-generate it) • Compression may cross assets (text?) • Internal linkage lacks persistent references • Concat indices often resemble code fixups • Embedded concats only make it worse Source Game- Friendly Derived Data Link Compress Concat Embed Illustrative – not necessarily in order; each step may also happen multiple times

10. Data Options • The biggies • Model data build on code build principles • Dynamic linking of data at (re)load time • Concat using relocatable, sector-based approach • (the irony of FAT32; sector sorting; tell FAT to order files?) • The others • Modularity and predictable ordering in all data, to facilitate a diff • Compression optional (or diff-friendly via window borders)

11. When Sending Data or Code… • Date-stamped resources make atomic changes clear, QA-able • Diff then compress before send • Merge at load time (no local storage, remember?) • Only send when game-relevant

12. Veering to a related thought experiment… • Lets suppose that you never had to QA anything, or worry about load times • Your end-users could theoretically get build results just like the team does (suspend disbelief, please… and Yes, MMOs appear to do this already…)

13. Hey, quality upload == quality build? • Conceptually, delivery of patches/upgrades/etc to customers is no different than delivery of build updates to your team (except for QA, load times, delivery infrastructure, blah blah) • HL2, Halo2, Oddworld all have facets of the data delivery techniques discussed earlier • Trim resource management tends to yield faster builds • Tends to prepare you for efficient online delivery too • Mod community has had many of these techniques for years – yet projects still let their build environment stray from the faith, often for very valid reasons • How might these techniques point us towards ‘data delivery nirvana’ conjoined with ‘build environment nirvana’? Can TCP/IP distribution of code and data be unified across target audiences (team/players)?

14. A few pathological cases… • Artist adds wrinkles to a character’s forehead, and all players see it immediately • Dev team and the community make simo-changes to levels, and everybody sees it at once • Content rights (ownership) is a characteristic of both the dev team and players • i.e. what content you can see/play, as well as your rights to make changes • We’re heading over the edge, so why stop now…

15. Straw Man (1) Warning: Blatant impossibilities ahead. No throwing of rotten tomatoes, please. • All tools are also servers of their data and services via TCP/IP • Maya, Max, Photoshop, your crazy facial animation tool • Via a uniform format (SOAP? XMLRPC?) • Capable of evincing change deltas, not just entire data units • Via a perfectly uniform resource system with perfect pointer management • Game displays insta-results and multi-user simo editing on live levels • Late linking, and persistent symbolic links allow data-based ‘edit and continue’ • All code and data compilation is always distributed and cached • Data validation is so good it isn’t possible for the team to get stopped • Oh, and automated code unit testing makes this true for code as well • Derived data is never required – only tasked out then later loaded when ready • Just-in-time auto-concatenation (by the game?) groups data for most efficient load times

16. Straw Man (2) OK, maybe one small tomato • Validation extends to code • A bot plays through the levels before you see them, finds bugs before QA • Bugs are always fully repeatable, often across builds • Code and scripting both always link late, fast, dynamic re-link • The game never halts – the code is so robust it even saves game state across major recompiles, effectively never halting • Publish to team members and publish to players only vary in the amount of QA • In fact, player distribution is just a version control sync

17. Discussion • What in the straw man, ignoring constraints, is undesirable? • What are key obstacles to achieving the straw man? • What other high-concept, perfect-world systems would you like to see in a next-generation resource management / data delivery system? (or open discussion to asset chain overall)

18. Cost Analysis Poll • What % of tech time in your org is spent creating system (vs. leaf): 20%, 40%, 60%, 80%? • Data full recompile: <10 min, <30 min, <1 hr, <2 hr, more? • Code full recompile: <5 min, <10 min, <30 min, <1 hr, more? • Who runs metrics on code effort vs. lost team time?

19. Next Gen Tools Poll • Who journals, or uses techniques to improve bug reproduction? • Who uses unit testing? Expects to use it? • Focusing on techniques to debug multithreaded pain? • Fully generic layer for tweaking data values (with a GUI layer where needed)? • Who sees integration of 3rd party tools as a major part of their next product? • Other techniques?

20. Next Gen Resource Chain Stealth Talk Ken Demarest Ken@demarest.com