Guided Content Optimization for Gears of War 3

2. Guided Content Optimization for Gears of War 3 Michael Noland Epic Games michael.noland@epicgames.com

4. <Gears 3 Beta Gameplay Video>

5. Overview • Our iterative process and some tips/tricks • Memory feedback loop • Performance feedback loop • Texture streaming improvements • Wrapup

6. Our Process and some tips

7. Information flow • Capture and process data • Generate action items • Fix issues and retrigger captures • Keep iterating • Streamline ID / capturing process for QA • Change information captured as new areas of interest are identified

8. Current process • Daily memory/perf standup • ~ Weekly reports on all maps • Quick turnaround on maps in flux • Periodic analysis of other captures • Continuous play-thru runs • MP soak tests • Cooked data report mining • Always loaded content packages

9. Implementation tips/tricks • Classify dark memory • Ideally traceable all the way back to a placed actor in the editor or a defined subsystem • Play thru maps on target as early as possible • Use artificial memory savers if necessary • reducepoolsize, nosound • Report adjusted numbers to keep your eyes on the goal • Establishes an ongoing baseline • BugItGo – Quick way to express a game location

10. Other data sources • Always loaded content pulled in by script • Periodically scrape; savings here help everything • Automatic cooked data reports • Top 10 lists showing large content items • Per level audio and animation breakdowns • ‘Content on-disc’ tags fed back to editor • Quickly narrow down which version is referenced • Fresh data available every day

11. The Memory feedback loop

12. Memory feedback loop • QA plays thru individual levels capturing data • QA runs a tool to process the raw data into CSVs • QA generates and sends out a hotspot report showing the overall trend and specific dips • Content or code makes changes and requests an updated report from QA

13. 1. Capturing raw data • memleakcheck – Logs out: • Loaded objects • Streamed in levels • Various other stats / metadata • Can be done fast or full • domemleakchecking 30 • Does a fast memleakcheck every 30 seconds

14. 1. Capturing raw data (cont.) • Reboot console • Turn on memleakchecking and on-screen level streaming report • Load a level and play thru it • Start playing the next level until all of the test level finishes streaming out

15. 2. Parsing raw data • MemLeakCheckDiffer • Extracts information from unstructured .memlk • Loads a grouping script that defines how to bucket objects together and in what order to report them • Two click processing for QA • Generates .CSV reports

16. 3. Example Hotspot Graph

17. 3. Hotspot Report • Lists lowest memory dips, with context • Loaded streaming levels • Commonly important bucket sizes • Static meshes • Skeletal meshes • Animation data • Sound data • BugItGo location

18. 4. Action Items • Sent to a fairly broad list: • Level LDs / artists, lead LD • Strike team (memory/perf programmers and QA) • Producers and game team • If dips are not well explained by an increase in LD buckets, someone will investigate further • Generated CSVs list all loaded assets • If still not accounted for, it’s dark memory that needs to be fixed

19. 4. Content Fixes • Loaded levels list is often enough • Cinematic levels left loaded too long • Levels not streamed in/out at the right time • Expensive assets in a geographically large submap • Transitions between maps can be problematic • Can look at a list of loaded assets • Merge/replace unique assets with library items • Stop non-visible assets from being pulled in • Actual cuts of noticeable content are fairly rare

20. 4. Code issues • May end up being a code problem or leak • Track genuine memory leaks with Malloc Profiler • Grow-only caches and pools are more common • Characterizing the expected amount is important • Round-trip thru an exhaustive level and back to the menu • Can factor into desired memory bar and ignore • Reset / purge when possible but plan for worst case

21. 4. Code improvements • Assets that are larger than expected • Re-evaluate your assumptions • A new game means new usage patterns • Some real world examples • Static mesh collision data • Blocking volume brushes • Improving animation compression • Non-uniform light environment samples

22. Balancing Act • Always consider memory savings in concert with performance • Trading off memory for speed (in either direction) only helps if you have room in the other

23. Lessons from the past • Went back to Gears 2 and retroactively pulled some stats from it to guide us thru Gears 3 • Full memlk runs for all SP maps • Gives approximate budgets, per-map slack • Used as approximations, not hard budgets • New code optimizations give us more bang / MB • New features and fixed costs take away some too

24. It’s working!

25. Performance feedback loop

26. Performance Feedback Loop • Gather appropriate profiles • Run once to identify problematic areas and then perform the appropriate profile captures • stat unitmax shows the time broken down into: • Render thread • Game thread • GPU time • Generate action items

27. Performance Feedback Loop • Capturing data • Capture on same run if the problem is steady-state • Otherwise, reload a checkpoint or start the level over • Files generated depend on the bottleneck • Render thread or game thread • Native CPU trace • Stats capture • Game thread • Gameplay Profiler • GPU • Native GPU trace (PIX)

28. Game thread • Associated profiles • Gameplay Profiler captures • Stats captures • Native CPU traces • Attack strategy • Depends on the actual bottleneck • Animation, Line checks, Pathfinding, etc… • Turn off expensive / low impact features • Optimize game code • Cut or modify triggering content

29. Stats Viewer • Counters or timers • Lots already in place • Easy to add new or temporary ones

30. Stats Viewer • Stats are hierarchical • Relatively low overhead to record

31. Gameplay Profiler • Allows seeing script and native game code in context; great for tracking down hitches

32. Render thread • Associated profiles • Native CPU trace • Stats Capture • Most commonly an issue in split screen • Attack strategies • Reduce draw calls • Turn off bad shadow casters • Merge sections • Code fixes for unexpected cases

33. GPU time • PIX • Works like magic with a solid marker setup • We mark each drawcall with: • Material name • Mesh / Skelmesh name • Actor name • Hierarchy of rendering • Prepass, Shadows, Opaque, Transparent, etc… • Enabling shader PDBs

34. Texture streaming

35. Texture Streaming • Eliminating texture streaming artifacts • Two main kinds: • Insufficient streaming time • Enough memory but not enough time to fill it • (level loads / transitions) • Totally over budget • Content uses too many unique textures

36. Texture streaming stats • Desired texture memory is updated constantly • Log out stats via memlk or ustats to look for over budget cases • QA also bugs locations with noticeable blurriness or mip popping

37. Isolating streaming issues • BugItGo to the bad spot on console or editor • Texture stats view shows which textures are loaded and how many mips • Can be connected to a remote console • Streaming system has a command to dump info on a particular texture • Lists everything it knows about a texture • Which objects reference it • Desired mip level calculation

38. WRAPUP

39. What’s next • Continuing to combat dark memory • Automatic capture file management • DB upload • Associated metadata • Automated reporting after content checkins • Empowering users to look at lower level data

40. Empowering users: Memory Analyzer

41. Conclusion • Gather the information as early as possible • Filter it down into actionable information • Tighten the feedback loop • Keep iterating

42. Any questions?

44. References • Many of the tools mentioned are documented in more detail on UDNhttp://udn.epicgames.com/Three/WebHome.html • Hotspot reporting processhttp://udn.epicgames.com/Three/HotSpotReportGeneration.html • UE3 performance and memory hubhttp://udn.epicgames.com/Three/PerformanceAndMemory.html