Saints Row Scheduler

2. Saints Row Scheduler • Randall Turner, Volition, Inc. • Randall.turner@volition-inc.com • Discussion Areas: • Saints Row project • Saint’s Row scheduler

3. Saint’s Row

4. Single Thread, CPU Bound

5. Alpha Kits • Two Hardware Threads • Dual Core • Hardware thread • Software thread • Many software threads on one hardware thread • Software threads != concurrency. Hardware threads do.

6. First Step: Alpha kit solution • Sim/Render Split (Still CPU bound, not as bad.) • Two HW Threads

7. Beta Kits • Betas = Six hardware threads, Woo Hoo! • But each hardware thread was slower than Alphas. • Homogeneous processing, unified memory

8. First Beta Soluton

9. Problem • Still only using mostly two HW threads (Sim, Render) • Badly CPU bound • No more major “natural” separation points

10. Interim Solution • Manually move processing to other HW threads. • Example – Shadow Calculations • Create a SW thread to process half of the object list • Process half the list when triggered.

11. Fixed Concurrent Processing • Uses an entire SW thread for one type of processing. • Wastes system resources • Hard coded to execute that processing type only. • Inflexible, Scales up badly.

12. Solution… Scheduler & Jobs

13. Goal • Make all movable so will use available space

14. Scheduler (Job Manager) • Jobs • Worker threads • Job Queues • Triggers

15. Jobs • Independently schedulable entity • No dependencies on other jobs. • Atomic with respect to other jobs • Essentially a function/data pair

16. Creating Jobs • Extract parallelism from loops. (Splatting) • Segregate memory usage • Thread safe interactions with systems • critical sections • double-buffering

17. Examples • Compute shadow silhouettes • Character animation • Object visibility culling • Building command buffers • Bad candidate - AI processing • Not easily turned into jobs due to interaction with large, difficult to thread safe data set.

18. Basic Scheduler Flow

19. Worker Threads • One per HW thread • Draws jobs from job queues • Associated with job “type” • Sim jobs, Render jobs, or “any” job. • “Soft” or “Hard” linkage • Dynamically configurable • Wakes up when jobs go into job queue • Can be deactivated

20. Job Queues • One per job type • Sim or Render • Implicit prioritization • High frequency, contention sensitive • Used critical sections on version 1.0

21. Triggers • Fire on completion of jobs • Scheduling Triggers – schedule more jobs • Event Triggers - activate suspended threads • Can tie together to create state machines.

22. First Pass

23. Final Pass • 90% CPU Utilization

24. Flow-Around • Callback to scheduler on interrupt • Allow “small” jobs to complete, then resume • Jobs contain duration info

25. Havok • Timestep called from Scheduler job • More flexible • Problem: Has own internal scheduler • Timestep Goes “Serial” • Jobs don’t flow • Solution: break out and splat serial processing ourselves.

26. Havok - serial extraction

27. Performance • Critical Sections • Scheduler v1.0 all critical sections • Blockage risk – CS owner preemption • Can stall ALL threads • Lock-Free • Faster – x2 no contention, x10-20 high-contention situations • NO blockage risk (Woo Hoo!) • GP Gems 6, Fober

28. Job Size • Scheduler overhead determines job size. • Small as possible. • Assume 5% overhead • Saints Row “ideal” size about 250 microseconds • 7 microseconds per job, 5% = 150 microsec, • 250 drops to ~ 3% overhed

29. Questions? For latest slides: • http://www.volition-inc.com/gdc07/

30. Extras • Pix with PVS on, night, high load • Release, w/o PIX samples, ~ 35ms

31. Lock Free URL’s • Fober implementation: • http://www.grame.fr/pub/LockFree.pdfhttp://www.grame.fr/pub/TR-050523.pdf • Toby Jones: • GP Gems 6 • Good coding style