Endowed Virtual Objects: Hyper-Realistic Object Properties for Virtual Environments

1. Endowed Virtual Objects: Hyper-Realistic Object Properties for Virtual Environments Robert W. Lindeman Worcester Polytechnic Institute Department of Computer Science Human Interaction in Virtual Environments Lab gogo@wpi.edu

2. Overview • Motivation of the problem • Current state of MMOVR • Proposed approach (at least a first pass) • Discussion R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

3. So, What's the Problem? • Not many of us build rich VR experiences • Mostly only use visuals and audio • Simple models/textures • Few buildings/objects • Mostly static, immovable environments • Simple character behaviors • Mostly single user • Mostly proprietary software • But we do… • Use interesting (though expensive) I/O devices • Conduct lots of empirical testing/comparisons R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

4. So, What are (some of) the Reasons? • Interesting (though expensive) I/O devices • We are not artists • We lack unified goals • But this is okay! • We lack a unified infrastructure (this is not okay) • No widely used "Windows API" for VR • No common entity description • We lack a market to drive big investment • Though the US military still has deep pockets, it is much more focused on short-term impact R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

5. Just a Second… • My definition of VR: • Fooling the senses into believing they are experiencing something they are not actually experiencing [Lindeman, 1999; Foley, 2007] • Here I mainly focus on systems with: • A potentially large number of players • Lots of manipulatable objects • I call these Massively Multi-player Online VR (MMOVR) systems • This work also has great benefit for smaller, single-player systems R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

6. Anatomy of MMOVR Systems • Server • Maintains state • Object locations/states • Player locations/states • Updates things • Object movement (e.g., physics) • AI • Disseminates state to clients • Client • Presents state to player • Visual rendering • Audio output • Handles input from player • Sends updates to server R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

7. Keys to Success • High fidelity (or realism) • Graphics, audio, haptics, behaviors, etc. • Low latency • Tracking • Collision detection • Rendering • Networking • Ease of use • Low cumber for users • Easy integration for programmers • Compelling Content • Get artists involved! R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

8. Current VR Systems • In most current VR systems, a monolithic approach is used • Everything is done in the render loop • Some threading used for expensive computation • Built by CS types • Including much of the art • Input by experts • HCI people • Domain experts (e.g., PTSD people) R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

9. My Monolithic System • TactaVest integrated with the C4 game engine R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

10. Big Things Happening Now • Multi-core CPUs • Instead of clock-speeds doubling, the number of cores will double every 18 months • Graphics Processing Units (GPUs) • Growth rate is much faster than CPUs • Cheaper to upgrade/add more • Mobile users • Variable display technologies • Variable network connections • Increased player sophistication • WoW, Halo3, GTA4, etc. • I will discuss this in terms of "Today" and "20 Minutes from Now" R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

11. Taking Advantage of New Stuff (1) • Multi-Core • Need to make infrastructure much more multi-threaded, multi-processing • Example: Movement of a bird • Today: One controller handles all birds • 20 Min: Each bird has its own controller • Example: Rendering • Today: One renderer for all graphics • 20 Min: Interest or importance rendering • Related: Can we push some "server" processing down to powerful clients? • Trust becomes an issue R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

12. Taking Advantage of New Stuff (2) • Graphics Processing Units (GPUs) • They are improving at a much faster rate than CPUs • They can be added more cheaply • They can be replaced much more cheaply • Offload as much as makes sense to GPUs R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

13. GPU vs. CPU Growth [Ian Buck, “Brook for GPUs”, SIGGRAPH 2004, talk slides: http://graphics.stanford.edu/papers/brookgpu/] R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

14. Taking Advantage of New Stuff (3) • Mobile users • Variable display technologies • Variable network connections • Provide hardware-appropriate experiences • Multiple model representations • Multi-modal Level of Detail (MLOD) • Texture qualities • Sound qualities • Methods for defining and requesting individual (or classes of) properties • Provide network-appropriate communication • Semi-autonomy for peers R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

15. Taking Advantage of New Stuff (4) • Increased player sophistication • The bar went up a long time ago! • We need to provide tools for non-CS types to create and incorporate content • We have modeling and animation tools • Need a way for content creators to make and disseminate content • Need a way for world-builders to use pre-made content R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

16. Programmer Call Sequence: Today Main program loads the scene into memory For each processing-entity (renderer, collision- detector, AI-master) Call the appropriate API functions so that each entity can construct its own representation of the scene Update loop starts Read input sensors For each processing-entity Call the appropriate API function with the new data Update loop ends R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

17. Programmer Call Sequence:In 20 Minutes Main program loads the scene from the Object Repository, constructing a unified representation (the Runtime Repository) Update loop starts (really a nop) Processing entities read and/or update object properties Processing entities "display" their stimuli to the user Update loop ends R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

18. VR-Object Properties • Multi-Modal Properties • We need to think of objects as they exist in the real world • Each object should have visual, auditory, haptic, etc. properties • A Unified (Multi-modal) Scene Graph should be used • Renderers (processing entities) walk the same graph • Behavioral Properties • AI, physics, etc. scripts should accompany each objects R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

19. Object Repository • A kind of "Wikipedia" of objects • Hierarchical ontology • Bird -> Eagle Turkey • Each entry contains (or inherits) • Multi-modal description • With multiple MLODs • Behaviors R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

20. VR-Object Usage • A world builder (person) defines a scene by grabbing objects from the Object Repository • These are loaded into the Runtime Repository when the system starts • Multi-modal properties are loaded into material nodes in unified scene graph • Behavioral properties are used by processing entities • Example: Birds • Artists build models, do some scripting • Programmers do some coding R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

21. Universe Transformation Transformation Viewer Material Transformation Table Material Material Mug Coffee Material Nodes on Steriods • Associate multi-modal properties in Material Nodes R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

22. Video of Unified Scene Graph Shailesh Gattewar, 2001, MS Thesis, The George Washington University R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

23. Main Program (Dispatcher) Output Manager (Visual) Filter Manager Bird Thread Object Repository Ouput Manager (Sound) Filter Manager Input Manager (Tracker) Filter Manager Endowed Properties 1.0 • General structure Processing Entities R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

24. Endowed Properties 1.0 (cont.) • Developed by five undergraduate students as their capstone project in 2007 • Clear separation between state maintenance and state presentation • Used game engine (C4) as front end • Hybrid client/server peer-to-peer network architecture • Simple multi-player paddle-ball game as proof of concept R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

25. Endowed Properties 1.0 (cont.) R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

26. Research Problems • Modalities are different • That's why we have separate scene graphs • Need to provide a synchronized experience • Rob: Hit the table! • Need • Unified description (maybe X3D?) • Place to host Object Repository • A way of selecting which properties the client is interested in (to avoid object bloat) R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

27. Summary • Multi-core CPUs • Split system up into very small, well-defined threads/processes • Use a message-passing layer to promote interoperability • Graphics Processing Units (GPUs) • Off load what you can • Mobile users • Separate state management from presentation • Increased player sophistication • VR Object Repository • Provide tools to artists! R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development

28. Discussion? • Rob Lindeman • gogo [at] wpi.edu • http://www.cs.wpi.edu/~gogo/ • http://www.cs.wpi.edu/~gogo/hive/ • Room: N21 R.W. Lindeman - WPI Dept. of Computer Science Interactive Media & Game Development