Remote-Rendering related research at Hasselt University (B)

1. Remote-Rendering related research at Hasselt University (B) dr. Peter Quax Expertise Center for Digital Media Multimedia and Communication Technology group

2. Remote Rendering for mobile devices • Results of the national project ‘architectures for mobile community content creation’ • ’03/’04 • Create a virtual city guide for use on PDAs and PC • 3D recognizable visualization • User-generated multimedia content • Community features • Communication

3. Remote Rendering for mobile devices • Issues • Only a single device capable of rendering accelerated 3D graphics (Dell Axim x51v) • Scenes could not be rendered in software (large textures) • Deployment on a 3G network (WiFi did not cover the entire city) • Axim X51v included Intel Graphics accelerator • No 3D engines available, create custom render engine based on OpenGL ES 1.0 • Local rendering provided very good user experience (formal testing), but devices too expensive to deploy on massive scale

4. Remote Rendering for mobile devices • Solution for low-end devices • Create a custom remote rendering system to enable hybrid deployment (local and remote rendering) • Rendering is separated from the application logic for scalability purposes (remote render is just another user to the system) • Use off-the-shelf PC hardware to render 25 viewports at once (in QCIF), segment the individual views and encode them in parallel • Limit user freedom (like shooter on rails) to guarantee Quality of Experience • H.263 QCIF transmission at 15fps on early UMTS networks was feasible • Conclusion : works fine, but rapidly outdated by local rendering solutions on mobile devices

5. Remote Rendering andInteractivity Requirements • One of the most challenging contexts for interactive applications : multi-player games • Number of studies available (see NetGames series of workshops), show that most demanding genre is First Person Shooter • Most popular genre among hard-core gamers • High number of interactions per time frame • Experiment objective : determine whether there is an indication for a boundary above which players become aware of network delay • Objective (in terms of score) • Subjective (in terms of experience) • Experiment in context of access network delay, but results also apply to remote rendering of these games

6. Remote Rendering andInteractivity Requirements • Do players perform worse when their network connection to the server is delayed (vs no impairment) ? • Objective measurement Yes !

7. Remote Rendering andInteractivity Requirements • Do players notice when their connection to the game server (simulating the access network) is delayed ? • Subjective measurement Yes !

8. Remote Rendering andInteractivity Requirements • Is there a threshold that can be defined ? • Below 60 ms, players are less likely to feel an impact (subjective and objective) • Subjective and objective measurements match • If the access network has a delay of 60ms, there is little hope for a remote rendering solution to be viable under these conditions (will only get worse) • Conclusion : look at other use cases (in gaming context) for Remote Rendering besides First-person shooter games

9. Remote Rendering for Massive Multiplayer Games • Architecture for Large-scale Virtual Interactive Communities (ALVIC) • Design a scalable client/server based solution for MMOGs (main focus on MMORPG genre) • Emphasis on practical issues • Deployment (firewall issues, delay experience) • Economical considerations (cost of deployment and maintenance) • Manageability (moderation, security, cheating) • Suitable for deployment on cloud platforms • As generically applicable as possible

10. Remote Rendering for Massive Multiplayer Games • ALVIC-NG uses a layered approach • Ensure scalability of each layer • Additional entities compared to traditional setups : proxy layer • Goal • Reduce the number of connections between clients and server(s) • Perform caching where possible • Be extendable towards Remote Rendering for low-end devices (media players, dedicated set top boxes,…)

11. Remote Rendering for Massive Multiplayer Games • What is unique about the proposed architecture ? • No sharding/instancing (like WoW) • Dynamic scalability • Servers assignment to spatial subdivision scheme is not fixed (like in Second Life) • Can scale from single to hundreds of servers without service interruption • Responsibilities are shared between clients (mostly static) and servers (everything dynamic)

12. Remote Rendering for Massive Multiplayer Games • Applicability of Remote Rendering • Client is relatively ‘dumb’, mainly concerned with visualization and local interaction handling • Proxies perform boundary testing and act as message gateways • Easy to add another shell of remote rendering servers to the architectural design • Transparent transition between local and remote rendering cases (e.g. from PC to mobile device or low-end console) • Decoupling of game logic and remote rendering infrastructure • Conclusion : more generic solution for an entire class of games

13. In-network adaptation of Remote Rendering streams • Growing interest in so-called Spectator Mode • OnLive, Xbox Live,... • Generate stream once, serve large number of viewers • What about heterogenous contexts ? • Multitude of devices (high-end PCs, thin client consoles, tablets, smartphones,…) • Multitude of access networks (100Mbit cable, 3G HSDPA, EDGE) • Create a reusable infrastructure for multiple purposes • Not limited to Remote Rendering (generic video streaming-compliant) • No adaptations needed to remote rendering system • Can be applied to off-the-shelf remote rendering solutions or video services

14. In-network adaptation of Remote Rendering streams • Reduce complexity required and load factor on remote rendering setup • Generate a single high-quality streams for all viewers • Intelligence required on two levels : • Network intelligence : bandwidth availability, channel quality, transit delay,… • Application intelligence : window size, user attention, capabilities of device,…

15. In-network adaptation of Remote Rendering streams • Adapt streams on-the-fly using an overlay network of intelligent proxies (NIProxy) • Place the adaptation nodes at the edge of the network (integrated at head-end of access network) • Proxy exchanges parameters (network & application) with client • Transcode video on demand, without server intervention (or use scalable video codecs)

16. Efficient delivery of omni-directional video • Omni-directional or panoramic video allows exploration by the user in recorded video sequences • Move orientation of the camera after capturing • Zoom in/out and maintain quality level • Like street view, but video-based • Distribution of these sequences is subject to optimization • Not really required for low quality sequences (e.g. 720p or less) • Sequences captured by our hardware are (typically) above 8000x1900 pixels • Cannot be streamed at once to a client (Set Top Box or Tablet device)

17. Efficient delivery of omni-directional video • Borrow ideas from remote rendering to optimize the flow • Perform pre-processing steps on the server to generate compliant streams • Do the precise customization for each user on the device (fat client) vs the server (thin client) • Retain interactivity (camera movement) as is typical in RR setups • Recompositing the scene for each client at server-side is not very scalable, so 2 alternatives : • Add meta-data to the stream and customize streams by manipulating the compressed bitstream (using H.264 slices) • Cut the generated stream into small segments that are individually decodable, have client request only those required

18. Efficient delivery of omni-directional video • Pre-processing steps for scalable delivery using segments Original sequence Parallel encoding and HLS-segmentation hardware (multi-core servers) Output files One file per quality level, per minute, per segment

19. Efficient delivery of omni-directional video • Pre-processing steps for scalable delivery using H.264 slices Original sequence Encoding hardware Output file One file per quality level

20. Efficient delivery of omni-directional video • Delivery to end-user over standard protocols • Easily scalable back-end • Server does not perform full rendering sequence, but composes custom viewport using slices (defined in the compressed domain) or delivers segments on demand Cluster of custom servers 3 streams: oneforeachresolution RTP-baseddelivery Viewportcropping

21. Efficient delivery of omni-directional video • End-user application • Based on (IP) Set Top Boxes • Second Screen applications (besides primary TV broadcast) on tablets (android/iPad) • Web platform (WebGL with HTML5 video features) • Currently in testing phase, will be rolled out on testbed in short term (EOY)

22. More information ? • "Adapting a Large Scale Networked Virtual Environment for Display on a PDA". Tom Jehaes, Peter Quax, Wim Lamotte, Proc. of ACE 2005, Jun. 2005. • "On the applicability of Remote Rendering of Networked Virtual Environments on Mobile Devices" . Peter Quax, Bjorn Geuns, Tom Jehaes, Gert Vansichem, Wim Lamotte, Proceedings of the International Conference on Systems and Network Communications, IEEE, Oct. 2006. • "Objective and Subjective Evaluation of the Influence of Small Amounts of Delay and Jitter on a Recent First Person Shooter Game". Peter Quax, Patrick Monsieurs, Wim Lamotte, Danny De Vleeschauwer, Natalie Degrande, Proc. of NETGAMES2004, ISBN 1-58113-942-X, Aug. 2004. • “Effective and Resource-Efficient Multimedia Communication Using the NIProxy “. Maarten Wijnants and Wim Lamotte. Proceedings of the 8th IEEE International Conference on Networks (ICN 2009) • "ALVIC-NG : State Management and Immersive Communication for Massively Multiplayer Online Games and Communities" . Peter Quax, Bart Cornelissen, Jeroen Dierckx, Gert Vansichem, Wim Lamotte. Journal on Multimedia Tools and Applications, Special Issue on Massively Multiplayer Online Games. Volume 45, Issue 1 (2009), Page 109. Springer. http://www.edm.uhasselt.be peter.quax@uhasselt.be