Curs 7

1. Curs 7 Arhitecturi SW pentru RV Networked VE

2. Sistemele de RV sunt complexe Dezvoltarea unei aplicații complete de RV este o activitate care necesită manipularea mai multor tehnologii și rezolvarea sarcinilor in anumite limite de timp. • Pentru dezvoltarea unui sistem complet, o medie de timp satisfacatoare este de 12 luni. • Teamwork-uljoaca un rol fundamental.

3. Complexitatea sistemelor de RV crește • Cresc cunoștințele noastre cu privire la mecanismele de percepție umană • Progresele tehnologice sunt lente dar constante: performanțele generale ale sistemului sunt în creștere și blocajele sunt eliminate • Complexitatea tot mai mare a sistemelor rezultate

4. Componente multiple • Un sistem modern de VR este o colecție de mai multe module de înaltă performanță • Fiecare componentă are un domeniu de cercetare propriu, și necesită tehnologii dedicate, instrumente și abordări specifice.

5. Grafica 3D • "Interactivitatea" necesită un “load” foarte mare poligoane;Multe tehnici și algoritmi există (sau sunt dezvoltate) pentru manipularea optimă a bazelor de date foartemari cu date geometrice;

6. Collision Detection It used to be a “big thing” in VR. • Collision detection algorithmssearchesforvirtualobjectsinterferences • Algorithms are almost optimal – very mature research field • There is little point in collision detection without physic simulation • Revived a bit lately: GPU-based collision detection

7. Physics simulation Interaction between virtual objects need to be simulated with virtual physics • Objects in a VE should react to stimulus (gravity, external forces, collisions) • Compound objects need to be considered, and mechanical effects of joints between rigid bodies simulated. • Well known simulation models, but again the hard task is simulation in real-time. • Opportunities & challenges: • Parallel computing and dedicated co-processors. • Real-time simulation of deformable objects.

8. Positional sound • Sound sources can be: • Sampled: playback of a pre-existing files (with additional, post processing effects) • Synthesized: sound is generated on-line • Audio sources usually needs to be positioned in 3D space (and they move around) • Even when the sources are static the listener is moving

9. Interaction Devices Management Users may interact with a system using various types of devices and technology Interaction can be 3DOF, 6DOF, force enabled, thought enabled and more… New forms of interactions are emerging such as using portable BCI devices Complex devices require dedicated APIs and access paradigms

10. Haptic feedback • Haptic-enabled manipulaton involves the user’s body perception-action feedback loop: still a vastly unexplored research field with many technological challenge. • Advanced software techniques and hardware devices need to be developed to achieve realistic haptic rendering of rigid and soft objects • Haptic interfaces are usually custom objects and depends on the applications

11. Inertia feedback In some VR systems the user is placed on a moving platform Stewart platform: six independently actuated legs, where the lengths of the legs are changed to position and orient the platform.

12. Networking “No man is an island” communication between remote VR nodes can be very important. Interleaved data streams: audio, video, geometry transformations, haptics and more.

13. Developing VR apps: the big picture • VR is really about interaction. From a software point of view, interaction is a time-critical activity • Consequence: any VR application is a collection of high performance, low latency, dedicated tasks • Tasks are not independent. There are cross-tasks implications that should always be considered

14. What is the “state” of a VR application? • In theory: a collection of information about the virtual world • In practice: a collection of data structures and variables 3D meshes Forces VR World “state” Velocities Positions Accelerations Orientations

15. The state evolves over time… t0 t1 t2 t3 …advancing in a discrete sequence of steps and following the simulation computation.

16. Why does the state keep evolving? VR World “state” • Time_01! • Time_02! • Time_03! • Button_A_pressed! • Joy_1_MOVED! • MOUSE_OVER! • Mainly to accommodate the effects of two things: • External events (mainly due to user input/tracking); • Internal events (timers, network events)

17. Q: How frequently does the state evolve? Simulation step t0 t1 t2 t3 A: Fast enough to keep up with the fastest activity taking place inside the simulation. Consequence: the internal update rate of the simulation depends on the VR system configuration and the tasks performed

18. Loops in a typical VR application ~10Hz Haptics Physics >1kHz ~ 60Hz 100-200 Hz Graphics ~30Hz Network

19. The graphics loop The state of a VR simulation need to be displayed at regular intervals (framerate) on a monitor, HMD, CAVE, etc State(t) ≈60Hz Update frequency = monitor refresh Anything more would be wasted

20. The collision detection loop Checking if moving objects collide with each other State(t) ??? • If collision checking is not “fast-enough” some collision could occur undetected

21. The physical simulation loop State(t) >=100Hz Objects responds to physical laws and interact Update frequency needs to occur at least at the same speed as display rate, but stability issues or haptic rendering often requires much higher speeds

22. Real-time physics in practice (I) • A world is a collection of bodies • Bodies from different worlds never interact • Each rigid body has: • a mass (and moment of inertia) • needed for motion calculation • obtainable by composing elementary solids • a shape or “geometry” • needed for collision detection • obtainable by composing elementary solids Note: we are considering rigid-bodies physics

23. Real-time physics in practice (II) • Physics materials are used to describe surface properties of objects var material = sim.getMaterial(idx); material.restitution = 0.1; material.staticFriction = 0.8; material.dynamicFriction = 0.5; material.apply();

24. Real-time physics in practice (III) • Simple objects are connected using a collection of physical joints • Many types of joints are available • Sometimes joints can be broken

25. Real-time physics in practice The physics loop: 1 2 3 • Apply all the external forces to the scene objects • Ask the Physics engine to compute the evolution of the scene during a certain time interval • Get the results back • Update the scene-graph • *optionally* : control some external device

26. The haptic loop(I) State(t) >=500Hz Haptic refresh rate is much higher than graphics refresh rate! Stiffness in haptic rendering is influenced by loop speed -> collision detection speed

27. The haptic loop (II) Position (x,y,z) Force (fx,fy,fz) 1 2 3 • Read the device current position* • Depending on that position compute a force • Ask the device to apply that force * Reading the device velocity can be useful too

28. So, does this mean that physics must run at 500 Hz too? Physics State(t) 500 Hz 500 Hz? • Well..., depends: • For optimal and more accurate computation and rendering of forces, yes. • To reduce computational workload “virtual coupling” can in some situations be used.

29. The idea behind virtual coupling Physics Virtual Coupler State(t) 500 Hz 100 Hz 100 Hz • A virtual spring is attached between the VR object and the tip of the haptic interface • The object is considered inamovable between two consecutive physics loops • 500Hz computation is only performed to compute (and apply) the virtual spring elongation forces

30. The inertia feedback loop State(t) ≈100Hz Some VR applications use also a motion platform Device control is typically performed at around 100Hz

31. Controlling a motion platform • The platform controller waits for: • x, y, z (floating point) • Roll, Pitch, Yaw • The system has a high mechanical inertia Position commands takes a while to get executed

32. Computing perceived acceleration Washout filter • A physic engine can be used to compute position, velocity and acceleration of the “head proxy” of the user. • Data is passed to a Washout filter to get “transformed” Position, Velocity, Acceleration

33. The data flow Physics engine Washout filter >=100Hz ≈ 60Hz x, y, z Roll, Pitch, Yaw Position, Velocity, Acceleration

34. From acceleration to inclination Washout filter • For low-frequency accelerations, the washout filters tries to use gravity to simulate forces. Position, Velocity, Acceleration

35. Network communication loop State(t) Node 1 State(t) Node 2 10Hz – 500Hz Communication between remote VR nodes can be very important Update frequency depends on the type of communication (LAN,WAN), interaction (with or without haptics) etc…

36. Conclusion: loops are everywhere ~10Hz >500Hz ~ 60Hz 100-200 Hz ~30Hz Requisite I for Integration: A precise, possibly deterministic, fine control on all the loops

37. Variables are shared between loops VR World “state” Haptics Graphics Physics Requisite II for Integration :A good framework takes care of data synchronization

38. Same object, multiple representations Haptic shapes and and physics shapes Physics shapes only

39. Events: loops can’t model everything VR_EVENT_2 VR_EVENT_1 VR_EVENT_4 VR_EVENT_3 ~30Hz

40. A better general overview Events and loops are orthogonal concepts: Requisite III for Integration : a good framework should handle both.

41. So, writing VR applications can be hard • Usually requires excellent C++ skills. • Hi-performance tools and libraries need to be properly handled (otherwise performances might suffer). • Plenty of tools available, but mixing them is not trivial. Redundancy in files/data structures/code waste a lot of resources. A LOT of work goes into the framework. Software architectures for modern Virtual Environments

42. When we discuss networked nodes we can be referring to: • Physical connections • the wires between the computers • Logical connections • how the messages flow trough the nodes

43. Nodes communicates over a network

44. Fundamentals of Data Transfer (1) Network bandwidth • the rate at which the network can deliver data to the destination host (bits per second, bps) Network reliability • a measure of how much data is lost by the network during the journey from source to destination host types of data loss: • dropping: the data does not arrive • corruption: the content has been changed

45. Fundamentals of Data Transfer (2) Network latency • the amount of time required to transfer a bit of data from one point to another • one of the biggest challenges: impacts directly the realism of the NVE experience • we cannot do much to reduce it origins • speed-of-light delay • endpoint computers, operating systems scheduling • the network itself, routers

46. Fundamentals of Data Transfer (3) Network protocol • a set of rules that two applications use to communicate with each other A protocol defines • packet formats: understanding what the other endpoint is saying • packet semantics: what the recipient should do when it receives a packet • error behaviour: what to do if (when) something goes wrong

47. BROADCAST based architectures In the ideal large-scale NVE design all nodes can communicate with others directly Advantages: • Simple to implement • Players can join at any time Disadvantages: • Only supported by LANs • System management and update a nightmare • eventually we cannot scale out

48. Client-server architectures Each player sends packets to other players via a server Advantages: • Works over a WAN • easy to have reliable communication • Server can compress multiple packets into a single packet Disadvantages: • Nodes are server-dependent: server failure brings the system down • Server slows down the message delivery

49. PEER to PEER architectures • Peer-to-peer: communication goes directly from the sending player to the receiving player (or a set of them) Advantages: • It works over WAN • We manage to avoid servers Disadvantages: • Nodes need to know network configuration • Each node has to send same data multiple times

50. Multiplayer Server-Network Players can locate in the same place in the NVE, but reside on different servers • Real World ≠ Virtual World • Server-to-server connections transmit the world state information • Each server serves a number of client players