Synergies between World Opera and Internet of Things

1. Synergies between World Opera and Internet of Things Frank Eliassen - Lucas Luiz Provensi - Roman Vitenberg Lille, 10.03.2010

2. Topics • World Opera • Gilgamesh Production • Scenarios • Location tracking • Motion recognition • Light Control • Multimedia sensor networks • Information services • Challenges

3. World Opera • Worldwide distributed operas • Singers that are remotely located can interact as if they are in the same stage. • Musicians distributed around the world can play together. • Audience can see the opera from one of the stages or from other remote place.

4. Gilgamesh • First World Opera Production (2012) • 3 Stages: Tromsø – Stockholm – Copenhagen; • Based on The Epic Of Gilgamesh, an epic poem from old Mesopotamia.

5. Gilgamesh • Virtual Stages Uruk Gilgamesh Cedar Forest Land of the Dead Humbaba Enkidu

6. Gilgamesh - Observations • The number of multimedia flows/streams coming in and going out will be different for each stage • It can change at runtime as well • At certain times, distributed actors may be interacting with each other while the musicians at all stages keep playing • Actor has a fixed assignment to a physical stage but may travel between virtual stages • A physical avatar to represent the actor at the others stages, like a puppet with a display in the face that shows the video stream of the face of the actor

7. Observations: Video vs Audio • Musicians are very sensitive to audio quality but do not care too much about video quality • Remote musicians are sensitive to maestro acting. They are not very tolerant to delay in the maestro's video streaming • When acting against distributed peers, actors are not tolerant to delay and they need a good video quality and enough resolution to see face details • Actors also need to hear audio, e.g., when in a synchronized dialog • Embed a speaker into the avatar so that the actors have the sensation that the voice is coming from that point

8. Location tracking • The maestro will be at the city stage, and he will represent the sun god and will be walking around the stage • The conductor location will be tracked to physically reproduce the movement of the maestro at the other stages.

9. Location tracking • Position of actors in a virtual world.

10. Movement Recognition Wii Mote (Nintendo) Composed by accelerometer and optical sensor. Can be extended with a gyroscope.

11. Movement Recognition Project Natal (Microsoft) Depth sensor consisting of an infrared projector combined with a monochrome CMOS sensor

12. Movement recognition • Replicate musicians movements on virtual entities or phenomena.

13. Movement recognition • Replicate actors movements on virtual avatars

14. Light control • Lighting sensors can provide information lighting condition to scene controller software.

15. Scenarios • Multimedia sensor networks • New sources of media content : views of different angles and audio collected at different points.

16. Services • Services offered to the audience at the stages and outside. • Information about the ongoing act; • Access to alternative media streams; • Shared comments with other users. Act 2 Gilgamesh and Enkidu fights the Bull of Heaven Bull Gilgamesh Enkidu

17. Challenges Delay Synchronity Adaptation Dependability Scalability

18. Thank you! • Questions?