Media Services in CINEMA

1. Media Services in CINEMA Presented by Kundan Singh Joint work with Henning Schulzrinne, Jonathan Lennox, Xiaotao Wu, Wenyu Jiang, Sankaran Narayanan, Ajay Nambi andAli Khwaja Computer Science Department, Columbia University April 25, 2003

2. N Purpose • What are media services? • What is CINEMA? • What services are provided by CINEMA? • Conferencing • Streaming media • Unified messaging • Interactive voice response • How can we use CINEMA? The discussion on services and specific CINEMA implementation components are intermixed. Not relevant or not implemented On-going work Yes, Implemented N O Y

3. Call routing services: pre-call, one party speed dial call forwarding “follow me” call filtering/blocking (in/out) do not disturb distinctive ringing Call handling features autoanswer Multi-party features call waiting call transfer (blind, consultative) conference call call park call pickup music on hold call monitoring O IP telephony services (PSTN) IP-telephony can’t win by just providing what PSTN has

4. O IP telephony services (Internet) • Presence-enabled calls • place call only if callee is available • Presence-enabled conferencing • call conference participants when all are online and not busy • IM conference alerts • receive IM when someone joins a conference • Unified messaging • receive email, IM alert for new voicemails

5. O Where the services reside? Double ringing sound when boss calls… Enter your authentication PIN for billing… • Common gateway interface (CGI) • Call processing language (CPL) • SIP servlet Use finger for locating user… B2BUA Endpoint Make call when boss is online … Proxy/registrar Endpoint Forward to office phone during day, and home phone during evening… • Language for End System Services (LESS)

6. N Application layer Transport (TCP, UDP) Network (IPv4, IPv6) Link layer Physical layer “Bricks” in media applications Interactive voice response Internet Telephony Internet Radio/TV Messaging and Presence Unified messaging Video conferencing SIP SAP RSVP RTCP Media G.711 MPEG RTP RTSP Quality of service Media transport Signaling Program Call routing Voice XML DTMF Mixing Speech/ text SDP

7. N … Audio Packet Transfer • Digitization(e.g., sampling at 8kHz, 16 bits per sample, i.e, 128 kb/s or 320 bytes per 20 ms) • Real-time compression/encoding(e.g., G.729A at 8 kb/s) • Transportto remote IP address and port number over UDP (Why not TCP?) • Processing on receiver side is the reverse

8. N Problems with UDP Sender 1 2 3 4 5 6 7 (a) (b) 1 2 3 5 7 6 timeline Receiver • Unreliable UDP • Packet loss • Out-of-order (very rarely) • Jitter (delay variation)

9. Y Receive Buffer • Receive buffer: to absorb jitter • Tradeoff in buffer size • Adaptive delay adjustment • Sequence number: to detect packet loss; Just ignore the loss! Sender 8 9 0 1 2 3 4 1 2 3 4 5 6 7 8 9 0 2 3 4 1 2 3 5 7 6 Receiver 1 2 7 8 9 0 2 3 1 2 5 7 8 9 0 2 1 2 3 5 6 7

10. Y Real-time Transport Protocol IP header V CC M Payload type Sequence number P X UDP header Timestamp (proportional to sampling time) RTP Header Source identifier (SSrc) Encoded Audio Optional contributors’ list (CSrc) msg RTP: media transport RTCP: QoS feedback sendto(…, msg, …) recvfrom(…, msg, …)

11. Y Web server Web browser HTTP HTTP Web browser meta file file Media server RTSP Media player Media player RTP Streaming media • Messages • RTSP vs HTTP • UDP, Stateful, Control, Timed • Applications • Recording, Playback, Live media, Conference

12. Existing Playback/recording of -law (Sun .au files) and raw Unicast address Multi-threaded; Portable (Unix/win32) Works with Apple QuickTime client UDP + TCP IPv4 + IPv6 Planned Multicast MPEG video Mp3, wav audio Media over TCP Live media Y Columbia rtspd

13. N RTP-based Conference -law Mixer Transcoder -law   G.729 G.729 -law -law

14. Y Why do we need signaling? • What is the IP address of Alice’s host? • What audio/video encoding can it support? • Session Initiation Protocol • Email-like address: alice@home.com • DNS • Proxy, redirect REGISTER as alice=>128.59.19.194 Call alice 128.59.19.194 Alice’s host 128.59.19.194 bob’s host home.com

15. Y Session description INVITE alice@home.com I can support -lawand G.729 Send me audio at 202.16.49.27:6780 Bob Alice 128.59.19.194 202.16.49.27 OK; I can support -law Send me audio at 128.59.19.194:8000 RTP To port 8000 RTP To port 6780

16. Y Columbia sipc

17. O Emergency service • Calling • E911, location • Assistance • Audio/video, remote-help • Notification • Earthquake, fire

18. Y PUA + PA Presence/event notification office.com bob@office.com Presence server PUA PA alice@home.com REGISTER SUBSCRIBE NOTIFY PUA registrar

19. Y Columbia IM and presence

20. O Endpoint call control • Language for End System Services (LESS) for endpoint service creation • Direct user interaction, direct media control • Handle converged information, e.g., call, presence, email • Example: when buddy is online, make a call <less name="online_call" require="generic presence ui"> <notification status="online" priority="0.5"> <address-switch field="origin"> <address is=“alice@office.com"> <call /> <alert sound=“ring.au" text="Calling …" /> </address> </address-switch> </notification> </less>

21. Y Priority.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout Urgent SIP proxy Phone Low-priority Voicemail Network call control • SIP-CGI • CPL • SIP servlets if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ /sip:boss@mycompany.com/) { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; } }

22. O Call transfer B A • REFER • Blind/ consultation/ attended C active call REFER C Referred-By: B INVITE C Referred-By: B BYE A active call

23. O SIP SIP B2BUA and third-party call control B • Back-to-back UA • Incoming call triggers outgoing call • Services • Calling card • Anonymizer A C INVITE OK (SDP1) ACK INVITE (SDP1) OK (SDP2) ACK INVITE (SDP2) OK ACK

24. Y Voicemail Endpoint redirects • Problems in PSTN • Design alternatives • Issues Redirect after 10s alice@vmail.yahoo.com Proxy controls Voicemail acts like a phone vmail.pl SIP_FROM SIP_TO stdin CGI-PROXY-REQUEST stdout If no response accept after 15s

25. Y INVITE (1) INVITE CANCEL OK INVITE email (3) OK (5) BYE sipum (2) SETUP (4) RTP rtspd Columbia sipum • Goals • Universal access • Scalability • Provider independent • Why SIP and RTSP? • Reuse existing infrastructure and tools

26. Y Columbia sipum • Retrieval • Web interface • rtsp://server/alice /inbox/1677.au, • sip:alice-1677-retrieve@server • press 1 to listen… • Configuration • Folders • Options • Email

27. Y Internet Internet user VXML Voice gateway HTML • Voice and telephony functions • VoiceXML browser Web server • Service logic (CGI, servlet, JSP) VoiceXML Gateway sipvxml PSTN Telephone IVR platform • Voice and telephony functions • (ASR, TTS, DTMF) • Service logic (application specific)

28. Y VoiceXML contd. <form> <field name=‘id’> <prompt> Your ID, please. </prompt> </field> <block> <submit next=“url”/> </block> </form> <form action=“url”> Enter your Id: <input name=‘id’> <input type=‘submit’> </form> Telephony, speech synthesis or audio output, user input and grammar, program flow, variable and properties, error handling, …

29. Y DTMF PSTN Internet Telephone SIP/PSTN gateway Internet user Audio D SIP INFO D RTP/ RFC2833

30. Y Media server rtspd Columbia sipvxml PSTN Internet • Unified messaging access • Email by phone • Event notification and scheduling • Audio volume control for conference • Advanced conference control Telephone SIP/PSTN gateway TTS, ASR, DTMF, XML, HTTP, RTSP sipvxml SIP phone Web server Email.tcl

31. Login Email formatting Listen, reply, delete, compose, forward Navigation -next, previous, jump Email formatting SIP based Text-to-speech Y IM Call Email + phone Email by phone Inbox Email procmail important mails Internet Email to IM SIP SIP VoiceXML browser TTS SIP HTTP Internet JSP Email servlet Inbox DB Email to phone

32. Who can initiate? authentication, billing, … Feedback to voice user to IM user initial IM greeting Talk-spurt detection Speech recognition Y IM Call IM + voice call SIP MESSAGE TTS ASR SIP INVITE RTP

33. O Unified messaging • The gaps among different media (audio, video, text), devices (PC, phone) and means of communications (Email, SIP, IM) disappear for messaging

34. O Notification • Calendar • Events • Conferences Schedule from a browser SIP call 2129397063 IM kundan@ Web server Calendar.cgi Email Kns10@… “at 6:00pm”

35. Y Phone announcement server Text or audio Destinations • Input • Range:93970?? • List: A, B, C • Example • Announcement • Emergency • Issues • Voicemail • Failure detection TTS PAS SIP . . .

36. O RTSP + TTS + ASR TTS ASR Media server SETUP rtsp://server/tts.cgi ?text=How+are+you. SETUP rtsp://server/asr.cgi PLAY RECORD RTP RTP SET_PARAMETER Text=I am fine, thank you.

37. N A D A D A D   A B+C+D D  A+B+C A+C+D A+B+D B    C C C B C B B Audio conference models (non-multicast) Topology star full-mesh ad-hoc No central point of failure Advantages Heterogeneous simple clients Typically only three party conferences Complex endpoints Complex signaling Disadvantages External server with high bandwidth link

38. O Centralized conferencing • Conference as URL • staffmeet@office.com • On the fly conferences • my.adhoc@server • Basic task: join/leave • Dial in, Refer dial in • Dial out, Refer dial out INVITE INVITE REFER INVITE server REFER

39. Y Periodic timer Playout delay Linear G711 G711 M - A=B+C D E A Linear Mixed linear DVI DVI M - B D E B M=A+B+C Linear GSM G711 D E M - C C Send thread Receive thread Columbia sipconf • G.711, GSM, DVI, G.722 mixing (decode-mix-encode) • Video replication; IM; text; VNC screen sharing

40. Y Columbia sipconf

41. O Conference control • Access control • Floor control • Notification • State • Participant: active, departed, booted, failed • dialog state, floor state • Media • sending: received-by-all, muted • receiving: receiving-all, anchor-only <conference-info version="0" state="full" entity=“meeting@example.com"> <user uri="sip:alice@home.com" display-name=“Alice"> <status>active</status> <media-status> <media-stream media-type="audio"/> </media-status> </user> <user uri="sip:bob@office.com" display-name=“Bob Wilson"> <status>departed</status> </user> </conference-info>

42. Y sipconf Conference + VoiceXML 1. INVITE sipvxml 2. Call accepted 3. Enter your four digit PIN 4. Entered 4-6-8-3 5. Authenticate user, 4683=>Alice 6. Enter the conference identifier 7. Entered 2-3-# 8. Permission to join, 23=>meet sipvxml 9. REFER meet@conference 10.Terminate the old call Caller 11.INVITE meet@conference Call transfer vs bridged mode

43. Y New conference applications The ease & flexibility of sipvxml enables us to build custom telephonic applications to suit our needs. e.g., Volume Check Application 1. INVITE sipvxml 2. Menu 1. Vol Check 2. Mic Check 3. User enters 2 sipvxml 4. User speaks out a voice sample 5. Voice sample is analyzed 6. SipVXML: Vol level too high/low/… Caller 7. User adjusts the vol level. sipconf 8. User now joins conference.

44. Y sipc SIP323 Netmeeting SIP/PSTN Conferencing • Automatic volume adjustments • Automatic load balancing • Delay adjustments • Conference recording • Local or RTSP Recording in a media server

45. Y Ncast MPEG distribution INVITE sip:xyz@cs.columbia.edu sip.cgi?sip:xyz@cs.columbia.edu

46. Y sip:john@cs.columbia.edu INVITE sip:john@cs.columbia.edu My owner’s SIP address is sip:john@cs.columbia.edu Help!!! (invoke sipc to call sip:john@cs.columbia.edu)

47. Y Summary of media services • Streaming media • Playback, recording, live-media • Unified messaging • Voicemail, video/IM integration, ... • Conferencing • Mixing, load-balancing • Interactive voice response • Tele-banking, VoiceXML

48. Y CINEMAColumbia InterNet Extensible Multimedia Architecture Distributed component architecture User-agent (client) applications CINEMA Application and proxy servers Libraries and APIs Management and control

49. Y Columbia SIP user agent: sipc • Media • Audio, video, text, white board • Screen sharing • Shared web browsing • Advanced • Presence, instant messaging • Programmable call handling • Conference control (on-going) • Emergency notification and handling (on-going) • Device control • Language for End System Services (on-going)

50. Y Web server Columbia SIP servers (CINEMA) Single machine Telephone switch rtspd: media server Local/long distance 1-212-5551212 sipconf: Conference server Quicktime RTSP PSTN RTSP clients Department PBX sipum: Unified messaging sipd: Proxy, redirect, Registrar server Internal Telephone Extn: 7040 713x SQL database SIP/PSTN Gateway Web based configuration SNMP (Network Management) H.323 siph323: SIP-H.323 translator Video encoder NetMeeting