System for reducing data load in multi-person web conferencing Team JABS Group # 1

1. System for reducing data load in multi-person web conferencingTeam JABS Group # 1 Shu-Chen (Joanne) Chen Akshay Kannan Boaz Avital Siddarth

2. Shu-Chen (Joanne) Chen • CS Major • Akshay Kannan • EECS Major with entrepreneurial interest • Boaz Avital • EECS Major with interest in databases • Siddharth Shah • EECS Major Students

3. Area of Invention • Our inventions is in the field of computer science and internet video communication

4. Software does not allow for massive video conferencing • Conferencing with multiple cameras causes unmanageable bandwidth load Pain Point

5. A system for reducing data load when conferencing with multiple people Only transmit and show the people who are actively speaking Proposition

6. Communication • All participants of a conversation are classified as either "Active" or "Inactive" • Every participant will receive data, however only Active participants will transmit data to other participants.

7. Detecting Activity • Audio is used to determine whether or not a user is an active participant. • On client side, volume is measured as a function of time. Any disturbances in VU levels will cause a user to be set as active.

8. Detecting Activity • If the amplitude of a user's audio waveform exceeds a given threshold for a given period of time, the user will be set as active. • If the amplitude of a user's audio waveform drops below a given threshold for a given period of time, the user will be set as inactive.

9. Detecting Activity- sketch

10. Communication- client/server paradigm • Our mechanism applies to both a client/server approach, as well as a distributed one • For a client/server mechanism, "Active" participants will be continuously transmitting audio and visual data to the server. • All participants, both "Active" and "Inactive" participants will be continuously streaming data from the server

11. A Diagram - Communication Internet/ Intranet B C D E Active Inactive

12. Communication- distributed paradigm • In a distributed approach, Active participants will be transmitting data onto other active or inactive nodes. • Inactive participants will be responsible for obtaining data from all other nodes, as well as aggregating and propagating data from Active nodes onto other nodes in the system.

13. Audio mixdown • Since only audio from Active participants is relevant, these audio streams will be mixed down and sent to the user. • This effectively eliminates background noise coming from inactive participants, since this data is not transmitted.

14. User Interface • To ensure a clean user interface, only video data from Active participants will be shown on the screen. • Other participants will be shown in collapsed form, and dynamically enlarged when that participant speaks and collapsed once he/she returns to inactivity.

15. Managing multiple participants • When video from multiple participants are shown on the screen, video boxes are dynamically resized to allow the most data to fit at any given time. • As users enter/leave their active state, video boxes are dynamically resized to allow a greater/lesser amount of boxes to be displayed

16. Diagram - Interface A A C User C Stops Talking

17. Additional Sketches (will be included) • More detailed diagram of audio signal processing to determine activity • Flowchart to demonstrate audio signal processing algorithm • Diagram to show audio mixdown • Include network diagram for distributed connections • Split interface diagram into multiple diagrams to show what happens as multiple users enter/leave a conversation Use Cases: • Conference participants sitting in front of online laptops • Include both physical and remote participants • Online classroom with lecturing professor where students can ask questions

18. Advantages over Prior Art VIDEO CONFERENCE SYSTEM USING VOICE- SWITCHED CAMERAS Edson et al. 3601530 • Our software implementation does not require custom apparatus. • Our implementation includes a flexible user interface. • Our implementation is much more scalable in both distance between and number of participants. Voice-actuated switching system Byung H. Lee et al 4449238 • Our software implementation does not require custom apparatus. • Our implementation includes a video component. • Our implementation allows for user selection of important feeds or feeds to ignore.

19. Innovations • Combines simple constructs in a novel and useful way. • Allows for much larger scale conferences, entirely over the internet, while still being able to see all contributors. • Links speaker relationships such as Asker and Answerer so both can be seen. • Maximizes screen space  usage for the viewer.

20. Feedback

21. Abhinav I like this idea especially because I have used Webex from a customer perspective. I think it has a lot of business potential especially in emerging markets where bandwidth costs are not as cheap as they are here. It will allow service providers to reduce costs as well. Do look into Cisco's patent portfolio - bandwidth optimization is very important and they might have something.

22. Chen, Der Yu (Dorian) It's really cool, but the timing is important in this case so that the software knows whether a person is not talking at all or just having a brief pulse.  Also, how does the softward know the difference in human voice, background noise...etc?  My skype always tells me that I have a lot of background noise...just a quick question.

23. -There are definitely necessary improvements in video conferencing and this is an interesting way to approach this inherent problem. -There might be some issues regarding people consistently tuning in and then out based on pauses in their speech and/or ambient noise that triggers the transmit command. Just something to think about. -Would you guys actually be able to program such a feature? If so, congratulations.  David Pastewka

24. Eugene Li I like the idea a lot - although I wonder if other video conferencing groups might already do this.  Abhinav mentioned looking at Cisco's patent portfolio, and I would suggest looking at other companies' patents, too.  I wonder if a company like Skype might already have looked into this.  Keep up the good work!

25. Jonas Heller The quality of the sound recorded and played back would have to be of very good quality, which offsets the usability of the software. Also, natural sounds in communication (small "yes", "no", "hmm", *sigh*) risks to be lost and also the transitions between speakers must feel natural. Otherwise interesting idea that is in line with present trends!

26. Eliot Sun Really interesting idea!  Definitely a unique way of approaching the bandwidth issue. Is this going to be a consumer or enterprise product?  Or does it not really matter? If it is an enterprise product, I feel like the companies that value video conferencing aren't concerned about the cost of extra bandwidth, and a video conference may be more valuable to them if they can see everyone at all times.

27. Constance Lu Good idea! It's straightforward enough to be patented. Just an aesthetics issue: You say that the inactive people will no longer be shown on the screen. Are you imagining:  1. boxes (showing the active people) will pop up and move existing boxes to the side so that there are K boxes at one moment when K people are active, or  2. a total of N boxes (for N participating people), K of which are empty when those K people are inactive?  [I hope that made sense.] I can't see right now how the program would work to identify which people are active or inactive, but with the four of you, I'm sure you'll think of something. Good luck!

28. Kibeum Ryoo I think it is brilliant idea to enable massive user conferencing. But I think the idea about switching between active/inactive users will little chaotic... since sometimes people will simulatenously speak.

29. Justin Woo With so many video conferencing applications out there today, I would check carefully the history of other patents from other companies. The idea sounds great - however, would this impede the live view of conference guests who are not actively participating (instead, showing them as just a picture)? If so, I think it might take away from the video conferencing experience, sacrificing quality for speed.

30. Stefani Na:It's a great idea for saving up bandwidth but like many have mentioned, it would take away the live conferencing experience. Many companies engage in live teleconferences for interactive purposes (interaction between offices in different countries, for example). It would also, in a way, decrease participation or even lead to distraction. Also, do check if prior art exists.

31. Kin Tat Lay I am not a cs major but this sounds very interesting. I would just worry about the cost and things like that.

32. Mike Sontag Sounds like an awesome idea! My one concern with it would be if the videos jump around too much with people talking and not talking too frequently

33. Henry Hsue Great idea. I wonder if you can add in the technology for audio feed as well. I would also consider implementing this technology within existing video conferencing software. Would it be possible to patent the code and sell it to existing video conferencing companies, or would you have to create your own video conferencing software?

34. Osama Asif     A wonderful and very capable idea..I am just wondering would you be patentin the codec, the platform, or some other apparatus specifically, or would you be patenting a series of combining elements together.. What competition is out there on the market right now, and what are firms doing currently as an alternative to what you propose?

35. Tiffany Vukasinovich I can see how this would be really useful, but are you sure it isn't already out there? 

36. Jeremy Mekdhanasarn I like the idea. The use-case for multi-player videogames is probably more applicable than business conferencing, as most conferences require that people take the time to be present in a location that already has equipment and software that can handle multiple people speaking at the same time (and speaking often overlaps).

37. Matt Samuels •Very interesting idea.  I’m not too technologically savvy, and therefore am not aware of any prior art in this field, but if you have the technical expertise to implement a system you could make some real $$$ on this product. •What is prior art in this area?

38. David Rockford • •I’m wondering if Each individual will have their own original square to be inside. In your Diagram-Interface slide, Person A is on the left and Person C is on the right. When Person  stops talking, only person A shows up. But if person A stops talking, then Person C starts talking, and finally person A starts talking again, will their position of who is on the left or right side change as well? (I hope this question makes sense). It will be easier on the people in a conference if each person has a set space and people do not switch around a lot.  • People also want to see the physical/emotional reactions others convey while one is talking.   This would not be present if this system is implemented. • •If a person is not talking and still wants to be seen (maybe they are drawing something or doing whatever), there should be a means in which they still show up on the feed. • •Constantly switching between part of a screen and full screen with different faces may be confusing/overwhelming to many people. Would need to try it out to see if it is a good approach to conference calls.

39. Stephen Gu Interesting idea. I think the main thing you need to worry about is the interface. If people's faces are just popping in and out of the screen, it might mess up the flow of conversation. I think one of the biggest draws of video conferencing is that you can see the people you are talking to and therefore you are not just talking to a blank screen, so it'll be interesting to see how you guys handle it.

40. Jerry Barnes (16559151) I actually think this is a really good idea. I have read the other comments and the majority of them seem helpful but I think that naturally when you are first developing a new idea there are things that will have to be smoothed out before moving forward. As a whole sounds like an invention birthed by necessity.