Automatic Video Editing

1. Automatic Video Editing Stefano Bocconi CWI Amsterdam Stefano.Bocconi@cwi.nl

2. Talk Outline Open Video Documentaries Rhetoric for Video Documentaries VOX POPULI, a video editing engine Future work

3. The General problem • Affordable means of producing video material • The Web as an interactive distribution channel • How to make video material accessible on a semantic level? • How to expose the “messages” contained in the footage?

4. The case for Open Video Documentaries • There is only one final version: what is shown is the choice of the author/editor • If the subject is controversial, the multiplicity of viewpoints can be lost • Users have high-level semantic questions, like: • Is opinion X supported by people? How? • Show me the argumentation of people that have opinion X

5. Video Material • Interview with America: 8 hours of footage with interviews and background material • Subject: the opinion of American people after 9-11 • Shooting: from 27-10-2001 to 01-11-2001 in Stamford (CT), New York (NY), Boston (MA) and Cleveland (OH) • Online at www.interviewwithamerica.com

6. Example (I) • Topic: War in Afghanistan, Show one pro opinion and refute it

7. Example (I) • Topic: War in Afghanistan, Show one pro opinion and refute it • “I am never a fan of military action, but I can not think of a more effective solution” • “War has never solved anything” • “Two billions dollar bombs on tents”

8. Qualifier Data Claim Warrant Condition Backing Concession Toulmin's Layout of Argumentation

9. Claim Claim Building Block for automatic editing Two billions dollar bombs on tents contradict I can not think of a more effective solution Claim weaken Concession I am not a fan of military actions support War has never solved anything

10. Q Q Q D D D CL CL CL W W W Cd Cd Cd B B B Cc Cc Cc Building Block for automatic editing Support / contradict Support / contradict

11. Processing the Statements • Statement formally annotated: • war best solution • Generated contradicting statements • war worst solution • war bad solution • diplomacy best solution • war not solution

12. Rhetoric applied to Video • Need to formalize means to generate opinionated arguments • The ability, in each particular case, to see the available means of persuasion (Aristotle) • Logos: appeal based on logic or reason • Ethos: appeal based on the character of the speaker • Pathos: appeal based on emotions

13. Ethos • The authority of the speaker • Rates the interviewee according to a simple user model • Interviewees are annotated with social characteristics: • Age • Education • Employment • Race • Religion • Sex

14. Pathos – Example (II) Appeal to the emotions of the viewer

15. Pathos • Images annotated with concepts • Based on a user model • Possible improvement: Ortony’s emotions theory concept Heider’s balance +/- +/- viewer interviewee +/-

16. Domain commitment • Requirements: • Domain independent: annotations and process can be applied to other domain (currently working on another domain) • Different material can be added if annotated • Relations between concepts (e.g. war – diplomacy) • Hierarchy of concepts • The resulting ontology can be user dependent

17. The annotations • Descriptive on visual information • Filmic • Interviewee (social) • Question • Rhetorical on verbal information • Rhetorical Statement • Applied to Media items • Interview segment • Statement • Image

18. VOX POPULI: Results • Provide support to access video material in an argumentative manner • Domain dependency located in the ontology and user model, not in the process • Data-driven: using existing material not crafted for the application • User-driven, purposeful composition of (short) video documentaries: some awareness of the created meaning

19. Other Approaches • Media Streams: no awareness of the “message” • Evolving Documentary: association via keywords, no composition of a message • Terminal Time: footage crafted for the task, closed system • Auteur: only visual (continuity-driven), different rhetoric

20. Steps Ahead based on verbal • Up till now results on the micro level • Define narrative strategies based on: • Evolution: Exposition, Inciting moment, Complication, Resolution (Freytag’s triangle). Toulmin’s model allows directionality (from data to claim) • Intensity: Toulmin’s model allows intensity (the qualifier), intensity also contained in the modifiers of the statements • Parallel narratives, possibly converging to the same claim

21. Filmic Goals • The visual narrative must support the verbal narrative, at least continuity: • Not so many constraints in the chosen genre • use metaphors, like the day, the journey • The visual should support the rhetoric • Invert the roles, use visual to construct a narrative and verbal to support it A visual analogous to Toulmin’s model?

22. Example (III)

23. Filmic Techniques • Rhythm • Counterpoint editing • Transitions • Spatial arrangement • Shaff’s cinesthetic structures: • Separation • Familiar image • Slow Disclosure

24. Questions? Thanks for your attention

25. More Future work • False implications • Logical fallacies • Process • Rule-Based Tunable • Self Rating

26. Vision • Generation of meaningful video sequence: bottom-up approach to the creation of meaning, how much knowledge needed to produce what quality of meaning

27. Genesis of the idea • Video Retrieval • Capable of retrieving (annotated) video sequences by content • The “hammer hitting a nail” User-Query • If the Video not available: automatic composition + = • Automatic Video Editing • Engine “knows” the hammer is hitting the nail

28. Example of Toulmin sentence • Concession: There are solutions • Claim: Nobody is trying to find a solution • Data: there is too much money to be made • Warrant: Solutions other than war means loosing money • Backing: Hospital supplies, food, oil