The Family Video Archive: Organizing Memories

1. The Family Video Archive:Organizing Memories or What I did on my summer vacation Gregory D. Abowd Assoc. Professor College of Computing and GVU Center

2. Acknowledgments • Andreas Lachenmann & Matthias Gauger (U. Stuttgart, Germany) • FVA Demo is their work • Rudinei Goularte (USP-Brazil) & Niels van Hese (U. Delft, Netherlands) • Earlier prototypes of FVA • Molly Stevens (GT IDT program) • Worked on related Living Memory Box • Shwetak Patel (PhD program) • He’s in charge now

3. Dedication • Richard G. Abowd, Jr. • My father, 1924-1998. • Founder of RGA Action Films. • His 8mm film collection is the original inspiration for this work.

4. Outline • Motivation for FVA • Demo of FVA • Annotation/tagging • Search/browse • Issues in the design • General challenges and the future

5. Motivation • Lots of video/movies • Some families have it now • It’s only getting easier to accumulate • Organization is hard • Worse than photographs • Makes it hard to enjoy viewing later on • If it were easier to browse videos… • Would our collections be more valuable? • Could you simplify the organization?

6. Related Work • Digital Photography • PhotoMesa (Bedersen, 2001), PhotoFinder (Shneiderman), MSR (Platt et al.) • Manual/automated annotation, visualization and automated clustering • Digital video • Plenty of work on improving nonlinear editing (e.g., Hitchcock system from FX-PAL) • Vision community works on scene detection, object detection (e.g., MPEG-4) • Annotation schemes (MPEG-7, Dublin CORE) • Browsing/searching (Informedia at CMU) • Commercial systems (e.g., VideoLogger from Virage)

7. What is our niche? • Family video • not professional quality • no closed caption information, so relatively no metadata • Not editing • getting right content • would feed an editing system • Interaction techniques that ride the balance between manual (tedious but rich and accurate) and automated (fast but error-prone) annotation. • Motivate better annotation at point of capture.

8. Goal • Create a system to annotate home movies with relevant metadata to facilitate browsing and searching. • Effective symbiosis between manual and automated techniques. • Control/accuracy trade-off with speed

9. The FVA prototype • Test case: RGA Action Films • 8mm film of Abowd family • 1955-1984 (then we switched to video) • Digitized into AVI format • Windows Media 9 codec • ~25 hours, ~17GBytes • BIG family • 12 siblings, 52 first cousins, 32 nieces/nephews) • I am currently adding my own family video from 1997-present

10. FVA Demo • There is MUCH more functionality than we will show here. • See http://swiki.cc.gatech.edu:8080/ahome/97 for more details, including full Java/JMF source, installation instructions and very thorough user manual. • Paper in MIR 2003 available

11. FVA Basics • Video archive is collection of video files • Broken up into arbitrary 5-minute chunks • Absolutely no other information, but I do have my father’s notes • User wants to divide files into scenes and associate meaning to scenes.

12. Tagging scenes descriptions • Hierarchy of tags • Who • Where • Event • other major categories • Completely user-defined

13. Scene metadata Albums Existing tagsuggestions Video window New tagsuggestions Video files Tag hierarchy scene boundary FVA: Annotation/Tagging

14. Detailed Operation • How to annotate a scene • Free text (important from experience) • Date (flexible accuracy and quick) • Drag & drop tags • Automated suggestions of existing and new tags • Select “important” tags • Tag extent in scene (defaults to whole scene)

15. Creating Scenes • One of the more arduous tasks. • Automated scene detection based on frame differencing. • Manually adjusted threshold. • Ultimately, this task goes away (e.g., iMovie) Threshold adjustment

16. Zoomable scene browser Tags Querywindow Zoomable timeline filter FVA: Searching/browsing

17. Detailed Operation • Zooming as filtering • Built using piccolo ZUI toolkit from Maryland • Semantic zooming based on tag hierarchy • Filter history visualized • Chaining filters • Timeline filtering • Video thumbnails • Album creation

18. Design Issues • “Expert” interface • Daunting to beginner • Interface doesn’t suggest “right” way to use it (e.g. tag hierarchy, searching) • Acquiring video is still difficult

19. Tagging • Tag hierarchy completely user defined • This idea is increasingly powerful. • There can be LOTS of them, so how to manage? • Dynamically drives the search interface. • Aliases provide further functionality (demo). • Parameterized tags (e.g. “John’s birthday). • Valid begin/end dates. • Tagging is made much easier, but it is still daunting and time-consuming. • Will anyone besides me ever do this? • You are never really done, but where is line of diminishing returns? • Symbiosis of manual/automated tasks

20. Reflection and Future • We built it, so what? • I always wanted this. My family loves it! • Basically, I am happy regardless of what comes next (there is a lesson there). • There are research opportunities

21. Future Work • Front-end • Link to an automated capture system, with stationary and mobile “feeds” • Tagging at point of capture (Who, Where, When, Event) • Back-end • Working with albums to feed nonlinear editing • Huge time saving for gathering content • Facilitating collaboration • Expert/novice interfaces and transitioning between • Alternatives to semantic zooming • Handling more dense archives • Are there more serious applications?

22. Potential Uses • Analyzing individual development • Early diagnosis of disorders (e.g., autism) • Tracking intervention therapies • Shared, historically significant archives • HCI video collection • eClass-like lecture archives • Finding Lost Objects service

23. Authoring Applications • Previous examples are all different enough from FVA • Adapting FVA not the right approach • Authoring environment to build FVA-like applications

24. Questions?