1 / 35

The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd

The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd Woodrow Wilson Indiana Teaching Fellow and Research Technologies & School of Informatics Indiana University Bloomington 2 May 2013. Introduction (1): Time Is of the Essence.

apria
Download Presentation

The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd Woodrow Wilson Indiana Teaching Fellow and Research Technologies & School of Informatics Indiana University Bloomington 2 May 2013

  2. Introduction (1): Time Is of the Essence • Gandini Juggling’s Mozart (Symphony 25, 1st mvmt) performance • How created? How did jugglers learn it? • I don’t know, but I do know was a huge pain! • Likewise for doing almost anything w/ complex temporal phenomena • …and timescale can be yoctoseconds or billions of years • Natural phenomena • Inflation of early universe, chemical reactions, animal movement, weather, illnesses, climate; development of individuals, of species, of geographic features, & of galaxies • Artificial & cultural phenomena • Movies, behavior of computer programs, football games, multimedia shows, operas, political crises, wars, development of civilizations 1 May 2013

  3. Introduction (2) • Complex temporal phenomena difficult to study or (for performances) to create, because don’t “sit still” • Can “play” repeatedly, but at expense of context • Usual answer: visualization • Good start, but we can (and need to) do much better than existing systems • For music, General MusicVisualizer (GMV) • Music ex: Hendrix/Star Spangled Banner: how to notate? • Musical audio: Shakuhachi solo: how to segment? (Nina Fales) • Might be a good solution to notating Hendrix! • Studying speech or animal sounds is a similar problem 3 July 2009

  4. GMV “demo”: Segment shakuhachi solo • Scenario: music-informatics researcher or ethnomusicologist comparing two audio-segmentation algorithms • …or composer comparing input & output of synthesis programs 20 Apr. 2009

  5. The Problem for “Multimedia” • “Multimedia” = performance w/ music & sync’ed movement (dance, etc.) or visuals (fireworks, etc.) • Requirements • 1. People need several ways to “look” at an instance simultaneously • 2. People need to be able to compare instances • 3. Has specialized notation (at least for music) • 4. Media: video, audio, still images, text • Audio independent of video is often important 1 May 2013

  6. The Solution for Multimedia • General MusicVisualizer => General Multimedia Workbench (GMW) • Early prototype of GMW: AWB Plus • Based on EVIA Annotator’s Workbench • Java program, developed for ethnography • Queen of the Night aria, in “GMW” • Fireworks show with music, in AWB Plus 28 Mar. 2010

  7. GMW “Screenshot”: Queen of the Night aria • Scenario: singer (or conductor) comparing videos of performances to learn role in opera, musical, etc. • …or stage director, choreographer, or lighting designer comparing previous versions to own ideas • …or scholar studying performances (perhaps juggling w/ music!) 4 Apr. 2009

  8. Fireworks show with music, in AWB Plus • From video & still photos of show by Mike Durbin • All timestamped => (semi)automatic alignment • Music added afterwards! 28 Mar. 2010

  9. The General Problem • Long-standing, difficult problems in all fields • Affect all uses: research, creation, “aesthetics”, organization • …plus deluge of data in many fields • Even arts & humanities are getting lots of hard data • Data promises to help, but not much help so far! • Need insight, not data; how to get there? • Widely recognized as important goal 1 May 2013

  10. Towards a Solution (1) • Review: multimedia features • 1. People need several ways to “look” at instance simultaneously • 2. People need to be able to compare instances • 3. Has specialized notation (at least for music) • 4. Media: video, audio, still images, text • Observations • Multimedia features #1 & 2 are true of every (and 3 & 4 of many) complex temporal phenomenon • Every temporal phenomenon is complex • People often want to relate phenomenon of one type to another 1 May 2013

  11. Towards a Solution (2) • The cross-discipline argument • Problems in all fields have much in common => potential for synergy • But people rarely share ideas or software across disciplines • Issue of “disjoint technical vocabulary/literature” (cf. Swanson 1988) • Ex: Are there periodic patterns in public opinion? => political scientists should try Fourier analysis, but… • The research vs. creation vs… argument • Problems of researchers, creators, etc. in a field have much in common • => A general temporal system could greatly reduce “reinventing the wheel” or doing without one 1 May 2013

  12. The General Solution • General temporal system applicable to very many disciplines is possible • General temporal system applicable to all types of use is possible • GMusicW needs so much flexibility, a GAnythingW wouldn’t need much more • Oh, yeah? With what magic? • The magic of Software Architecture! 1 May 2013

  13. Architecture for a General System (1) • Main tasks for framework: (temporal) synchronization, UI support • Software for common audio & video uses: 15 Mar. 2009

  14. Architecture for a General System (2) For sequential art & movies based on sequential art (John Walsh): 15 Mar. 2009

  15. General Temporal Toolkit/Workbench (GTT/W) • “Temporal”, not just “Multimedia” • Supports multiple: coordinated, editable, interactive visualizations & sonifications (eventually “tactilizations”, “olfactizations”?) • …of multiple instances • …of any combination of temporal phenomena • …plus data mining & analysis (Solution 2) • For all applications (research, creative, other) • Myosin V, in AWB Plus 5 Mar. 2010

  16. Important Feature 1: Better Human/Machine Partnerships Figure slightly modified from Yu et al (2008) • Integrate info visualization & analysis/data mining (Shneiderman 2002; Yu et al 2008) => closed loop: use visual perception to generate hypotheses for analysis; present results of analysis visually • “HCIR”, visual analytics • …or substitute synthesis for analysis, e.g., for composers 7 Apr. 2009

  17. Important Feature 2 & 3: Allow All Sensory Modes; Automatic Synchronization • Sensory Modes • Visual: visualization is most generally useful, but not the only answer • Auditory: sound is central for many applications • sonification is surely valuable for some non-audio phenomena • Tactile: important for the blind • Other (olfactory): important for ?? • Don’t rule out support for all sensory modalities • Automatic Synchronization • Of audio (at least) has made great progress in last 10 years 2 May 2013

  18. Myosin V, in AWB Plus • From nanobioscience work by Craig & Linke • Myosin V crawling along a strand of acetin 28 Mar. 2010

  19. Big Issue: Time Representation • Resolution/precision • Cosmology: Planck time to quadrillions(?) of years • Precision & handling of uncertainty • Historical forms: “1685?”, “fl. mid-15th century”, etc. • Damaged sources: “25th of [unreadable] 1743”, etc. • Handle with Probability Density Functions? • Absolute & relative time • Calendars are incredibly messy • Design of universal time representation is well along 1 May 2013

  20. Huge Issue: Getting Off the Ground (1) • AWB Plus prototype, based mostly on EVIA Annotators Workbench; some of Variations • may also use CIShell, Chen Yu’s system for “visual mining of multimedia data” • all from IUB • Other possible open-source starting points • Sonic Visualiser, SyncPlayer, SIMILE Timeline, etc. • Connections to general tools for nontemporal visualization • Network Workbench (Katy Börner/IU SLIS): networks • Google Map API: “space” (surface of the earth) • Connections to semi-general tools • Ex: Max/MSP, Pd: musical audio, video • Ex: imagej for image display/processing 1 May 2013

  21. Getting Off the Ground (2) • Support from communities of users is almost essential • Have seen strong interest for: • Ethnomusicology field work • Opera scholarship • “Could revolutionize study of opera” • Conducting • Teaching electronic music • Study of computer-aided musical improvisation • Study of comic books/movies • Research on artificial life • Software engineering, research software • Nanoscience/nanotechnology • Study of French 1 May 2013

  22. Conclusions • How do I know applications are realistic? • Many probably aren’t, but many, many possibilities exist! • Have ca. 30 usage scenarios, ca. half written/endorsed by experts • Some examples (all experts in respective fields) • Ruth Stone: ethnomusicology field work • Philip Gossett, Nick Cook: study of music/opera performance • Paul Nadler: conducting • John Gibson: teaching electronic music • Elaine Chew: annotating video of computer-aided musical improvisation • John Walsh: comic books/movies • Larry Yaeger: artificial life • Fabian Fagerholm, Ben Kovitz: software engineering, research S/W • Amar Flood: nanoscience/nanotechnology • Personal knowledge/experience for a few 1 May 2013

  23. End • Thanks to Eric Isaacson, Geoff Chirgwin, Will Cowan, Allen Winold, Paul Sturm, Ben Kovitz, Craig Stewart, Eric Wernert • THE END 1 May 2013

  24. Extra Slides • Following slides (with background like this one) are just in case… rev. 18 Feb. 2009

  25. Good Design for Music Can Be Good for Many Things • Cf. “Why Studying Music is Both Difficult and Important” (Byrd 2009) • Music is an art => people use elements in unusual ways • Music is a performing art =>performances & symbolic representations • Much music hascomplex synchronization requirements • Music involves many different instruments, often in groups. Leads to: • Arrangements/transcriptions for other instruments • Versions for players with different levels of skill • Notation may represent sounds or actions • Music is often combined with text via singing, narration, etc. • Music is extremely popular, so: • Some works exist in many versions, arrangements for different ensembles, etc. • Handling challenges is important, even on purely economic grounds rev. 18 Feb. 2009

  26. HCI: Searching, Browsing, & Visualization • Visualization is essential for browsing, merely helpful for searching • In browsing, user finds everything; the computer just helps • Browsing is obviously good because it gives user more control, but few systems emphasize it. Why? • “Users are not likely to be pleasantly surprised to find that the library has something but that it has to be obtained in a slow or inconvenient way. Nearly all items will come from a search, and we do not know well how to browse in a remote library.” (Lesk, p. 163) • For “and”, read “as long as” • Searching is more natural on computer, browsing in real world • Effective browsing takes very fast computers—widely available now • Effective browsing has subtle UI demands • Cf. HCIR, visual analytics, visual searching, etc. 7 Mar. 2009

  27. Why juggling? Who cares? • A surprising domain, but realistic • Features 1 & 2 apply • Feature 3 applies in part: has established (though not graphic) notation, “siteswap” • Many juggling programs available • GTW framework has support for: • Control of tempo, including pausing or going backwards • UI for (temporal, not spatial) zooming in on details • Synchronization of multiple videos and/or animations • Framework for auto. synchronization • Framework for combining independent visualizations • Animal motion in general is much more complex => more need for GTW! • Ex: dancing (Labanotation, etc.)

  28. Structure in Basic Representations of Music & Audio Western Music Notation: very complex, irregular structure; some parts well-defined, some not—and what’s well-defined isn’t well-defined Audio: no explicit structure MIDI: simple, regular, well-defined structure 10 Feb. 09

  29. Basic Representations of Music & Audio 1. Audio (e.g., CD, MP3): like speech 2. Time-stamped Events (e.g., MIDI file): like unformatted text 3. Music Notation: like text with complex formatting • Time scales of graphs: #1, milliseconds; #2 & 3, seconds • Essential difference among forms: “knowledge representation” = explicit structure 10 Feb. 09

  30. “Isn’t it a mistake to use music notation this way?” • Chris Raphael’s question about Hendrix transcription • It’s obviously useful: easy to find phrases, “Taps”, etc. • …but seriously misleading in places • But CMWN is “always” misleading! • Is it useful enough to justify danger of misleading? • Knowledge representation has inevitable bias (Davis et al 1993); notation has more bias (Wiggins et al 1993) • Fundamental issue of transcription in ethnomusicology • Conclusion: use it, but be careful • Cf. my “Logician General’s Warning” on classification • …in fact, transcribing requires classifying constantly 12 Feb. 09

  31. Sequential-Art/Movie: The Hard Goodbye (1) • From Frank Miller’s “Sin City” series • John Walsh (SLIS): want to compare comics, movies of them, etc. 18 Feb. 2009

  32. Sequential-Art/Movie: The Hard Goodbye (2) • From Frank Miller’s “Sin City” series • John Walsh (SLIS): want to compare comics, movies of them, etc. 18 Feb. 2009

  33. Types of Visualizations of Music (and more) • Is visualization static or dynamic? • Dynamic = time represented by time • Static = time represented by space • What features are visualized? • What basic representation? Audio, symbolic, both? • Easy to generalize to plays (score = script) & other text phenomena, dance, etc. 18 Feb. 2009

  34. Types of Visualizations of Music (and more) • Hendrix example uses coordinated visualizations • Generalization of parallel, aligned, synchronized, etc. • How are multiple visualizations coordinated? • Parallel panes of a single window • Superimposed in a single window • Separate coordinated windows • Forms 1 & 2 apply directly to audio (incl. sonification) • Easy to interpolate between forms 1 & 2 • Categories in the real world are rarely discrete 26 Feb. 2009

  35. The Ultimate Music System • Original goal: visualizer that can do anything with music • Handle any no. & combination of visualizations • Static visualizations: audio, any kind of notation, structural diagrams, etc. • Dynamic visualizations: video, etc. • Automatic (or near-automatic) synchronization • Support OS-level technologies (QuickTime, etc.) • Easy-to-learn UI allowing high degree of control • User may want frequent extreme zoom changes => help with • If it could do all that, should be useful for lots (domains with >=2 Features) besides music! 20 May 08

More Related