350 likes | 357 Views
This paper introduces the concept of a General Temporal Workbench, a system that allows users to explore and analyze complex temporal phenomena across various disciplines. The workbench provides multiple ways of visualizing and comparing instances, specialized notation for music, and supports various media types such as video, audio, images, and text. The goal is to provide researchers and creators with a tool that enables them to gain insight into temporal phenomena and make meaningful connections across different fields.
E N D
The General Temporal Workbench A Universal System for Exploring Time-based Phenomena Donald Byrd Woodrow Wilson Indiana Teaching Fellow and Research Technologies & School of Informatics Indiana University Bloomington 2 May 2013
Introduction (1): Time Is of the Essence • Gandini Juggling’s Mozart (Symphony 25, 1st mvmt) performance • How created? How did jugglers learn it? • I don’t know, but I do know was a huge pain! • Likewise for doing almost anything w/ complex temporal phenomena • …and timescale can be yoctoseconds or billions of years • Natural phenomena • Inflation of early universe, chemical reactions, animal movement, weather, illnesses, climate; development of individuals, of species, of geographic features, & of galaxies • Artificial & cultural phenomena • Movies, behavior of computer programs, football games, multimedia shows, operas, political crises, wars, development of civilizations 1 May 2013
Introduction (2) • Complex temporal phenomena difficult to study or (for performances) to create, because don’t “sit still” • Can “play” repeatedly, but at expense of context • Usual answer: visualization • Good start, but we can (and need to) do much better than existing systems • For music, General MusicVisualizer (GMV) • Music ex: Hendrix/Star Spangled Banner: how to notate? • Musical audio: Shakuhachi solo: how to segment? (Nina Fales) • Might be a good solution to notating Hendrix! • Studying speech or animal sounds is a similar problem 3 July 2009
GMV “demo”: Segment shakuhachi solo • Scenario: music-informatics researcher or ethnomusicologist comparing two audio-segmentation algorithms • …or composer comparing input & output of synthesis programs 20 Apr. 2009
The Problem for “Multimedia” • “Multimedia” = performance w/ music & sync’ed movement (dance, etc.) or visuals (fireworks, etc.) • Requirements • 1. People need several ways to “look” at an instance simultaneously • 2. People need to be able to compare instances • 3. Has specialized notation (at least for music) • 4. Media: video, audio, still images, text • Audio independent of video is often important 1 May 2013
The Solution for Multimedia • General MusicVisualizer => General Multimedia Workbench (GMW) • Early prototype of GMW: AWB Plus • Based on EVIA Annotator’s Workbench • Java program, developed for ethnography • Queen of the Night aria, in “GMW” • Fireworks show with music, in AWB Plus 28 Mar. 2010
GMW “Screenshot”: Queen of the Night aria • Scenario: singer (or conductor) comparing videos of performances to learn role in opera, musical, etc. • …or stage director, choreographer, or lighting designer comparing previous versions to own ideas • …or scholar studying performances (perhaps juggling w/ music!) 4 Apr. 2009
Fireworks show with music, in AWB Plus • From video & still photos of show by Mike Durbin • All timestamped => (semi)automatic alignment • Music added afterwards! 28 Mar. 2010
The General Problem • Long-standing, difficult problems in all fields • Affect all uses: research, creation, “aesthetics”, organization • …plus deluge of data in many fields • Even arts & humanities are getting lots of hard data • Data promises to help, but not much help so far! • Need insight, not data; how to get there? • Widely recognized as important goal 1 May 2013
Towards a Solution (1) • Review: multimedia features • 1. People need several ways to “look” at instance simultaneously • 2. People need to be able to compare instances • 3. Has specialized notation (at least for music) • 4. Media: video, audio, still images, text • Observations • Multimedia features #1 & 2 are true of every (and 3 & 4 of many) complex temporal phenomenon • Every temporal phenomenon is complex • People often want to relate phenomenon of one type to another 1 May 2013
Towards a Solution (2) • The cross-discipline argument • Problems in all fields have much in common => potential for synergy • But people rarely share ideas or software across disciplines • Issue of “disjoint technical vocabulary/literature” (cf. Swanson 1988) • Ex: Are there periodic patterns in public opinion? => political scientists should try Fourier analysis, but… • The research vs. creation vs… argument • Problems of researchers, creators, etc. in a field have much in common • => A general temporal system could greatly reduce “reinventing the wheel” or doing without one 1 May 2013
The General Solution • General temporal system applicable to very many disciplines is possible • General temporal system applicable to all types of use is possible • GMusicW needs so much flexibility, a GAnythingW wouldn’t need much more • Oh, yeah? With what magic? • The magic of Software Architecture! 1 May 2013
Architecture for a General System (1) • Main tasks for framework: (temporal) synchronization, UI support • Software for common audio & video uses: 15 Mar. 2009
Architecture for a General System (2) For sequential art & movies based on sequential art (John Walsh): 15 Mar. 2009
General Temporal Toolkit/Workbench (GTT/W) • “Temporal”, not just “Multimedia” • Supports multiple: coordinated, editable, interactive visualizations & sonifications (eventually “tactilizations”, “olfactizations”?) • …of multiple instances • …of any combination of temporal phenomena • …plus data mining & analysis (Solution 2) • For all applications (research, creative, other) • Myosin V, in AWB Plus 5 Mar. 2010
Important Feature 1: Better Human/Machine Partnerships Figure slightly modified from Yu et al (2008) • Integrate info visualization & analysis/data mining (Shneiderman 2002; Yu et al 2008) => closed loop: use visual perception to generate hypotheses for analysis; present results of analysis visually • “HCIR”, visual analytics • …or substitute synthesis for analysis, e.g., for composers 7 Apr. 2009
Important Feature 2 & 3: Allow All Sensory Modes; Automatic Synchronization • Sensory Modes • Visual: visualization is most generally useful, but not the only answer • Auditory: sound is central for many applications • sonification is surely valuable for some non-audio phenomena • Tactile: important for the blind • Other (olfactory): important for ?? • Don’t rule out support for all sensory modalities • Automatic Synchronization • Of audio (at least) has made great progress in last 10 years 2 May 2013
Myosin V, in AWB Plus • From nanobioscience work by Craig & Linke • Myosin V crawling along a strand of acetin 28 Mar. 2010
Big Issue: Time Representation • Resolution/precision • Cosmology: Planck time to quadrillions(?) of years • Precision & handling of uncertainty • Historical forms: “1685?”, “fl. mid-15th century”, etc. • Damaged sources: “25th of [unreadable] 1743”, etc. • Handle with Probability Density Functions? • Absolute & relative time • Calendars are incredibly messy • Design of universal time representation is well along 1 May 2013
Huge Issue: Getting Off the Ground (1) • AWB Plus prototype, based mostly on EVIA Annotators Workbench; some of Variations • may also use CIShell, Chen Yu’s system for “visual mining of multimedia data” • all from IUB • Other possible open-source starting points • Sonic Visualiser, SyncPlayer, SIMILE Timeline, etc. • Connections to general tools for nontemporal visualization • Network Workbench (Katy Börner/IU SLIS): networks • Google Map API: “space” (surface of the earth) • Connections to semi-general tools • Ex: Max/MSP, Pd: musical audio, video • Ex: imagej for image display/processing 1 May 2013
Getting Off the Ground (2) • Support from communities of users is almost essential • Have seen strong interest for: • Ethnomusicology field work • Opera scholarship • “Could revolutionize study of opera” • Conducting • Teaching electronic music • Study of computer-aided musical improvisation • Study of comic books/movies • Research on artificial life • Software engineering, research software • Nanoscience/nanotechnology • Study of French 1 May 2013
Conclusions • How do I know applications are realistic? • Many probably aren’t, but many, many possibilities exist! • Have ca. 30 usage scenarios, ca. half written/endorsed by experts • Some examples (all experts in respective fields) • Ruth Stone: ethnomusicology field work • Philip Gossett, Nick Cook: study of music/opera performance • Paul Nadler: conducting • John Gibson: teaching electronic music • Elaine Chew: annotating video of computer-aided musical improvisation • John Walsh: comic books/movies • Larry Yaeger: artificial life • Fabian Fagerholm, Ben Kovitz: software engineering, research S/W • Amar Flood: nanoscience/nanotechnology • Personal knowledge/experience for a few 1 May 2013
End • Thanks to Eric Isaacson, Geoff Chirgwin, Will Cowan, Allen Winold, Paul Sturm, Ben Kovitz, Craig Stewart, Eric Wernert • THE END 1 May 2013
Extra Slides • Following slides (with background like this one) are just in case… rev. 18 Feb. 2009
Good Design for Music Can Be Good for Many Things • Cf. “Why Studying Music is Both Difficult and Important” (Byrd 2009) • Music is an art => people use elements in unusual ways • Music is a performing art =>performances & symbolic representations • Much music hascomplex synchronization requirements • Music involves many different instruments, often in groups. Leads to: • Arrangements/transcriptions for other instruments • Versions for players with different levels of skill • Notation may represent sounds or actions • Music is often combined with text via singing, narration, etc. • Music is extremely popular, so: • Some works exist in many versions, arrangements for different ensembles, etc. • Handling challenges is important, even on purely economic grounds rev. 18 Feb. 2009
HCI: Searching, Browsing, & Visualization • Visualization is essential for browsing, merely helpful for searching • In browsing, user finds everything; the computer just helps • Browsing is obviously good because it gives user more control, but few systems emphasize it. Why? • “Users are not likely to be pleasantly surprised to find that the library has something but that it has to be obtained in a slow or inconvenient way. Nearly all items will come from a search, and we do not know well how to browse in a remote library.” (Lesk, p. 163) • For “and”, read “as long as” • Searching is more natural on computer, browsing in real world • Effective browsing takes very fast computers—widely available now • Effective browsing has subtle UI demands • Cf. HCIR, visual analytics, visual searching, etc. 7 Mar. 2009
Why juggling? Who cares? • A surprising domain, but realistic • Features 1 & 2 apply • Feature 3 applies in part: has established (though not graphic) notation, “siteswap” • Many juggling programs available • GTW framework has support for: • Control of tempo, including pausing or going backwards • UI for (temporal, not spatial) zooming in on details • Synchronization of multiple videos and/or animations • Framework for auto. synchronization • Framework for combining independent visualizations • Animal motion in general is much more complex => more need for GTW! • Ex: dancing (Labanotation, etc.)
Structure in Basic Representations of Music & Audio Western Music Notation: very complex, irregular structure; some parts well-defined, some not—and what’s well-defined isn’t well-defined Audio: no explicit structure MIDI: simple, regular, well-defined structure 10 Feb. 09
Basic Representations of Music & Audio 1. Audio (e.g., CD, MP3): like speech 2. Time-stamped Events (e.g., MIDI file): like unformatted text 3. Music Notation: like text with complex formatting • Time scales of graphs: #1, milliseconds; #2 & 3, seconds • Essential difference among forms: “knowledge representation” = explicit structure 10 Feb. 09
“Isn’t it a mistake to use music notation this way?” • Chris Raphael’s question about Hendrix transcription • It’s obviously useful: easy to find phrases, “Taps”, etc. • …but seriously misleading in places • But CMWN is “always” misleading! • Is it useful enough to justify danger of misleading? • Knowledge representation has inevitable bias (Davis et al 1993); notation has more bias (Wiggins et al 1993) • Fundamental issue of transcription in ethnomusicology • Conclusion: use it, but be careful • Cf. my “Logician General’s Warning” on classification • …in fact, transcribing requires classifying constantly 12 Feb. 09
Sequential-Art/Movie: The Hard Goodbye (1) • From Frank Miller’s “Sin City” series • John Walsh (SLIS): want to compare comics, movies of them, etc. 18 Feb. 2009
Sequential-Art/Movie: The Hard Goodbye (2) • From Frank Miller’s “Sin City” series • John Walsh (SLIS): want to compare comics, movies of them, etc. 18 Feb. 2009
Types of Visualizations of Music (and more) • Is visualization static or dynamic? • Dynamic = time represented by time • Static = time represented by space • What features are visualized? • What basic representation? Audio, symbolic, both? • Easy to generalize to plays (score = script) & other text phenomena, dance, etc. 18 Feb. 2009
Types of Visualizations of Music (and more) • Hendrix example uses coordinated visualizations • Generalization of parallel, aligned, synchronized, etc. • How are multiple visualizations coordinated? • Parallel panes of a single window • Superimposed in a single window • Separate coordinated windows • Forms 1 & 2 apply directly to audio (incl. sonification) • Easy to interpolate between forms 1 & 2 • Categories in the real world are rarely discrete 26 Feb. 2009
The Ultimate Music System • Original goal: visualizer that can do anything with music • Handle any no. & combination of visualizations • Static visualizations: audio, any kind of notation, structural diagrams, etc. • Dynamic visualizations: video, etc. • Automatic (or near-automatic) synchronization • Support OS-level technologies (QuickTime, etc.) • Easy-to-learn UI allowing high degree of control • User may want frequent extreme zoom changes => help with • If it could do all that, should be useful for lots (domains with >=2 Features) besides music! 20 May 08