Digital Media: Why Do We Need It, and How Do We Teach It as REAL Computer Science?

1. Digital Media: Why Do We Need It, and How Do We Teach It as REAL Computer Science? Jennifer Burg Department of Computer Science Wake Forest University Winston-Salem, NC 27109 http://www.cs.wfu.edu/~burg burg@wfu.edu This material is based on work supported by the National Science Foundation under Grants No. DUE-0127280and DUE-0340969.

2. Overview of Curriculum Material: • Primer* • Computer Science • Art* Three companion books to be published by Prentice-Hall, with on-line learning supplements. *Primer and Art books are being written by Dr. Yue-Ling Wong, Wake Forest University.

3. Computer Science Material

4. Curriculum Material Creates a Bridge Between Digital Media as Computer Science and Digital Media As Art • Primer (Yue-Ling Wong) – foundational material for computer scientists and artists; tied more to practice, but introducing the science behind the tools in a “gentle” way • The Science of Digital Media (Jennifer Burg) – for computer science majors; focuses on the science and mathematics behind digital media tools like Photoshop, Audition, Illustrator, Premiere, Flash, and Director • The Art of Digital Media (Yue-Ling Wong) – for digital art students

5. Components of The Science of Digital Media • Textbook • Interactive tutorials • Worksheets • Programming exercises • MATLAB mathematical modeling exercises (all currently available on the web at http://digitalmedia.wfu.edu)

6. All material is freely available on the web at http://www.wfu.edu. • Answer sheets to worksheets and implemented programs are at a password-protected site. • This talk will focus on The Science of Digital Media.

7. How I Use the Material • A course called Digital Media covers Chapters 1 through 5 and is taught to computer science majors (juniors and seniors). • Pre-requisites are Calc I, CS1 and CS2, and data structures. • Chapters 1 through 5 cover digital imaging and digital audio. • Digital video is introduced in the course (briefly). • In class, we cover the science and mathematics of digital media. • “Behind the scenes,” students do a game program as their semester-long term project, putting together digital photographs, vector graphics digital sound, and interactivity using Macromedia Director and its scripting language Lingo. • Students are expected to learn “hands-on” things on their own. • There is a pre-test and post-test for each chapter. • Students also do worksheets and on-line tutorials. (Note that in this course, we do not do the programming exercises.) • Second course called Digital Video and Animation, does an overview of central concepts of Chapters 1 and 5 and then focuses on Chapters 6-8, which cover digital video and multimedia programming.

8. Software Needed for the Courses Described on Previous Slide • Image processing (e.g., Photoshop) • Vector graphics (e.g., Illustrator or Freehand) • Digital audio processing (e.g., Audition or Sound Forge) • Video processing (e.g., Premiere or Final Cut) • Multimedia programming (e.g., Director of Flash) (I use Director). • Optional -- A programming environment (e.g., C++ or Java), if you choose to do the programming assignments. • Optional – MATLAB or Mathematica for mathematical modeling exercises. (If your students don’t have access to MATLAB or Mathematica, but YOU do, you can do the mathematical modeling exercises as in-class demos.

9. Alternative Ways to Use Computer Science Curriculum Material • Teach a course in Algorithms for Digital Media and focus on the programming assignments. OR • Focus on only one medium per course (digital imaging, audio, or video). OR • Cover all of the book (The Science of Digital Media) in an undergraduate course, but not in complete depth. Then use the entire book again in a graduate course and go into more depth in the mathematics and science.

10. Discussion:Are courses in Digital Media suitable for a rigorous computer science program? Is Digital Media REALLY computer science? Yes! Not only is it computer science, but it’s a more interesting way to engage the students’ interest in science.

11. Imagine • You are students. • I’m your teacher teaching you about two-dimensional arrays. “Two-dimensional arrays are data elements stored in contiguous blocks of memory, and blah blah blah…” Scribble scribble scribble on the board…

12. OR… I can show you what you can do with two-dimensional arrays. Image dithering

14. What can students learn in this exercise? • Two-dimensional arrays, matrix operations • Dynamically vs. statically allocated arrays • Array/index vs. pointer/offset notation • File I/O • Not to mention • Loops • Control structures • Function call and return • Etc.

15. And there are other digital media topics that deal with two-dimensional arrays Convolutions and unsharp mask

16. Imagine • You are students. • I’m your teacher teaching you about trees. “A tree consists of nodes. A non-empty tree has a root node, and n child nodes, where n >= 0, and blah blah blah…” Scribble scribble scribble on the board…

17. OR… I can show you what you can do with a tree. Octree algorithm for indexed color

18. indexed color by Photoshop our own implementation of indexed color

19. This version has more color gradient. With some fine-tuning of the algorithm, we can do even better.

20. Octree Algorithm for Indexed Color When the algorithm reaches a point where it wants to create a 257th leaf node This is what an image looks like when the node with the most children is collapsed. This is what an image looks like when the node with the least children is collapsed. Note that there is more color detail in the sky. Photographs courtesy of Kristian Damkjer, who used this curriculum material at the University of Florida in the fall of 2005.

21. What can students learn in this exercise? • The tree data structure • Recursion and recursive descent of a tree

22. Students also can be motivated to learn mathematics through digital media Show how colors can be modeled mathematically and understood as occupying a graphical space.

23. Students can be given a motivation for learning logarithms Human perception of sound intensity is proportional to the logarithm of the air pressure amplitude. This is why we use decibels to measure sound. Decibels are defined in terms of logarithms.

24. We can capture the students’ interest in fairly complex mathematical operations – e.g., the Fourier transform.

25. In short, relating concepts to images, sound, motion, and interactivity motivates students to learn.

26. Where does digital media* currently fit in the computer science curriculum? • Not included in ACM’s list of major subdisciplines within computer science • computer science • information systems • software engineering • computer engineering • information technology (See http://www.acm.org/education/curric_vols/CC2005_Final_Report2.pdf.) *I’ll use the term “digital media” to denote a field of study and thus will consider it a singular noun. (Just didn’t want you to think I don’t know my grammar. )

27. Where does digital media currently fit in the computer science curriculum? • In Computer Science Computing Curricula 2001, not included as one of the essential courses or areas within traditional computer science • i.e., architecture, graphics, databases, operating systems, algorithms • The closest thing is “Graphics and Visual Computing,” with some elements covered in Net-Centric Computing, Algorithms and Complexity, and Programming Languages • See http://www.computer.org/portal/cms_docs_ieeecs/ieeecs/education/cc2001/cc2001.pdf and http://www.cs.wfu.edu/~burg/papers/BurgWongStrokSIGCSE2004.pdf

28. Where does digital media currently fit in the computer science curriculum? • Digital media courses in computer science departments exist mostly as electives. • Digital media is sometimes covered superficially in CS100-type courses. • Digital media also appears as • an interdisciplinary field of study • a course or courses within disciplines outside of computer science (e.g., art, communications) • a department all its own

29. Why do we need digital media? • Enrollments are falling in computer science. • Even students who like working with computers may perceive computer science as irrelevant and detached from the REAL world of computers. • We don’t always create a good link between science and practice. Digital media is an ideal area in which to make this connection.

30. Why have computer scientists been reluctant to include digital media in the computer science curriculum? • It seems inherently interdisciplinary. • (Well, that’s one of its charms!) • It seems SOFT! • (Really, it’s NOT! Consider the topics in the computer science curriculum material at http://digitalmedia.wfu.edu.)

31. Combining the science of digital media with the practice of digital media • Emphasize the science in classroom teaching, readings, worksheets, etc. • In “the background,” students are working with application programs such as Photoshop, Audition, Premiere, Flash, Director, and/or Javascript. • I have my students do a game program as their term project, developed incrementally as they study each medium. • Another good assignment is an on-line resume/porfolio.

32. Just in case you wondered: Why do I use Director instead of Flash for on-line tutorials? • Director is more powerful than Flash in the following ways: • Better support for video formats. • Better support for sound. • More Xtras. • Allows for launching of external applications. • Director now allows you to use Javascript syntax as an alternative to Lingo. • See http://www.macromedia.com/software/director/resources/integration/flash/quick_comparison.html

33. After the break • Do tutorials and exercises • Do a mathematical modeling worksheet • Discuss ways to use the material • Discuss your own curriculum needs • Talk about good projects and assignments • Talk about whether digital media really should be in computer science

34. The Challenge • Link science to practice • Show that knowing the science results in better practice • Get scientists and artists communicating and collaborating on creative projects