Teaching Programming to Everyone through Media Computation

1. Teaching Programming to Everyone through Media Computation Mark Guzdial College of Computing/GVU Georgia Institute of Technology

2. Story • Perlis’ challenge: Computer science is more important than calculus • Evidence that we’re not there yet • And what the solution might look like • Our attempt: Introduction to Media Computation • Potential tie to mathematics classes • Results so-far

3. Computer science is more important than Calculus • In 1961, Alan Perlis argued that computer science is more important in a liberal education than calculus • Explicitly, he argued that all students should learn to program. • Calculus is about rates, and that’s important to many. • Computer science is about process, which is important to everyone

4. How close are we to being able to teach everyone CS? • Not very • CS1 is one of the most despised courses for non-majors • At many departments, CS retention rates are lower than the rest of campus • At Georgia Tech: 65% for 1995 cohort, vs. 73% for Engineeering • Drop-out rates near 50% at many institutions • Female enrollment in CS has been dropping nationally

5. Why? • Several recent studies and books claim that CS instruction tends to dissuade anyone but white males • “Tedious,” “taught without application relevance,”“boring,” “lacking creativity,” “asocial”

6. The best uses for computing technologies will come from others • Thomas Edison vs. D.W. Griffith • Suggestion: D.W. Griffith knew things that Edison didn’t. • Consider George Lucas today • If we want computing technologies to become useful, they have to get out of our hands. • It can’t be just through applications. • Computer science will never have the potential that it might, if future practitioners hate our introductory course!

7. The Challenges • We need to motivate CS, potential CS, and non-CS students to care about computing • We need to make it social, creative, relevant, exciting, and not tedious • Which is how many of us already see Computing, but that’s not getting communicated

8. Our Attempt: Introduction to Media Computation • A course for non-CS and non-Engineering majors • International Affairs, Literature, Public Policy, Architecture, Management, Biology, etc. • 120 students this semester,planning 400-600 in the Fall • 2/3 female in this semester’s CS1315 • Focus: Learning programming within the context of media manipulation and creation

9. Motivating the Computing • As professionals, these students will often the use the computer as a communications medium. • All media are going digital,and digital media are manipulated with software. • Knowing how to program, then, is a communications skill.

10. Programming as a Communications Skill • Knowing how to program means to understand one’s tools. • Maybe means can transfer tool skills more easily • Students tell us that they’re excited to learn how PhotoShop works. • And it means that, if you have to, you may be able to grow your own

11. Python as the programming language • Huge and contentious issue • Use in commercial contexts legitimizes the choice • Industrial Light & Magic, Google, Nextel, etc. • Minimal syntax • Looks like other programming languages • Potential for knowledge transfer • Actually using Jython (http://www.jython.org) for Java class libraries

12. def clearRed(picture): for pixel in getPixels(picture): setRed(pixel,0) def greyscale(picture): for p in getPixels(picture): redness=getRed(p) greenness=getGreen(p) blueness=getBlue(p) luminance=(redness+blueness+greenness)/3 setColor(p, makeColor(luminance,luminance,luminance)) def negative(picture): for px in getPixels(picture): red=getRed(px) green=getGreen(px) blue=getBlue(px) negColor=makeColor(255-red,255-green,255-blue) setColor(px,negColor)

13. def chromakey(source,bg): for x in range(1,getWidth(source)): for y in range(1,getHeight(source)): p = getPixel(source,x,y) # My definition of blue: If the redness + greenness < blueness if (getRed(p) + getGreen(p) < getBlue(p)): #Then, grab the color at the same spot from the new background setColor(p,getColor(getPixel(bg,x,y))) return source

14. Use a loop!Our first picture recipe original def decreaseRed(picture): for p in getPixels(picture): value=getRed(p) setRed(p,value*0.5) Used like this: >>> file="/Users/guzdial/mediasources/barbara.jpg" >>> picture=makePicture(file) >>> show(picture) >>> decreaseRed(picture) >>> repaint(picture)

15. Recipe to Increase the Volume def increaseVolume(sound): for sample in getSamples(sound): value = getSample(sample) setSample(sample,value * 2) Using it: >>> f="/Users/guzdial/mediasources/gettysburg10.wav" >>> s=makeSound(f) >>> increaseVolume(s) >>> play(s) >>> writeSoundTo(s,"/Users/guzdial/mediasources/louder-g10.wav")

16. A Sunset-generating function • How do we turn this beach scene into a sunset? • What happens at sunset? • Tried increasing the red, but that failed. • New Theory: As the sun sets, less blue and green is visible, which makes things look more red.

17. A Sunset-generation Function def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7)

18. SlowSunset Just one canvas repeatedly being manipulated def slowsunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(0,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas) def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99) Not showing you writeFrame() because you know how that works.

19. SlowSunset frames

20. Introducing IF:Making Barb a redhead def turnRed(): brown = makeColor(57,16,8) file = r"C:\Documents and Settings\Mark Guzdial\My Documents\mediasources\barbara.jpg" picture=makePicture(file) for px in getPixels(picture): color = getColor(px) if distance(color,brown)<50.0: redness=getRed(px)*1.5 setRed(px,redness) show(picture) return(picture) Original:

21. Generalizing Algorithms • We talk about algorithm complexity later in the course, after the media is done. • We talk about different approaches to the same problem, where the criteria might be aesthetics or correctness, instead of speed or size • For example, generating greyscale • During the media, we point out similar themes in different functions. • We refer to them as “sub-recipes”

22. Scaling the picture down def copyBarbsFaceSmaller(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)/2)): sourceY = 25 for targetY in range(100,100+((200-25)/2)): color = getColor(getPixel(barb,sourceX,sourceY)) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 2 sourceX = sourceX + 2 show(barb) show(canvas) return canvas

23. Scaling the picture up def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor(getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas

24. Recipe for halving the frequency of a sound This is how a sampling synthesizer works! def half(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5 play(target) return target Here’s the piece that does it

25. Compare these two def copyBarbsFaceLarger(): # Set up the source and target pictures barbf=getMediaPath("barbara.jpg") barb = makePicture(barbf) canvasf = getMediaPath("7inX95in.jpg") canvas = makePicture(canvasf) # Now, do the actual copying sourceX = 45 for targetX in range(100,100+((200-45)*2)): sourceY = 25 for targetY in range(100,100+((200-25)*2)): color = getColor( getPixel(barb,int(sourceX),int(sourceY))) setColor(getPixel(canvas,targetX,targetY), color) sourceY = sourceY + 0.5 sourceX = sourceX + 0.5 show(barb) show(canvas) return canvas def half(filename): source = makeSound(filename) target = makeSound(filename) sourceIndex = 1 for targetIndex in range(1, getLength( target)+1): setSampleValueAt( target, targetIndex, getSampleValueAt( source, int(sourceIndex))) sourceIndex = sourceIndex + 0.5 play(target) return target

26. Both of them are sampling • Both of them have three parts: • A start where objects are set up • A loop where samples or pixels are copied from one place to another • To decrease the frequency or the size, we take each sample/pixel twice • In both cases, we do that by incrementing the index by 0.5 and taking the integer of the index • Finishing up and returning the result

27. Ties to Mathematics • I don’t know how to exploit them yet, but significant mathematics issues do arise here. • Integration, in understanding how to digitize sound (and pictures, to an extent) • Discrete representations of motion and curves.

28. Digitizing Sound: How do we get that into numbers? • Remember in calculus, estimating the curve by creating rectangles? • We can do the same to estimate the sound curve • Analog-to-digital conversion (ADC) will give us the amplitude at an instant as a number: a sample • How many samples do we need?

29. Nyquist Theorem • We need twice as many samples as the maximum frequency in order to represent (and recreate, later) the original sound. • The number of samples recorded per second is the sampling rate • If we capture 8000 samples per second, the highest frequency we can capture is 4000 Hz • That’s how phones work • If we capture more than 44,000 samples per second, we capture everything that we can hear (max 22,000 Hz) • CD quality is 44,100 samples per second

30. Simple Motion def movingRectangle(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

31. A Few Frames frame00.jpg frame02.jpg frame50.jpg

32. if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) When frame = 1, addRectFilled(canvas,10,5,50,50,red) When frame = 2, addRectFilled(canvas,20,10,50,50,red) When frame = 49, addRectFilled(canvas,490,285,50,50,red) When frame = 50, 50-(50-50) = 50 addRectFilled(canvas,500,250,50,50,red) When frame = 51, 50-(51-50)=50-1=49 addRectFilled(canvas,490,285,50,50,red) When frame = 99, 50-(99-50)=50-49=1 addRectFilled(canvas,10,5,50,50,red) The trick here is all mathematics

33. Can we move more than one thing at once? Sure! def movingRectangle2(directory): for frame in range(0,100): #99 frames canvas = makePicture(getMediaPath("640x480.jpg")) if frame < 50: #Less than 50, move down # Generate new positions each frame number addRectFilled(canvas,frame*10,frame*5, 50,50,red) if frame >= 50: #Greater than 50, move up addRectFilled(canvas,(50-(frame-50))*10,(50-(frame-50))*5, 50,50,red) # Let's have one just moving around addRectFilled(canvas,100+ int(10 * sin(frame)),4*frame+int(10* cos(frame)),50,50,blue) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers differently framenum=str(frame) if frame < 10: writePictureTo(canvas,directory+"//frame0"+framenum+".jpg") if frame >= 10: writePictureTo(canvas,directory+"//frame"+framenum+".jpg")

34. addRectFilled(canvas,100+ int(10 * sin(frame)), 4*frame+int(10* cos(frame)),50,50,blue) • What’s going on here? • Remember that both sine and cosine vary between +1 and -1. • Int(10*sin(frame)) will vary between -10 and +10 • With cosine controlling y and sine controlling x, should create circular motion • frame=1 • x is 108, y is 9 • frame=2 • x is 109, y is 4

35. Frames from two motions at once

36. Using your personal pictures

37. And messin’ with them

38. Data-first Computing • Real users come to a user with data that they care about, then they (unwillingly) learn the computer to manipulate their data as they need. • CS1315 works the same. • We use pictures of students in class demonstrations. • Students do use their own pictures as starting points for manipulations. • They started doing this in the second week • How often do students use their second week of CS1 on their own data? • How does that change the students’ relationship to the material?

39. Rough overview of Syllabus • Defining and executing functions • Pictures • Psychophysics, data structures, defining functions, for loops, if conditionals • Sounds • Psychophysics, data structures, defining functions, for loops, if conditionals • Text • Converting between media, generating HTML, “flattening” media and saving to a database • Movies • Then, Computer Science

40. Computer science as a solution to their problems • “Writing programs is hard! Are there ways to make it easier? Or at least shorter?” • Object-oriented programming • Functional programming and recursion • “Movie-manipulating programs take a long time to execute. Why?” • Algorithmic complexity • “Why is PhotoShop so much faster?” • Compiling vs. interpreting • Machine language and how the computer works

41. Tools to support media computation MediaTools for exploring media JES: Jython environment for students

42. Assignments encourage collaboration • Homework are all collaborative • Quizzes are preceded by nearly-identical, collaborative pre-quizzes • Two “take-home exams” (programming assignments) are non-collaborative • “Lablets” on application software are collaborative

43. Assignments encourage creativity • For several homeworks, the task is to manipulate media in some way, but we don’t care what media • For example, creating a collage or building an animation • Encouraging homework results to be posted to CoWeb (collaborative website) in galleries

44. First Homework assignment Homework 1: Write a program named hw1 to accept a picture as input, and change its pixels as follows: • Set the green component to 125% of its current value • Decrease the blue by 25% • Decrease the red by 75%

45. Solutions shared in the CoWeb

46. Grade distribution Much better than anticipated.

47. Homework #3: Make a collage with images that you modify by code only—any images you want

48. Grades on Homework #3

49. Take-Home Exam #2 Given a folder with images and sounds in it, create an index HTML page with links to each image and sound. You will write a function called indexPage that takes a string which is the path to a directory. You will create a page in that directory named index.html. Index.html should be an HTML page containing a link to every JPEG file and every WAV in the directory. At the top of the page, put a heading (level 1) with the phrase "Directory listing of sounds and images in " and then the directory, e.g., "Directory listing of sounds and images in C:\Documents and Settings\Mark Guzdial\mediasources" The links should be each an item in an unordered (<ul>) list. The anchor text in each link should be the filename of the image or sound. The destination (href) should be the same filename. • For each image, on the same line as the filename, list the horizontal and vertical size (in pixels) of the image. • For each sound, list the length of the sound in seconds. This is a NON-COLLABORATIVE ACTIVITY! You may not talk to anyone about their code, nor look at anyone else's code, nor allow anyone to see your code. This is a TAKE HOME EXAM. It is an "open book" exam. You may use your book, any slides, any material in the CoWeb, and any programs you've written (even with others) that you already have direct access to.When you turn in your exam, you are to enter into the Comment area the statement: "I did not provide nor receive any aid on this exam." IF YOU CANNOT MAKE THAT STATEMENT TRUTHFULLY, DO NOT SUBMIT YOUR EXAM! ANY EXAM WITHOUT THAT STATEMENT WILL NOT BE GRADED.

50. Take-Home Exam Results