Movies

1. Movies Barb Ericson ericson@cc.gatech.edu Georgia Tech

2. Movies, animations, and video…oh my! • We’re going to refer generically to captured (recorded) motion as “movies.” • This includes motion entirely generated by graphical drawings, which are normally called animations. • This also includes motion generated by some kind of photographic process, normally called video.

3. Psychophysics of Movies:Persistence of Vision • What makes movies work is yet another limitation of our visual system: Persistence of vision • We do not see every change that happens in the world around us. • Instead, our eye retains an image (i.e., tells the brain “This is the latest! Yup, this is still the latest!”) for a brief period of time. • If this were not the case, you would be aware of every time that your eye blinks because the world would “go away” for a moment.

4. 16 frames and it’s motion • If you see 16 separate pictures in one second, and these pictures are logically sequenced, • That is, #2 could logically follow from the scene in #1. • 16 pictures of completely different things doesn’t work, • You will perceive the pictures as being in motion. • 16 frames per second (fps), 16 pictures in a second, is the lower bound for the sensation of motion.

5. Beyond 16 fps • Early silent pictures were 16 fps. • Motion picture standards shifted to 24 fps to make sound smoother. • Videocameras (digital video) captures 30 fps • How high can we go? • Air force experiments suggest that pilots can recognize a blurb of light in 1/200th of a second! • Video game players say that they can discern a difference between 30 fps and 60 fps. • Bottomlines: • Generate at least 16 fps and you provide a sense of motion. • If you want to process video, you’re going to have 30 fps to process (unless it’s been modified elsewhere for you.)

6. Processing movies • Our frames are going to be JPEG pictures. • One JPEG file per frame. • So, if we’re going to be processing movies, we’re going to generating or processing sequences of JPEG files.

7. Using MediaTools • To generate a series of frame pictures in a folder from an MPEG file. • To play a folder of frame pictures and to save it as a JMV file. • (JPEG Movie format.) • To play JMV or MPEG movies.

8. MPEG? QuickTime? AVI? JMV? • MPEG, QuickTime, and AVI are compressed movie formats. • They don’t record every frame. • Rather, they record some key frames, and then store data about what parts of the screen change on intervening frames. • MPEG is an international standard, from the same people who invented JPEG. • AVI is a Microsoft standard. • QuickTime is an Apple standard. • JMV is a file consisting of JPEG frames in an array. • All frames represented

9. Generating other kinds of movies • Other tools can generate other kinds of movie formats. • QuickTime Pro (http://www.apple.com/quicktime) can read a sequence of JPEG images and produce MPEG, AVI, or QuickTime movies. • ImageMagick (open source toolkit) can also read a sequence of JPEG images and produce MPEG movies.

10. Why do we compress movies? • Do the math: • One second of 640x480 pixels at 30 fps • 30 (frames) * 640 * 480 (pixels) = 9,216,000 pixels • With 3 bytes of color per pixel, that’s 27,648,000 bytes or 27 megabytes of information per second. • For a 90 minute feature movie (short), that’s 90 * 60 * 27,648,000 = 149,299,200,000 bytes (149 gigabytes) • A DVD stores 6.47 gigabytes of data. • So even on a DVD, the movie is compressed.

11. MPEG movie = MPEG frames plus MP3 soundtrack • An MPEG movie is actually a series of MPEG frames composed with an MP3 soundtrack. • It’s literally two files stuck together in one. • We’re not going to deal with sound movies for now. • The real challenge in doing movie processing is generating and manipulating frames.

12. Get the frames in order • Many tools (including os.listdir()) can process frames in order if the order is specified. • We specify the order by encoding the number of the frame into the name. • If you put in leading zeroes so that everything is the same length, the order is alphabetical as well as numerical.

13. Simple Motion def makeRectMovie(directory ): for num in range (1 ,30): #29 frames (1 to 29) canvas = makeEmptyPicture (300 ,200) addRectFilled(canvas ,num * 10, num * 5, 50,50, red) # convert the number to a string numStr=str(num) if num < 10: writePictureTo(canvas ,directory+"\\frame0"+numStr+".jpg") if num >= 10: writePictureTo(canvas ,directory+"\\frame"+numStr+".jpg") movie = makeMovieFromInitialFile(directory+"\\ frame01.jpg") return movie

14. Try it out >>> rectM = makeRectMovie("c:\\Temp\\rect\\") >>> playMovie(rectM) • The directory must exist before you call makeRectMovie. It is where the frames for the movie will be written.

15. When num = 1, addRectFilled(canvas,10,5,50,50,red) When num = 2, addRectFilled(canvas,20,10,50,50,red) When num = 10, addRectFilled(canvas,100,50,50,50,red) When num = 20, addRectFilled(canvas,200,100,50,50,red) The trick here is all mathematics

16. Important cool thing: You can draw past the end of the picture! • addText, addRect, and the rest of the drawing tools will work even if you go beyond the edge of the drawing. • Drawings will clip what can’t be seen in them, so you don’t get an array out of bounds error. • This is a big deal, because it means that you don’t have to do complicated math to see when you’re past the end of the drawing. • But only for the drawing functions. • If you set pixels, you’re still on your own to stay in range.

17. Making a tickertape movie def tickertape(directory,string): for num in range(1,100): #99 frames canvas = makeEmptyPicture(300,100) #Start at right, and move left addText(canvas,300-(num*10),50,string) # Now, write out the frame # Have to deal with single digit vs. double digit frame numbers # differently numStr=str(num) if num < 10: writePictureTo(canvas,directory+"\\frame0"+numStr+".jpg") if num >= 10: writePictureTo(canvas,directory+"\\frame"+numStr+".jpg")

18. Tickertape Movie frame10 frame40 frame20 frame30

19. Can we move more than one thing at once? Sure! def movingRectangle2(directory ): for num in range (1 ,30): #29 frames canvas = makeEmptyPicture (300 ,250) # add a filled rect moving linearly addRectFilled(canvas ,num*10,num*5, 50,50,red) # Let’s have one just moving around blueX = 100+ int (10 * sin(num)) blueY = 4*num+int (10* cos(num)) addRectFilled(canvas ,blueX ,blueY ,50,50, blue)

20. Moving two things at once -cont # Now , write out the frame # Have to deal with single digit vs. double digit numStr=str(num) if num < 10: writePictureTo(canvas ,directory +"\\frame0 "+ numStr +". jpg") if num >= 10: writePictureTo(canvas ,directory +"\\frame "+ numStr +". jpg")

21. blueX = 100+ int (10 * sin(num)) blueY = 4*num+int (10* cos(num)) addRectFilled(canvas ,blueX ,blueY ,50,50, blue) • What’s going on here? • Remember that both sine and cosine vary between +1 and -1. • Int(10*sin(num)) will vary between -10 and +10 • With cosine controlling y and sine controlling x, should create circular motion • num=1 • x is 108, y is 9 • num=2 • x is 109, y is 4

22. Frames from two motions at once

23. Slowly making it (very) sunset • Remember this code? def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.7) value=getGreen(p) setGreen(p,value*0.7) • What if we applied this to create frames of a movie, but slowly increased the sunset effect?

24. SlowSunset Just one canvas repeatedly being manipulated def slowSunset(directory): canvas = makePicture(getMediaPath("beach-smaller.jpg")) #outside the loop! for frame in range(1,100): #99 frames printNow("Frame number: "+str(frame)) makeSunset(canvas) # Now, write out the frame writeFrame(frame,directory,canvas) def makeSunset(picture): for p in getPixels(picture): value=getBlue(p) setBlue(p,value*0.99) #Just 1% decrease! value=getGreen(p) setGreen(p,value*0.99)

25. Pull out writeFrame def writeFrame(num ,dir ,pict ): # Have to deal with single digit vs. double digit numStr=str(num) if num < 10: writePictureTo(pict ,dir+"// frame00"+numStr+".jpg") if num >= 10 and num <100: writePictureTo(pict ,dir+"// frame0"+numStr+".jpg") if num >= 100: writePictureTo(pict ,dir+"// frame"+numStr+".jpg")

26. SlowSunset frames

27. Dealing with real video • We really can’t deal with real video. • Dealing with each frame takes a lot of processing. • If you were going to process each frame as fast as it was coming in (or going out), you’d have 1/30th of a second to process each frame! • We cheat by • Saving each frame as a JPEG image • Processing the JPEG images • Convert the frames back to a movie