Cosc 5/4730

1. Cosc5/4730 Mobile Media API (MMAPI) v1.1 JSR 135 and Android media Part 2: Pictures and Video

2. Emulators • The emulators all have problems, much of the following code has been tested on the actual phones • Video playback code works, but the video may not always displayin the emulators • android • Video/picture capture does not work as documented. • Blackberry • Camera, you can choose a picture with the emulator to use or some emulators will use your background image • Video capture does not work as documented.

3. Blackberry MMAPI Play media

4. Video formats. • MP4 • H.264, MPEG-4 with simple profile level 3, and H.263 • AVI (MPEG4 profile level 3) • WMV (simple profile, WMV3) • at most the video resolution can be the size screen of the device. • 30 frames per second

5. Playing video • Similar to audio • use the Player, realize, play, etc… • But you need a “View Player field” • same idea of the “View finder” when we get to taking a picture • We get the viewer from the VideoControl object. • We can use the MediaKeyListener for “media” key events (assuming the phone has them), but there is no easy way to display controls, like play, pause, etc..

6. The Basics • Just like audio, get the player and realize() player = javax.microedition.media.Manager.createPlayer( "file:///SDCard/BlackBerry/videos/the-empire.3gp"); player.realize(); • Get the VideoControl pieces and get a “View Player field” VideoControlvideoControl = (VideoControl) player.getControl( "VideoControl"); Field videoField = (Field) videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field" ); • add the field to the screen add(videoField); • I found it didn’t work unless I went fullscreen. audio worked, but no video videoControl.setDisplayFullScreen(true); • start playing. player.start();

7. MediaKeyListenerPlayListener • Use the PlayListener so we figure out when the media has finished and be able to close down stuff. • Plus deal with other issues as necessary • use the MediaKeyListener to deal with “standard” blackberry media buttons as well.

8. For simplicity… • Since I had to run in full screen • I added menus to play, pause, stop. • volume up/down are handled by the MediaKeysListener, plus the rest media keys too. • Except they don’t exist on the phones I have access too. • The mute button on many BB is a play/pause toggle button. Including bold and storm.

9. Volume Control • Handled via a VolumeControl object. volume = (VolumeControl) player.getControl("VolumeControl"); • intgetlevel() and setLevel(int) • setMute(boolean) • true mute, false unmute;

10. seeking • player.getMediaTime() returns a long that represents the spot in the media • 1,000,000 is about 1 sec • For a 102 second video, the duration is 102,898,000 • since I can’t see smaller then a second, assuming the video is 102 seconds + almost 1 more second. • player.setMediaTime(long) allows you to change the position it is playing.

11. Buffering. • Get content off the web • like I do with many examples… except buffering doesn’t work. • The advanced Blackberry Development book has an example of how to create a streaming video buffer from none streaming media, such mp4 and 3gp files.

12. Code example • To run video player example • Note, it assumes you have an SDCARD and that file:///SDCard/BlackBerry/videos/the-empire.3gp this file exists. • It’s in the /res/ directory.

13. Using native player • Blackberry has a native media player. • You can launch it to play audio/video • Remember you lose control and you app is now in the background. • First get the registry with you full class name. Registry reg = Registry.getRegistry( "edu.cs4755.VideoCapture"); • Then create a invocation of the file you want to play and invoke it. Invocation invocation = new Invocation( "file:///SDCard/Blackberry/videos/testrecording.3gp"); reg.invoke(invocation); • If this all reminds you of a Android, it’s very close an intent.

14. Android android.media Play media

15. Supported formats • In general Android’s support is consistent with other mobile phones. • It supports the 3GP (.3gp) and MPEG-4 (.mp4) file formats. • 3GP is a video standard derived from MPEG-4 specifically for use by mobile devices. • As far as codecs go, Android supports H.263, a codec designed for low-latency and low-bitrate videoconferencing applications. H.263 video is supported in either MPEG-4 (.mp4) or 3GP (.3gp) files. Android also supports MPEG-4 Simple Profile in 3GP files (.3gp) as well as H.264.

16. Android • First method • For greater flexibility you will need to use the mediaPlayer and a surfaceView • MediaPlayer like the audio and use a surfaceView to display the video. • There are examples in the API demo, plus several of the books. • The second method uses a VideoView to display. • The VideoView uses a MediaController to start and stop and provide functions to control the video play back. • With a MediaPlayer like the audio. prepare(), then start() • This is one I’ll cover in this lecture.

17. VideoView • VideoView is a View that has video playback capabilities and can be used directly in a layout. • We can then add controls (play, pause, forward, back, etc) with the MediaController.

18. ViewView example • Get the ViewView out of the layout vv= (VideoView) this.findViewById( R.id.VideoView); • Setup where the file to play is Uri videoUri = Uri.parse(Environment.getExternalStorageDirectory().getPath() + "/example.mp4"); vv.setVideoURI(videoUri); • play the video vv.start();

19. Adding media controllers. • Very simple vv = (VideoView) this.findViewById(R.id.VideoView); vv.setMediaController(new MediaController(this)); • Now media controls will show up on the screen.

20. Using native media player • Call for an Intent and send it. Uri data = Uri.parse(VideoFile); intent.setDataAndType(data, "video/mp4"); startActivity(intent); • Remember, your app is now in the background.

21. Blackberry and Android Displaying a picture

22. Displaying Pictures • For both android and blackberry • See code already covered to how display pictures. • bitmapfield,ImageView, etc…

23. Blackberry MMAPI Taking a picture

24. Taking a picture. • Taking a picture requires that you get a player and then VideoControl, finally create what amounts a view finder. • Then you can take the picture, which comes back as bytes (but in the format you spec’d, such as jpeg) • After that you do what you want with it.

25. Example • First get the player part. try { Player player= Manager.createPlayer( "capture://video?encoding=jpeg" ); player.realize(); player.start(); • Now get the Video Control VideoControlvideoControl= (VideoControl)player.getControl( "javax.microedition.media.control.VideoControl"); • Create a “View Finder” Field CameraView= (Field)videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); add(CameraView); //display the viewfinder, but you don’t have too. } catch (Exception e) { System.out.println("Error"+e.getMessage()); e.printStackTrace(); }

26. Capture:// • My code used the default, but you can specify • "encoding=jpeg&width=1600&height=1200&quality=superfine" • "encoding=jpeg&width=1600&height=1200&quality=fine" • "encoding=jpeg&width=1600&height=1200&quality=normal" • "encoding=jpeg&width=1024&height=768&quality=superfine" • "encoding=jpeg&width=1024&height=768&quality=fine" • "encoding=jpeg&width=1024&height=768&quality=normal" • "encoding=jpeg&width=640&height=480&quality=superfine" • "encoding=jpeg&width=640&height=480&quality=fine" • "encoding=jpeg&width=640&height=480&quality=normal" • The supported imageType(s) can also be queried by invoking • System.getProperty("video.snapshot.encodings")

27. “View Finder” • This line requires a bit of explaining • Field CameraView = (Field) videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); • InitDisplayMode has two options: • USE_GUI_PRIMITIVE • In this mode, arg can be the fully qualified class name of primitive GUI components, for example javax.microedition.lcdui.Item or net.rim.device.api.ui.Field. Depending on the value of arg, initDisplay returns an Item or a Field. • USE_DIRECT_VIDEO • When this mode is used, arg must be the fully qualified class name of Canvas, for example javax.microedition.lcdui.Canvasor a subclass of it. • In this case, initDisplay will return an instance of Canvas and the video will be directly rendered on this Canvas. • Then we cast it as field to display it as specified in the arguments.

28. Get the picture • Finally we can now take the picture. • Use a method called Snapshot • byte picData[] = videoControl.getSnapshot(null); • Null says take the default (or whatever we already specified) and use the default camera settings. • You could again, specify the image quality in the getSnapshotparameters. • To get the image into a bitmap object • image = Bitmap.createBitmapFromBytes(picData, 0, picData.length, 1);

29. Android Taking a picture

30. What to use • Android packages • import android.hardware.CameraDevice; • import android.hardware.CameraDevice.CaptureParams; • Permissions and features • <uses-permission android:name="android.permission.CAMERA" /> • This too, if you change the how the camera is functioning. • <uses-feature android:name="android.hardware.camera" /> • <uses-feature android:name="android.hardware.camera.autofocus" />

31. Taking a picture • In brief • CameraDevicecameraDevice = CameraDevice.open() • To preview you need a surfaceHolder then use setPreviewDisplay(surfaceHolder) and cameraDevice.startPreview() • Finally use the takePicture(…) to get the picture • release() and close() the CameraDevice to release it.

32. “View Finder” • The view finder is implemented via a SurfaceHolder and SurfaceView • In the layout, the a surfaceView is used. • Example: public class PicCapture extends Activity implements OnClickListener, SurfaceHolder.Callback, Camera.PictureCallback{ … cameraView= (SurfaceView) this.findViewById(R.id.CameraView); surfaceHolder= cameraView.getHolder(); surfaceHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); surfaceHolder.addCallback(this); • To finally take the picture we need all this too. cameraView.setFocusable(true); cameraView.setFocusableInTouchMode(true); cameraView.setClickable(true); cameraView.setOnClickListener(this);

33. “View Finder” (2) • The code to implement the surfaceHolder can be very simple. • When created open the camera and set the display surfaceCreated() { camera = Camera.open(); try { camera.setPreviewDisplay(holder); catch (IOException exception) { camera.release(); } } • Once it’s ready, start the preview surfaceChanged() { camera.startPreview(); }

34. “View Finder” (3) • When we are done surfaceDestroyed() { camera.stopPreview(); camera.release(); }

35. Get the Picture • Using the Camera.PictureCallBack we implement, we can get the data for the picture and decide what to do it with it. • In its simplest form, which doesn’t nothing with the picture public void onPictureTaken(byte[] data, Camera camera) { • byte[] data is the picture that was taken • this just restarts the preview. camera.startPreview(); }

36. Get the Picture (2) • To take the picture we use the • camera.takePicture method in the onClick method for the SurfaceView public void onClick(View v) { camera.takePicture(null, null, null, this); } • takePicture (Camera.ShutterCallback shutter, Camera.PictureCallback raw, Camera.PictureCallbackpostview , Camera.PictureCallback jpeg) • We only need the last to get the picture and it show on the previous slide. • shutter • the callback for image capture moment, or null • raw • the callback for raw (uncompressed) image data, or null • postview • the callback with postview image data, may be null • jpeg • the callback for JPEG image data, or null

37. Android Example Code • A note for the example code • The example code will put any pictures taken into the “Camera roll” and can be seen with the gallery app. • Remember it uses touch to take the picture.

38. Android android.media Recording VIDEO

39. First… • Most examples and code on the web and from the android books, DO NOT work. • lots of subtle errors • debugging is made more difficult, because • the camera throws errors • CameraInput Recording is not ready … frame dropped. • the AudioFlinger shows constant buffer overflow warnings. • And this is when the app is working correctly.

40. Uses a surfaceView • Like a taking a picture, we need a view finder which uses a surfaceView. • Call for the MediaRecorder • and setup the encoding. • both audio and video.

41. Example SurfaceView • in onCreate recorder = new MediaRecorder(); //setup recorder settings Next Slide SurfaceViewcameraView = (SurfaceView) findViewById(R.id.CameraView); holder = cameraView.getHolder(); holder.addCallback(this); holder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS); • implemented methods. public void surfaceCreated(SurfaceHolder holder) { recorder.setPreviewDisplay(holder.getSurface()); recorder.prepare(); }

42. Example SurfaceView (2) public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) { } public void surfaceDestroyed(SurfaceHolder holder) { if (recording) { recorder.stop(); recording = false; } recorder.release(); finish(); }

43. Recorder Settings • We need to set the sources for audio and video recorder.setAudioSource( MediaRecorder.AudioSource.MIC); recorder.setVideoSource( MediaRecorder.VideoSource.DEFAULT); • MediaRecorder.VideoSource.CAMERA should work as well. • Now we need to setup encoders. • In android 2.2 we can use profiles instead of setting everything manually. • CamcorderProfile.QUALITY_HIGH or QUALITY_LOW CamcorderProfilecpHigh = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH); recorder.setProfile(cpHigh); • And set the output location recorder.setOutputFile("/sdcard/videocapture_example.mp4");

44. Recorder Settings (2) • Manual settings could look like this: recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4); recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); recorder.setVideoEncoder(MediaRecorder.VideoEncoder.MPEG_4_SP); • QUALITY_HIGH settings are MP4 file • QUALITY_LOW settings are 3gp file • The manual list is very long, see the android doc’s or Apress - Pro Android Media Developing Graphics, Music, Video, and Rich Media Apps for Smartphones and Tablets, Chapter 11 for a full list of settings.

45. Recording • To record recorder.start(); • To stop recorder.stop(); • The file should be there at this point. • Remember when you are done recorder.release();

46. Example code • The example code • need an Sdcard to test code and the file will be located /sdcard/videocapture_example.mp4 • The code uses an extended surfaceView call captureSurface, instead of just a surfaceView • The code is all there, but rearranged from the slides. • Honesty, the code just didn’t work without an extended SurfaceView • The code starts up with the viewfinder, touch the screen to start recording, again to stop recording. It will then launch the native media player to replay the video.

47. Blackberry MMAPI Recording VIDEO

48. Video Recording. • Use all the same things from pic capturing • need a “view finder” • We the capture line • And RecordControl • use startRecord() to start it (needs be a thread) • use stopRecord() to pause recording • use commit() to finish it. • Player • choose to setRecordLocation or stream.

49. Example code • To get the “view finder” going player = Manager.createPlayer( "capture://video?encoding=video/3gpp&mode=standard" ); player.realize(); player.start(); videoControl= (VideoControl)player.getControl( "VideoControl"); recorder = (RecordControl)player.getControl("RecordControl"); CameraView= (Field)videoControl.initDisplayMode( VideoControl.USE_GUI_PRIMITIVE, "net.rim.device.api.ui.Field"); add(CameraView);

50. Capture line • capture://video?encoding=video/3gpp&mode=standard • standard video based on specifications of the phone • capture://video?encoding=video/3gpp&mode=mms • creates a video based on MMS standards • intended for sending the video via mms • You can ask the phone which encoding it can use with: • String encodingsString = System.getProperty( "video.encodings" ); • And parse the string.