Cosc 5/4730

1. Cosc 5/4730 Mobile Media API (MMAPI) v1.1 JSR 135 and Android media Part 1: Audio media

2. Blackberry MMAPI Playing Audio

3. Mobile Media API (MMAPI) background • The MMAPI is built on a high-level abstraction of all the multimedia devices that are possible in a resource-limited device. • This abstraction is manifest in three classes that form the bulk of operations that you do with this API. These classes are the Player and Control interfaces, and the Manager class. • Another class, the DataSource abstract class, is used to locate resources, but unless you define a new way of reading data you will probably never need to use it directly.

4. Background (2) • you use the Manager class to create Player instances for different media by specifying DataSource instances. • The Player instances thus created are configurable by using Control instances. • For example, almost all Player instances would theoretically support a VolumeControl to control the volume of the Player

5. Manager • This class is used for creating players and it provides three methods to indicate the source of media • These methods are all static • createPlayer(DataSource source) • createPlayer(InputStream stream, String type) • createPlayer(String locator) • where locator is a URI • As note, with Eclipse, you need to import manager implicitly or it gets confused…

6. CreatePlayer • The JavaDocs are very lengthy for createPlayer syntax, but here are some examples:

7. Player State Model • Players are created in the UNREALIZED state • and are typically realized • then prefetched, and then started. • When they reach the end of their media file, or stop() is called, they return to the PREFETCHED state.

8. Supported A/V • To get a list of supported protocols and Content you can use • String[] Manager.getSupportedProtocols( String contentType) • String[] Manager.getSupportedContentTypes( String protocol) • Example: (storm2 9550) • String capture[] = Manager.getSupportedContentTypes("capture"); • returns video/mpeg and audio/x-wav • String wav[] = Manager.getSupportedProtocols("audio/x-wav"); • returns file, http, https, capture • String file[] = Manager.getSupportedContentTypes("file"); • returns audio/midi audio/mpeg audio/x-wav audio/amr audio/x-gsm audio/mp4 audio/aac audio/x-ms-wma video/mp4 video/3gpp video/3gpp2 video/x-msvideo video/quicktime video/mpeg video/x-ms- sf video/x-ms-wm video/x-ms-wmv

9. Playing audio • Playing Tones import javax.microedition.media.*; import javax.microedition.media.Manager; import javax.microedition.media.control.*; … Player player = Manager.createPlayer(Manager.TONE_DEVICE_LOCATOR); player.realize(); //Must realize() before getting tonecontrol ToneControltc = (ToneControl) (player.getControl("ToneControl")); tc.setSequence(new byte[]{ToneControl.VERSION, 1, ToneControl.C4, 8, ToneControl.C4 + 2, 8}); // D4 player.start(); //Now the two tones will play, NON-BLOCKING! //Note no GUI interface is needed to play audio

10. Playing Tones • See the java docs ToneControl (in the control package) for more information on tones • They can be stored in a file • The ToneControl interface specifies a tone sequence format to be used for programming a tone sequence. In addition, this tone sequence format can also be used as a file format for specifying tone sequences. • A file with the ".jts" extension must be used to store tone sequences in this format. "audio/x-tone-seq" designates the MIME type for this format.

11. Playing Tones (2) InputStream in = getClass().getResourceAsStream("/file.jts"); Player player = Manager.createPlayer(in, "audio/x-tone-seq"); player.realize(); player.prefetch(); player.start(); // non-blocking.

12. Playing streaming media Player p = Manager.createPlayer( "http://www.cs.uwyo.edu/~seker/courses/4730/example/MEEPMEEP.WAV"); • NOTE, Takes the network parameters as well, so ;deviceside=true;interface=wifi if you are running via wifi, The above requires the MDS-CS to on for the simulator. p.prefetch(); //this calls realize(), so we don’t have too. p.setLoopCount(2); //play it twice p.start();

13. Listener • Since playing audio (and media) is controlled via a non-blocking method • The listener allows you program to know when something "has happened" • such as the audio has finished playing. • implement PlayerListener • playerUpdate(Player player, String event, Object eventData) • event is the one that tells you want is going on.

14. playerUpdate Events • BUFFERING_STARTED: Posted when the Player enters into a buffering mode. • BUFFERING_STOPPED: Posted when the Player leaves the buffering mode. • CLOSED: Posted when a Player is closed. • DEVICE_AVAILABLE: Posted when the system or another higher priority application has released an exclusive device which is now available to the Player. • DEVICE_UNAVAILABLE: Posted when the system or another higher priority application has temporarily taken control of an exclusive device which was previously available to the Player. • DURATION_UPDATED: Posted when the duration of a Player is updated. • END_OF_MEDIA: Posted when a Player has reached the end of the media. • ERROR: Posted when an error had occurred. • RECORD_ERROR: Posted when an error occurs during the recording. • RECORD_STARTED: Posted when recording is started. • RECORD_STOPPED: Posted when recording is stopped. • SIZE_CHANGED: Posted when the size of the video is changed either because the source video size or the display size is changed. • STARTED: Posted when a Player is started. • STOPPED: Posted when a Player stops in response to the stop method call. • STOPPED_AT_TIME: Posted when a Player is stopped as responding to the setStopTime call using the StopTimeControl. • VOLUME_CHANGED: Posted when the volume of an audio device is changed.

15. PlayerListener Example public void playerUpdate(Player player, String event, Object eventData) { if (event == PlayerListener.END_OF_MEDIA) { player.deallocate(); player.close(); } } • Where player is created… p.addPlayerListener(this);

16. Android android.media Playing Audio

17. Android • android.media is similar in nature to MMAPI • Get a MediaPlayer • use the MediaPlayer class, with the create() method to get a "player". • States: prepare()  start() • You can pause(), then start() without calling prepare() • Also seekTo() without needed prepare() again. • with stop() and reset(), then you must prepare() again • See next slide for states and where to go… • supported media formats http://developer.android.com/intl/zh-CN/guide/appendix/media-formats.html

18. mediaplayer states

19. Example • When it is stored as a Resource (res/raw directory) MediaPlayer mp = new MediaPlayer.create(getBaseContext(), R.raw.laser); mp.start(); //create calls prepare • For media store somewhere else MediaPlayer mp = new MediaPlayer(); mp.setDataSource("http://www.cs.uwyo.edu/~seker/courses/4730/example/MEEPMEEP.WAV"); • OR on filesystem, “/sdcard/file.mp3” mp.prepare(); mp.start();

20. As a note. • You can also play the audio from video files with the method as well. • There just won’t be any picture, just the audio.

21. Audio Capture • This can not be tested with the android or blackberry simulators • Android simulators may through errors, saying device doesn’t exist. • Blackberry seems to just ignore the call to record. • So at least you can test the interface and make sure everything else is working. • You have to test all the code on the phone. • All code examples on the website were tested and worked on the phones. • From eclipse, plugin the phone with USB. Click Debug AS • Blackberry: on device • Android: eclipse will ask you if you want to use the phone or emulator.

22. Blackberry MMAPI Capturing Audio

23. Recording Audio • You can use the following supported locator stringsin the createPlayer( str) • AMR: capture://audio OR • capture://audio?encoding=amr • capture://audio?encoding=audio/amr • PCM: capture://audio?encoding=pcm OR • capture://audio?encoding=audio/basic • GSM: capture://audio?encoding=gsm OR • capture://audio?encoding=audio/x-gsm • QCP: capture://audio?encoding=audio/qcelp

24. Audio Capture • To record via the mic Player player = Manager.createPlayer("capture://audio"); player.realize(); • Get the RecordControl and configure the record stream RecordControlrc = (RecordControl) player.getControl("RecordControl"); • Setup a ByteArrayOutput to record the audio stream ByteArrayOutputStream output = new ByteArrayOutputStream(); rc.setRecordStream(output); rc.startRecord(); • Now start it recording, and this is blocking, so do all this on a thread. player.start();

25. Audio Capture (2) • Stop recording rc.commit(); • Get the data byte data[]= output.toByteArray(); • Now close down everything player.close(); output.close(); • You can now save the data in a file, play it back, etc… • To play it back ByteArrayInputStreamrecordedIS = new ByteArrayInputStream(data); Player player2 = Manager.createPlayer(recordedIS, "audio/amr"); player2.prefetch(); player2.start();

26. Blackberry • Blackberry adds • net.rim.device.api.media • which provides interfaces to bluetooth devices and mediakeylistener. • MediaKeyListenermaps key events to media actions identical to those implemented by the built-in media player • So volume up/down, etc… • These do NOT require a the app to be signed to use the APIs.

27. Android android.media Capturing Audio

28. Recording Audio • Simpler then Blackberry, but also less options. • Get a new instance of the MediaRecorder() • Configure it for recording the mic, set the audio type • And configure the output FILE. (no outputStream) • Like playing, you prepare() and start(). • No thread needed. • Stop() when done recording and then release(). • The audio is now located in the file.

29. Recording Audio (2) • You need to set the following permissions in the AndroidManifest.xml file • android.permission.WRITE_EXTERNAL_STORAGE • android.permission.RECORD_AUDIO

30. Example Code private void startRecording() { • get MediaRecorder mRecorder= new MediaRecorder(); • configure it mRecorder.setAudioSource(MediaRecorder.AudioSource.MIC); mRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP); mRecorder.setOutputFile(mFileName); mRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB); • Prepare to record try { mRecorder.prepare(); } catch (IOException e) {} • start recording mRecorder.start(); } private void stopRecording() { mRecorder.stop(); mRecorder.release(); mRecorder = null; }

31. Audio settings • mRecorder.setAudioSource(); • MediaRecorder.AudioSource.MIC • MediaRecorder.AudioSource.CAMCORDER • Audio source with same orientation as camera, otherwise mic. • MediaRecorder.AudioSource.VOICE_CALL • Voice call uplink + downlink audio source • http://developer.android.com/reference/android/media/MediaRecorder.AudioSource.html • mRecorder.setOutputFormat(); • MediaRecorder.OutputFormat.THREE_GPP • Recommended setting by Andriod. • Also, AMR_NB, AMR_WB, RAW_AMR • MediaRecorder.OutputFormat.MPEG_4 • Using an MPEG-4 container format may confuse some desktop players. • http://developer.android.com/reference/android/media/MediaRecorder.html#setOutputFormat%28int%29 • mRecorder.setAudioEncoder(); • MediaRecorder.AudioEncoder.AAC • AAC audio codec • MediaRecorder.AudioEncoder.AMR_NB • AMR (Narrowband) audio codec • MediaRecorder.AudioEncoder.AMR_WB • AMR (Wideband) audio codec

32. Recording incoming phone calls • As of the current time (2010) • It doesn't appear to be possible via JavaME/Blackberry/Android to record the audio "from the other end" of the conversation. • Some phones allow you record the mic during a phone call (Blackbery and Android), but not the "headset speaker“ • But Android has a setting for recording, so it may work (as of 2011) • Many JavaMe phones, simply don't run any other programs during a phone call and so you can not record any part of the phone call.

33. References • JavaME/Blackberry • http://java.sun.com/javame/reference/apis/jsr135/ • http://docs.blackberry.com/en/developers/deliverables/11942/CS_record_video_without_using_the_camera_app_734825_11.jsp • http://www.forum.nokia.com/document/Java_Developers_Library_v2/?content=GUID-9CAAF2D4-D395-4021-BBE6-602DFD275B03.html • http://www.forum.nokia.com/document/Java_Developers_Library_v2/GUID-8F9BE454-FC5E-4B87-B805-765DEFA66C99//overview-summary.html • http://www.forum.nokia.com/document/Java_Developers_Library_v2/GUID-8F9BE454-FC5E-4B87-B805-765DEFA66C99//javax/microedition/media/control/ToneControl.html#tone_sequence_format • http://developers.sun.com/mobility/midp/articles/mmapioverview/ • http://www.mobiquil.com/jme_mmapi.html • Android • http://developer.android.com/intl/zh-CN/guide/topics/media/index.html • http://developer.android.com/guide/topics/media/index.html

34. Q A &