Java Media Framework

1. Java Media Framework • The Java Media Framework (JMF) is an application programming interface (API) for incorporating time-based media into Java applications and applets • The JMF 1.0 API (the Java Media Player API) enabled programmers to develop Java programs that presented time-based media • JMF 2.0 API extends the framework to provide support for • capturing and storing media data, • controlling the type of processing that is performed during playback, • performing custom processing on media data streams • JMF 2.0 defines a plug-in API to allow developers to customize and extend JMF functionality

2. JMF Media Processing Model

3. Media Streams • Often contain multiple channels • Tracks • Example • MPEG-1 file usually 2 tracks • Audio track • Video track • Demultiplexing • Multiplexing

4. Example • Process an mpeg-1 a/v media stream • Transcode video track to H.263 • Transcode audio track to GSM • Steps • Demultiplex to obtain tracks • Decompress video track • Compress using H.263 • Decompress audio track • Compress using GSM • Multiplex two tracks • Save to file

5. JMF Design Goals • Enable input, processing and output of time-based media • Provides common cross platform API for accessing underlying media frameworks • Extensible • support additional content types and formats, optimize handling of supported format, create new presentation mechanisms

6. Supported Content Types • Supported types • Not always both decode and encode • Differences between platform independent and dependent versions • Audio • WAV, GSM, MIDI, etc • Image • JPEG, etc • Video • H.261, H.263, MPEG-1, Quicktime, AVI, etc

7. Recording, processing, and presenting time-based media

8. High-level Architecture

9. Some JMF Base Interfaces • Clock • Controller • Control

10. Time Model • The Clock interface • Defines basic timing and synchronization operations • Contains a Timebase • Based on the system clock • time-base time • Simply provides current time • Clock marks time for a particular media stream • media time • Current position within a media stream

11. Time Model

12. Clock • Playback rate • How fast the Clock is running in relation to its TimeBase • Examples: • rate of 1.0 represents normal running time • rate of 2.0 means presentation will run at twice the normal rate • Clock Transform • Media-time = media start-time+ Rate(time-base time – time-base start-time)

13. Example • Example: Have a 20 sec MPEG video stream • MediaStartTime= 10 secs, • TimeBaseTime= 3 secs • TimeBaseStartTime= 0 secs, TimeBaseTime–TimeBaseStartTime= 3 secs • Media-time= media start-time+ Rate(time-base time – time-base start-time) • So if Rate = 1.0, MediaTime= ?? • Alternatively, if rate = -2.0, MediaTime= ??

14. Achieving Synchronization • Example • Want to force a video renderer to sync to the timebase of an audio renderer • X = audio_renderer.getTimeBase() • Video_renderer.setTimeBase(X) • Both objects would now use the same source of time.

15. Controller Interface • Controller Interface • Defines basic state and control mechanism for an object that controls, presents or captures time-based media • Two types of Controller: Players and Processors (considered later…)

16. Controllers

17. Controller lifecycle

18. Controller Events • JMF objects can post a MediaEvent • Events posted by a Controller: • TransitionEvents • Posted when a controller changes state • Change notification events • e.g. RateChangeEvent • ControllerClosedEvents • Posted when Controller shuts down • Corresponding listener interface for each type of JMF object that can post MediaEvents

19. JMF Event Model

20. Controls • Mechanism for setting and querying attributes of an object • Certain objects expose Controls • e.g. often used by PlugIns to provide access to their Control objects • Examples • FrameRateControl • GainControl • Can associate listener for when volume changes

21. Key objects in JMF • Managers • DataSources • Players • Processors • DataSinks

22. General Managers • Intermediary objects • Enables new implementations of key interfaces • 4 main types • Manager • PlugInManager • PackageManager • CaptureDeviceManager

23. The Manager object • Manager object used for instantiating: • DataSources, • used to deliver time-based multimedia data • Players, • used to control and render multimedia data • Processors, • used to process data and output the processed data • DataSinks, • takes a DataSource as input and renders the output to a specified destination

24. The Manager Object

25. Data Model in JMF • Data Sources • Media players use DataSources to manage the transfer of media-content • DataSource encapsulates location of media and the protocol used to deliver the media • Typically Identified by a: • URL, MediaLocator

26. Capture • Capture devices represented as DataSources • e.g. microphone, video capture board, etc… • Devices can deliver multiple data streams • e.g. audio and video from a camera • e.g. multiple audio tracks from a recording session • You may then wish a single DataSource to contain multiple SourceStreams • Manager.createMergingDataSource(SourceStreams)

27. Push and Pull Data Sources • Data sources can be categorized according to how data transfer is initiated • Pull Data-Source Client initiates the data transfer • e.g. HTTP and FILE • Push Data-Source Server initiates the data transfer • e.g. broadcast and multicast media

28. Players • Processes an input stream and renders it at a precise time • Does not provide any control over the processing that it performs or how it renders the media data

29. Players • Player extends the Controller interface. • Has a lifecycle • Sends media events • Player as a MediaHandler • player = Manager.createPlayer(myDataSource); player = Manager.createPlayer(myMediaLocator); player = Manager.createPlayer(myUrl);

30. Players

31. UI Components • Players provides access to UI Components • Player (or Processor) can provide two UI components • Visual component • Control-panel component • Can retrieve these components using methods: • getVisualComponent() • getControlPanelComponent()

32. Player States Continued • Players post transitional events as they move between states • ControllerListener • Is the Player in an appropriate state? • Only certain methods make sense in certain states • e.g. calling getTimeBase method on an unrealized player gives an error

33. Processors • Can also be used to present media data • Specialized type of Player that provides control over processing performed on the input media stream

34. Processing

35. Processor Stages

36. Additional Processor States • Two additional stand-by states: • Configuring • Configured – can use TrackControls

37. Processing Controls • For a given track, can control processing operations performed by the Processor by using the TrackControl for that track. TrackControl[] = processor.getTrackControls() • Can explicitly select: • Effect, Codec and Renderer plug-ins to use TrackControl[1].setCodecChain( array_of_codecs )

38. Configuring the Processor • Consider using a processor to transcode an (audio+video) QuickTime movie – changing mpeg video track to h.263… p = Manager.createProcessor(dataSource) p.configure() p.setContentDescriptor.QUICKTIME tcs[] = p.getTrackControls() • Returns an array, e.g. 2 track controls…

39. Configuring the Processor Format f0 = new VideoFormat(VideoFormat.h263, new Dimension(width, height), Format.NOT_SPECIFIED, Format.byteArray, (float)frameRate); l Format f1 = new AudioFormat(AudioFormat.mpeg, 8000 8, 2); tcs[0].setFormat(f0) tcs[1].setFormat(f1) p.realize() p.start()

40. Processor Summary • A Processor does not have to output data as a DataSource, such a processor (i.e. one that renders the data) is effectively a configurable player.

41. Data Storage and Transmission • DataSink • Used to read data from a DataSource and render the media to an output destination • Typical actions… • Write data to a file, across a network etc

42. Using the DataSink MediaLocator dest = new MediaLocator(file://newfile.wav); dsink = Manager.createDataSink(ds, dest); dsink.addDataSinkListener(this); dsink.open(); p.start(); dsink.start(); • Wait for EndOfStream event • Close DataSink and remove Listener • dsink.close()

43. Example • Applet Movie Player

44. Applet Movie Player • Simple Java Applet that demonstrates how to create a simple media player with a media event listener. It will play the media clip right away and continuously loop. <!-- Sample HTML <applet code=TVApplet width=587 height=510> <param name=file value=“playme.mov"> </applet> -->

45. Basic Steps • Initialisation… • Retrieve applet’s FILE parameter • Use this to locate media file and build URL • Create Player using the Manager object • Register applet as a ControllerListener

46. Steps 1 & 2: Resolving a URLfor the media stream // The applet tag should contain the path to the // source media file, relative to the html page. if ((mediaFile = getParameter("FILE")) == null) Fatal("Invalid media file parameter"); try { // Create a url from the file name and the url // to the document containing this applet. if ((url = new URL(codeBase, mediaFile)) == null) Fatal("Can't build URL for " + mediaFile);

47. Step 3: Using Manager toCreate a Player // Create an instance of a player for this media try { player = Manager.createPlayer(url); } catch (NoPlayerException e) { System.out.println(e); Fatal("Could not create player for " + url); }

48. Step 4: Register applet as aControllerListener // Add ourselves as listener for player's events player.addControllerListener(this); } catch (MalformedURLException e) { Fatal("Invalid media file URL!"); } catch (IOException e) { Fatal("IO exception creating player for " + url);

49. Controlling the Player… • Starting the Player public void start() { if (player != null) player.realize(); • Stopping the Player public void stop() { if (player != null) { player.stop(); player.deallocate(); } }

50. Responding to media events • Need to Implement ControllerListener • When the Player is realized • Posts a RealizeCompleteEvent • Get the Visual component if (( visualComponent = player.getVisualComponent())!= null) { cPanel.add(visualComponent); • Get the Control Panel component • When the media has reached the end… • Posts an EndOfMediaEvent • Rewind and start over player.setMediaTime(new Time(0));