Processors

1. Processors • Can also be used to present media data • Specialized type of Player that provides control over processing performed on the input media stream

2. Processing

3. Processor Stages

4. Additional Processor States • Two additional stand-by states: • Configuring • Configured – can use TrackControls

5. Processing Controls • For a given track, can control processing operations performed by the Processor by using the TrackControl for that track. TrackControl[] = processor.getTrackControls() • Can explicitly select: • Effect, Codec and Renderer plug-ins to use TrackControl[1].setCodecChain( array_of_codecs )

6. Configuring the Processor • Consider using a processor to transcode an (audio+video) QuickTime movie – changing mpeg video track to h.263… p = Manager.createProcessor(dataSource) p.configure() p.setContentDescriptor.QUICKTIME tcs[] = p.getTrackControls() • Returns an array, e.g. 2 track controls…

7. Configuring the Processor Format f0 = new VideoFormat(VideoFormat.h263, new Dimension(width, height), Format.NOT_SPECIFIED, Format.byteArray, (float)frameRate); l Format f1 = new AudioFormat(AudioFormat.mpeg, 8000 8, 2); tcs[0].setFormat(f0) tcs[1].setFormat(f1) p.realize() p.start()

8. Processor Summary • A Processor does not have to output data as a DataSource, such a processor (i.e. one that renders the data) is effectively a configurable player.

9. Data Storage and Transmission • DataSink • Used to read data from a DataSource and render the media to an output destination • Typical actions… • Write data to a file, across a network etc

10. Using the DataSink MediaLocator dest = new MediaLocator(file://newfile.wav); dsink = Manager.createDataSink(ds, dest); dsink.addDataSinkListener(this); dsink.open(); p.start(); dsink.start(); • Wait for EndOfStream event • Close DataSink and remove Listener • dsink.close()

11. Example • Applet Movie Player

12. Applet Movie Player • Simple Java Applet that demonstrates how to create a simple media player with a media event listener. It will play the media clip right away and continuously loop. <!-- Sample HTML <applet code=TVApplet width=587 height=510> <param name=file value=“playme.mov"> </applet> -->

13. Basic Steps • Initialisation… • Retrieve applet’s FILE parameter • Use this to locate media file and build URL • Create Player using the Manager object • Register applet as a ControllerListener

14. Steps 1 & 2: Resolving a URLfor the media stream // The applet tag should contain the path to the // source media file, relative to the html page. if ((mediaFile = getParameter("FILE")) == null) Fatal("Invalid media file parameter"); try { // Create a url from the file name and the url // to the document containing this applet. if ((url = new URL(codeBase, mediaFile)) == null) Fatal("Can't build URL for " + mediaFile);

15. Step 3: Using Manager toCreate a Player // Create an instance of a player for this media try { player = Manager.createPlayer(url); } catch (NoPlayerException e) { System.out.println(e); Fatal("Could not create player for " + url); }

16. Step 4: Register applet as aControllerListener // Add ourselves as listener for player's events player.addControllerListener(this); } catch (MalformedURLException e) { Fatal("Invalid media file URL!"); } catch (IOException e) { Fatal("IO exception creating player for " + url);

17. Controlling the Player… • Starting the Player public void start() { if (player != null) player.realize(); • Stopping the Player public void stop() { if (player != null) { player.stop(); player.deallocate(); } }

18. Responding to media events • Need to Implement ControllerListener • When the Player is realized • Posts a RealizeCompleteEvent • Get the Visual component if (( visualComponent = player.getVisualComponent())!= null) { cPanel.add(visualComponent); • Get the Control Panel component • When the media has reached the end… • Posts an EndOfMediaEvent • Rewind and start over player.setMediaTime(new Time(0));

19. Extensibility • Can extend JMF functionality in two ways: • Using Plug-ins • Effectively implementing custom processing components that can be interchanged with standard components used by a Processor • By direct implementation • i.e implementing directly the Controller, Player , Processor, DataSource, or DataSink interfaces • e.g. implementing a player to utilise a h/w MPEG decoder • e.g. integrating existing media engines

20. Interface Plugin • The base interface for JMF plug-ins. • A PlugIn is a media processing unit that accepts data in a particular format and processes or presents the data. • Registered through the PlugInManager. • Methods… • Open(), Close(), getName(), reset() • Sub-interfaces… • Codec, effect, demultiplexer , multiplexer, etc.

21. Codecs • One input and one output • Methods… • setInputFormat() • setOutputFormat() • getSupportedInputFormats() • getSupportedOutputFormats() • process() • input buffer, output buffer

22. Example 2 • Accessing individual frames

23. FrameAccess • Problem • How to access individual decoded video frames from a Processor while processing the media. This could be used for scanning the decoded data; computing statistics for each video frame, etc.

24. FrameAccess • Solution… • use a ‘pass-through’ plugin codec as a callback when individual frames are being processed. • Steps: • 1) Build the pass-through codec. • 2) Create a processor from the input file. This processor will be used as a player to playback the media. • 3) Get the TrackControls from the processor. • 4) Set your codec on the video track: TrackControl.setCodecChain(your_codec[])

25. Basic code // Get Video track as a track control TrackControlvideoTrack = null; for (int i = 0; i < tc.length; i++) { if (tc[i].getFormat() instanceof VideoFormat) { videoTrack = tc[i]; break; } } // Instantiate & set frame access codec to data flow path. Codec codec[] = { new PreAccessCodec(), new PostAccessCodec() };

26. PreAccessCodec public class PreAccessCodec implements Codec { void accessFrame(Buffer frame) { long t = (long)(frame.getTimeStamp()/10000000f); System.err.println("Pre: frame #: " + frame.getSequenceNumber () + ", time: " + ((float)t)/100f + ", len: " + frame.getLength()); } ………………… • Other methods, e.g. getSupportedInputFormats etc.

27. The Codec’s Process method public int process(Buffer in, Buffer out) { // This is the "Callback" to access individual frames. accessFrame(in); // Swap the data between the input & output. Object data = in.getData (); in.setData(out.getData ()); out.setData(data ); // Copy the input attributes to the output out.setFormat(in.getFormat()); out.setLength(in.getLength()); out.setOffset(in.getOffset());