The Java Media Framework: javaxdia

1. The Java Media Framework: javax.media • Access to devices & media files • Resources • Sun JMF web site • http://java.sun.com/products/java-media/jmf/index.jsp • http://java.sun.com/products/java-media/jmf/2.1.1/apidocs/ • Sample chapter from book, “Essential JMF - Java Media Framework” Gordon & Talley, 1998 • Terrazas et al. (2002). Java Media API’s. SAMS Pub.

2. Topics • General contents of JMF • Digital video primer • Classes • java.awt.image.BufferedImage • Abstract Window Toolkit class • java.awt.image.ColorModel • javax.media.format.RGBFormat • Pixel representation • Classes • javax.media.Format; • javax.media.MediaLocator; • javax.media.protocol.DataSource; • javax.media.CaptureDeviceManager;

3. Digital Video Primer • Cameras return “frames” of visual information • Each frame is a 2D matrix • Components of these arrays are pixels • Usually integers • Dimensions: height by width • E.g., 160 x 120 pixels • Frames are returned from the camera at a certain rate • I.e., the frame rate • E.g., 15 frames per second • Frames are often represented in programs as one-dimensional arrays int[] frame; // in Java

4. Frames as 1D Arrays int [] frame = null; // code to get frame from camera… int x = 10, y = 20; // get some pixel // 0 <= x <= width-1; 0 <= y <= height-1 // (0, 0)-- upper left of image int pixel = frame[y * width + x]; // assumes a row oriented representation

5. Class java.awt.image.BufferedImage • An Image comprised of a data buffer for image, plus color model information • Represents color or grayscale images in various formats • E.g., TYPE_BYTE_GRAY, TYPE_USHORT_GRAY, TYPE_USHORT_565_RGB, TYPE_INT_RGB • Methods include public ColorModel getColorModel() • ColorModel (java.awt.image.ColorModel) describes the format of the image public BufferedImage getSubimage(int x, int y, int w, int h) public int[] getRGB(parameter list…)

6. Classjava.awt.image.ColorModel • Abstract class used to construct representations of colors (color models) for images • Methods include • public abstract int getRed(int pixel) • public abstract int getGreen(int pixel) • public abstract int getBlue(int pixel)

7. Classjavax.media.format.RGBFormat • The RGBFormat class of the javax.media.format package • Subclass of VideoFormat (javax.media.format.VideoFormat) • “32-bit packed RGB stored as 32-bit integers would have the following masks: redMask = 0x00FF0000, greenMask = 0x0000FF00, blueMask = 0x000000FF.” (From JMF API)

8. Pixel Representation • Pixels as returned from the BufferedImage class getRGB method • Represented using lower 24 bits of int’s • High order byte: Red • Middle byte: Green • Low-order byte: Blue • This format because the Java programs we are using set up the image formats this way • RGBFormat (javax.media.format.RGBFormat) image formats were obtained from the camera

9. Example • 1) Given a pixel in RGBFormat as described, use bit manipulation operations to extract out the R, G, and B components • 2) Use bit manipulation operations to create a new RGBFormat pixel given 8 bit R, G, and B components

10. Outline of Using JMF to Access USB Camera Devices • Determine available camera devices • CaptureDeviceManager.getDeviceList • Obtain a DataSource • Use Manager to create this DataSource, based on a MediaLocator • Obtain a Processor • From Manager based on the DataSource • Obtain a PushBufferDataSource • From the Processor

11. Classjavax.media.CaptureDeviceManager • Methods public static CaptureDeviceInfo getDevice(java.lang.String deviceName) • Get a device by name public static java.util.Vector getDeviceList(Format format) • Returns a Vector containing a list of CaptureDeviceInfo descriptions of devices available on the computer of the format given • Formats returned may be more than that specified • E.g., if a given device supports YUV and RGB formats, then all formats (including both YUV and RGB) will be returned in the result of the getDeviceList call • If Format is null, returns the list of all available devices on the system

12. Classjavax.media.CaptureDeviceInfo • Methods include public Format[] getFormats() • Returns an array of Format objects • Specific Format’s available for the video device public MediaLocator getLocator()

13. Classjavax.media.Format • Format class of javax.media package • Constructor Format(java.lang.String encoding) • Device/media specific description of a media format • E.g., for a video (visual) device • "RGB, 160x120, Length=38400, 16-bit, Masks=31744:992:31, PixelStride=2, LineStride=320, Flipped” • E.g., for an a audio device • "PCM, 44.1 KHz, Stereo, Signed"

14. Classjavax.media.MediaLocator • Constructor • MediaLocator(java.lang.String locatorString) • Locator strings are similar to URLs • Example locatorString • “vfw://0” • This is for my USB web camera • MediaLocator’s can describe files e.g., MediaLocator m = new MediaLocator(“file://media/example.mov”);

15. Classjavax.media.Manager - 1 • Access point for obtaining system dependent resources • Player’s, DataSource’s, Processor’s, DataSink’s, TimeBase • DataSource • Object used to deliver time-based multimedia data that is specific to a delivery protocol. • Examples of protocols: http, ftp, file • Once you have a “usable” (formatted) DataSource, you can display the media (via a Player), or manipulate the information (via a Processor)

16. Class javax.media.Manager - 2 • Manager methods include public static DataSource createDataSource(MediaLocator sourceLocator) Returns a data source for the protocol specified by the MediaLocator. When you call createDataSource on the media locator of a video capture device (obtained from the CaptureDeviceManager), the returned DataSource will implement the CaptureDevice interface public static DataSource createCloneableDataSource(DataSource source) - create a DataSource that can be cloned; this enables DataSource to be processed by different tasks • public static Processor createProcessor(DataSource source) - Processors are used in controlling the processing of media

17. Classjavax.media.protocol.DataSource • An abstraction for media protocol-handlers. DataSource manages the life-cycle of the media source by providing a simple connection protocol. • Methods include • DataSource(MediaLocator source) • connect • start, stop • Start and stop data transfer

18. Interface CaptureDevice • A capture device is a DataSource of type PullDataSource, PullBufferDataSource, PushDataSource or PushBufferDataSource. It also implements the CaptureDevice interface… A CaptureDevice DataSource contains an array of SourceStream's. These SourceStreams provide the interface for the captured data streams to be read. (From JMF API) • Methods include • public FormatControl[] getFormatControls()

19. InterfaceFormatControl • Objects implementing the FormatControl interface can be used to set the Format of the CaptureDevice to the Format desired • Methods • public Format setFormat(Format format) • Returns null if the format is not supported. Otherwise, it (typically) returns the format that's actually set.

20. Interface javax.media.Processor • Processes and controls time-based media data. Extends the Player interface. Unlike a Player, which processes data as a "black box" and only renders data to preset destinations, a Processor supports a programmatic interface enabling control over media data processing and access to output data streams. • Processing performed by a Processor is split into three stages: • Demultiplexing - into separate tracks of data • Data transcoding - of each track into other formats • Multiplexing - to form an interleaved stream • Data transcoding and multiplexing processes are programmable.

21. Interface javax.media.Processor • Processor is a Player is a Controller • Methods include DataSource getDataOutput() get the output DataSource from the Processor public void addControllerListener(ControllerListener listener) • From Controller interface • Events get posted to the listener, I.e., the public void controllerUpdate(ControllerEvent event) method gets called for that listener public void realize() • Constructs media dependent portions of Controller. public void prefetch() public void start() public int getState() • States: Unrealized, Configuring, Configured, Realizing, Realized, Prefetching, Prefetched, and Started.

22. Controller Life Cycle • Controller’s have five resource-allocation states: Unrealized, Realizing, Realized, Prefetching, and Prefetched. • These states provide programmatic control over potentially time-consuming operations. For example, when a Controller is first constructed, it's in the Unrealized state. While Realizing, the Controller performs the communication necessary to locate all of the resources it needs to function (such as communicating with a server, other controllers, or a file system). The realize method allows an application to initiate this potentially time-consuming process (Realizing) at an appropriate time.

23. Classjavax.media.protocol.PushBufferDataSource • A data source that manages data in the form of push streams. The streams from this DataSource are PushBufferStream’s and contain Buffer objects • We we use getDataOutput to get a DataSource from our Processor, and this DataSource is actually a PushBufferDataSource • Methods public abstract PushBufferStream[] getStreams() • Return streams that this source manages. The collection of streams is entirely content dependent. The ContentDescriptor of this DataSource provides the only indication of what streams can be available on this connection. • For our USB camera capture device, there is only one PushBufferStream

24. Interface javax.media.protocol.PushBufferStream • A read interface that pushes data in the form of Buffer objects. Allows a source stream to transfer data in the form of a media object. The media object transferred is the Buffer object as defined in javax.media.Buffer. The user of the stream will allocate an empty Buffer object and pass this to the source stream in the read() method. The source stream allocates the Buffer object's data and header, sets them on the Buffer and sends them to the user.

25. Interface javax.media.protocol.PushBufferStream • Methods include public Format getFormat() public void setTransferHandler(BufferTransferHandler transferHandler) • Register an object to service data transfers to this stream. public void read(Buffer buffer) • Read from the stream without blocking. • (Not documented regarding what happens if you read and there is no data available-- Is the buffer empty or is an IOException raised).

26. Interface javax.media.protocol.BufferTransferHandler • Implements a callback from a PushBufferStream. • A PushBufferStream needs to notify the data handler when data is available to be pushed. • Methods include public void transferData(PushBufferStream stream) • Notification from the PushBufferStream to the handler that data is available to be read from stream. The data can be read by this handler in the same thread or can be read later.

27. Classjavax.media.Buffer • A media-data container that carries media data from one processing stage to the next inside of a Player or Processor. Buffer objects are also used to carry data between a buffer stream and its handler. • Maintains information including the time stamp, length, and Format of the data it carries, as well as any header information that might be required to process the media data.

28. Classjavax.media.util.BufferToImage • A utility class to convert a video Buffer object to an AWT Image object • you can then render the Image to the user display with AWT methods. • Methods include public BufferToImage(VideoFormat format) public java.awt.Image createImage(Buffer buffer) Converts the input Buffer to a standard AWT image and returns the image.

29. (From Above) Class java.awt.image.BufferedImage • An Image comprised of a data buffer for image, plus color model information • A subclass of java.awt.Image • Represents color or grayscale images in various formats • E.g., TYPE_BYTE_GRAY, TYPE_USHORT_GRAY, TYPE_USHORT_565_RGB, TYPE_INT_RGB • Methods include public ColorModel getColorModel() • ColorModel (java.awt.image.ColorModel) describes the format of the image public BufferedImage getSubimage(int x, int y, int w, int h) public int[] getRGB(parameter list…)

30. Capturing the Webcam Images PushBufferStream PushBuffer; // created above Buffer ImageBuffer = new Buffer(); BufferToImage ImageConverter = new BufferToImage((VideoFormat) (PushBuffer.getFormat())); // assume a new frame is ready PushBuffer.read(ImageBuffer); BufferedImage InputImage = ((BufferedImage) (ImageConverter.createImage(ImageBuffer))); int[] CurrentFrame = InputImage.getRGB(0, 0, VideoWidth, VideoHeight, null, 0, VideoWidth); From ImageProcessingThread.java (Lab1)