Real-time and Retrospective Analysis of Video Streams using MPEG-7

1. Real-time and Retrospective Analysis of Video Streams and Still Image Collections using MPEG-7 Ganesh Gopalan, College of Oceanic and Atmospheric Sciences, Oregon State University

2. Introduction • HD video streams have potential to improve understanding of deep sea eco-systems • However, volume and complexity associated with the HD streams and formats can be overwhelming • Our approach: Use industry standards to transform video into a data type vs. treating it as viewing material

3. MPEG-7 Overview • Multimedia content description interface • Consists of low-level descriptors and high-level description schemes • Low-level descriptors provide statistical information about the pixel values in content • Description Schemes are used to represent semantic information

4. Low Level Descriptors • Structures that describe content in terms of the distribution of edges, colors, textures, shapes and motion • Descriptors extracted using MPEG-7 Experimental Model (XM) software • The input is a still image or a frame from video • The output is an XML description of the statistical information

5. Examples of Low Level Descriptors • Edge Histogram • Homogeneous Texture • Color Layout • Color Structure • Motion Activity • Descriptors are rotation and scaling invariant

6. Descriptor Extraction and Search • Phase 1: descriptor XML for collection of frames/still images is generated and cached • Phase 2: difference between query image descriptor from those values cached in phase one is computed • The cache can be augmented with the descriptors from a new video or still image collection

7. Description Schemes • Description Schemes attempt to model the reality behind the content • Low level descriptors can be used to tag objects of interest; the tags are then used to construct a high level description • A search can then be performed against the higher level description schemes

8. High Definition Video Search Engine • Applied MPEG-7 to the development of an HD search engine • Extracted descriptors for approximately 10,000 frames from 2.5 hours of high definition content • Content provided by the University of Washington from “Visions 05 Cruise” • Also applied to search for eddies in satellite image collections; super-cells in radar images

9. Application Architecture • .NET Windows Forms front end with an embedded Windows Media Player • SQL Server back-end • Common Language Run-time Integration for development of stored procedures to manage MPEG-7 XML • Procedures can be written in .NET languages rather than SQL

11. Creating a CLR Stored Procedure CREATE FUNCTION FindUsingVisualDescriptor ( @uid int, @token uniqueidentifier, @queryImage varbinary(MAX), @descriptorName nvarchar(256) ) RETURNS nvarchar(MAX) AS EXTERNAL NAME MPEG7Document.StoredProcedures.FindUsingVisualDescriptor; GO

12. Creating an HTTP Endpoint CREATE ENDPOINT MPEG7 STATE = Started AS HTTP ( SITE = ‘XXX.XXX.XX.XXX', PATH = '/MPEG7Endpoint', AUTHENTICATION = (BASIC), PORTS = (SSL), SSL_PORT = 444 ) FOR SOAP (WEBMETHOD 'FindUsingVisualDescriptor' (NAME = 'looking.dbo.FindUsingVisualDescriptor', FORMAT = ALL_RESULTS), …)

13. User Interface • UI allows conversion of video into frames using ffmpeg • Descriptors of choice are then generated for all frames • Descriptors are persisted to the server

14. Retrospective Search • A query image initiates the search • The descriptor value for the given image is compared with those cached from the video frames or still images • The top 100 frames that are closest to the query image are returned

15. Retrospective Search Example

16. Real-time Event Detection • In this case, we have a set of known images that have objects of interest • Descriptors of frames from a real-time stream are compared on a continuous basis with those in the “event library” • When the difference in descriptor values is below a threshold, an event has been detected

17. Example of an Event

18. Reference Event

20. Use of Multi-Core Systems • The descriptor extraction process can be made faster by taking advantage of multiple processors or cores • The total number of frames can be divided up amongst the available processors • Threads extract the descriptors concurrently to generate chunks of XML • The threads then signal each other to combine the chunks into a single file with the descriptor XML

21. Challenges • Shadows and other lighting issues can create false positives • May be necessary to use multiple descriptors for classification • Processing high definition video at 30fps is computationally intensive • Scaling to a large number of images such as on the web presents a challenge

22. Conclusion • MPEG-7 supports a rich framework for content-based searches through its low level descriptors • Detected content can be tagged effectively using the high level description schemes that can be used to locate, search through and distribute content

23. Future Directions • Need to explore ways to speed up descriptor extraction using GPUs or hybrid GPGPUs. • Explore Cloud Services to implement video services – transcoding video on the fly for different devices, descriptor extraction using HPC clusters, streaming services • Explore the Surface Computer as a UI

24. Acknowledgements • We are thankful to Professor John Delaney from the University of Washington for providing the HD footage • We are also thankful to the NSF funded LOOKING team for supporting this effort