CS257 Modelling Multimedia Information LECTURE 2

1. CS257 Modelling Multimedia InformationLECTURE 2 Dr Lee Gillam

2. Learning Outcomes At the end of the module students should be able to: • 1.   Apply a standard metadata set to describe media items in an archive • 2.   Apply appropriate techniques for modelling text documents • 3. Compare and apply different kinds of metadata for image data • 4.   Distinguish and apply appropriate data models for temporal media (video/audio) • 5.   Describe video content formally, in terms of objects, events and temporal relationships between events • 6.    Design and implement a synchronised multimedia presentation including a variety of media types and temporal relationships • 7.       Explain the theory behind hypermedia systems and assess how this is put into practice in current hypermedia applications • 8. Explain, contrast and evaluate the modelling of multimedia information to support media access, especially retrieval and browsing, in a variety of applications - these include personal media collections, organisation-wide media archives and web-based search engines

3. Overview of LECTURE 2 • Part 1: Multimedia Data • How do we describe images? • Multimedia and the “information explosion” • What multimedia encodings are there? • Part 2: Metadata • Why do we need metadata standards? • Interoperability between people and systems • What are the issues and challenges in making and using metadata standards? • Describing multimedia at different levels of abstraction; adding metadata at different stages of the media production process • Manual creation (cost) / automation; formalisation / restrictiveness • Objectivity / subjectivity – metadata is context-sensitive, time-variant, culturally variant

4. LECTURE 2:LEARNING OUTCOMES • After the lecture and lab exercise, you should be able to: • Demonstrate an awareness of issues and challenges for creating metadata • Explain why we need metadata standards, and explain how different kinds of metadata interoperability are required in different applications • Give a reasoned argument for choosing one multimedia data coding format over another, for a given application

5. Part 1: Multimedia data Multimedia data generally produced by humans for humans. • How do we describe images? • What do people doing with images? • Differences between seeing and understanding • What can we express, and how? • How much information is there - multimedia and the “information explosion”?

6. Engaging? • Which languages do you speak? • English, French, German, Mandarin Chinese, Portuguese, Arabic, Spanish, Urdu, Gujurati • What country are you from? • England, Wales, Portugal, Saudi Arabia • What countries have you visited?

7. Set Exercise 1-1 Guildford art gallery has digitised its collection of paintings and wants to make them available for searching online. What metadata would you specify for: • The general public to search for paintings? • Artist name [characters – subfields: surname, given name, other names], Gallery found in [city, country], Date/time [day, month, year, hours, minutes, seconds] • Art experts to search for paintings? • Medium (e.g. canvas), Materials (oil, watercolours), Genre (renaissance, impressionists)

8. Set Exercise 1-1 • The general public to search for paintings? • Art experts to search for paintings?

9. Set Exercise 1-1 Mona Lisa • Madonna with baby? Painted Painted Leonardo da Vinci Born near Country Florence Italy

10. Set Exercise 1-1 Related work: The CLEF Cross Language Image Retrieval Track (ImageCLEF) 28,133 images and descriptions from the photographic collection provided by St Andrews University Library

11. Set Exercise 1-1 Related work: The CLEF Cross Language Image Retrieval Track (ImageCLEF) JV-.087636 London. St Margaret's Church [Westminster]. Raleigh Window looking west. St Margaret's Church, London. Church interior with recessed gothic arches, and stained glass window in end wall; bible on eagle lectern left, pulpit right. Registered 1923 J Valentine & Co London, England JV-87636 pc/jf/mbTECH: Real Photograph. [arches unclassified],[London all views],[stained glass unclassified],[churches & chapels],[buildings - stone],[architecture - Gothic],[Collection - J Valentine & Co]

12. Set Exercise 1-1 Related work: The CLEF Cross Language Image Retrieval Track (ImageCLEF) <DOC> <DOCNO>stand03_2065/stand03_22080.txt</DOCNO> <HEADLINE>London. St Margaret's Church [Westminster]. Raleigh Window looking west.</HEADLINE> <RECORD_ID>JV-.087636</RECORD_ID> <DESCRIPTION>St Margaret's Church, London. Church interior with recessed gothic arches, and stained glass window in end wall; bible on eagle lectern left, pulpit right. Registered 1923 J Valentine and Co London, England JV-87636 pc/jf/mbTECH: Real Photograph.</DESCRIPTION> <CATEGORIES>[arches unclassified],[London all views],[stained glass unclassified],[churches and chapels],[buildings - stone],[architecture - Gothic],[Collection - J Valentine and Co]</CATEGORIES> <SMALL_IMG>stand03_2065/stand03_22080.jpg</SMALL_IMG> <LARGE_IMG>stand03_2065/stand03_22080_big.jpg</LARGE_IMG> </DOC>

13. Set Exercise 1-1 Related work: The CLEF Cross Language Image Retrieval Track (ImageCLEF) Boats on Loch Lomond Tay bridge rail disaster Great Yarmouth beach Metal railway bridges Culross abbey Road bridges Animals by the photographer Lady Henrietta Gilmour Ruined castles in England London bridge Damage due to war Golf course bunkers Portraits of Robert Burns Men and women processing fish A baby in a pram Picture postcard views of St Andrews Seating inside a church Woodland scenes Scottish marching bands Home guard on parade during World War II Tea rooms by the seaside Fishermen by the photographer Adamson Ships on the river Clyde Portraits of Mary Queen of Scots North Street St Andrews War memorials in the shape of a cross

14. Set Exercise 1-1 Golf course bunkers Golfplatz Bunker Bunkers de terrain de golfe Un bunker in un percorso di golf ; Bunkers in un campo di golf B&#x00FA;nkers en un campo de golf ; Pista de golf Bunkers op een golfbaan A relevant image will show a picture of a golf course in which a bunker can be clearly identified. The picture must be a photograph or a postcard, but not a drawing, e.g. a plan of the golf course. A bunker is a sandy hollow formed by wearing away of the turf, or nowadays an artificial sand-hole with a built-up face. An example relevant document is [stand03_1714/stand03_7020].

15. Set Exercise 1-1 Automatic query translation? Golf course shelter Bunkers of ground of gulf A bunker in a distance of golf bunkers in a golf course B??nkers in a golf course Track of golf Bunkers on a wave job

16. <Description> XRay: next to the diaphysis of the right femur, there is a large soft tissue mass with an oval ossification of 5 cm in length with multiple punctate calcifications (difficult to see an this digital radiograph). The cortex on the medial side of the distal part of the femoral diaphysis is of irregular outlines. MRI: demonstrates a large tumour mass of the right thigh which involves the medial and distal portions of the anterior rectus muscle and the vastus medialis. The lesion is relatively well delineated from the subcutaneous soft tissues, but not from the muscle. It is in intimate contact with the cortex of the femoral diaphysis which is slightly thin in its distal portion (on T1, coronal a) The superficial femoral vessels are partially encased by tumour and displaced posteriorly (red arrow on axial view on T2). There is subcutaneous soft tissue oedema and thickening of the adjacent skin. ….. CT of the chest: There are nodules in both lungs and in the right subpleural space consistent with metastases (considering the clinical situation). There are bilateral pleural effusions, larger on the left than on the right, and partial left posterior atelectasis. CT of the pelvis: demontrates a large metastatic mass consisting of large iliac adenopathies which encase the iliac vessels which however remain patent. Macroscopy: firm, elastic, relatively homogeneous, whitish mass with an ill defined ossified centre of about 5cm in length (arrow on macroscopy 2). Microscopy: there is proliferation of epitheloid cells with abundant cytoplasm and moderately atypical cyto-nuclear features (microscopy 3), separated by strands of collagen (stained blue on microscopy2). Small foci of tumour necrosis ….. </Description> <Diagnosis>Epitheliod Fibrosarcoma</Diagnosis> …. <ClinicalPresentation> ; <Commentary>; <KeyWords> ;<Anatomy> ; <Hospital> ; <Department> ; <Date> ; <Language> ; <Birthdate> ; <Age> <Creation> ; <DateTime> ; …

17. Describing images • In groups of 2 or 3, provide 4 keywords (single words, or phrases) to describe what you see in each of these signs:

18. Describing images • 40 • Circle • Red • White

19. Describing images • Circle • Red • White • Measurements • Triangles

20. Describing images • Red • White • Triangle • Car • Wavy lines

21. Describing images • Yellow • Black • Red • Two arrows

22. Describing images • Rectangle

23. Describing images • What is common? • Colours: Red White Yellow Blue • Shapes: Triangle Circle Rectangle • What differs? • “Content”

24. Understanding images • In groups of 2 or 3, provide 4 keywords (single words, or phrases) to describe what you understand about each of these signs:

25. Understanding images • 40 miles per hour speed limit (UK): kilometres (France)

26. Understanding images • Height restriction (imperial and metric)

27. Understanding images • Slippery road warning (warning)

28. Understanding images • The two right-hand lanes are closing within 800 yards, and you need to move into the two left hand lanes

29. Understanding images • Motorway (M62) - information

30. Describing images • What is common? • What differs?

31. Understanding images? • Understanding vs. seeing - easier or more difficult? • What did you agree on? Disagree on? • What classes exist? • Types: Signs giving orders; Warning signs; Direction signs; Information signs; Road works signs • “Why are you asking us to do all of this?”

32. What’s on a DVD? “moving picture” subtitles audio description dubbed speech scene indexing bibliographic information ….. Understanding images? Image from: http://www.ee.columbia.edu/~winston/research/trecvid2003/

33. What’s beyond the DVD? catalogues reviews & reviewers recommenders ….. Understanding images? Image from: http://www.ee.columbia.edu/~winston/research/trecvid2003/

34. Recall from Lecture 1: • It’s becoming easier and easier to create, distribute and ‘view’ multimedia data BUT • As the amount of multimedia data increases, it becomes harder to find what you want – “information explosion” • Multimedia information only has value if it can be found (retrieved) and used

35. “information explosion” • Data at the exascale - a few new exabytes (billion GB) of unique information / year • According to research at Berkeley, print, film, magnetic, and optical storage media - about 5 exabytes of new information in 2002; equivalent to 37,000 libraries the size of the Library of Congress book collections (29+m books) • New Media and ever-larger volumes of information: • Written: 24-hour news coverage published on news-wires and more recently using Really Simple Syndication (RSS) • Spoken: digital radio and on-demand podcasts; continuous televisual coverage through the variety of available digital television channels; broadband television efforts such as IET.tv, and on-demand programme schedules referred to as “timeshift TV”; CCTV

36. “information explosion”

37. “information explosion” • Multimedia data coding formats: • Text: TXT, PDF, DOC, HTML …. • Image: .JPEG , GIF …. • Audio: MP3 , WAV, MIDI …. • Video: .MPEG , AVI …. • Some formats emphasise compression • Some formats enable flexible editing / interaction with the multimedia content

38. “information explosion” • Average digital photograph in 24-bit colour uncompressed requires about 6Mb of storage • 1 second of digital video at 640x480 pixels uncompressed requires 27Mb; 1 hour requires 97Gb • multiply by 4 for full-screen video (1.23m pixels)! [Dunckley 2003: p. 26; p. 37] • HDTV ~ 2 million pixels (1920x1080)?

39. “information explosion” • Lossless Compression – a compression technique is lossless when no information is lost after compression-decompression • Lossy Compression – some information is lost after compression-decompression: depends how much loss can be tolerated [Dunckley 2003: p. 31] • Transcoding (moving between formats) may introduce loss also • Format can impact on data handling

40. “information explosion” • The compression of image and video data can take advantage of the fact that some changes to the data (i.e. loss of detail) are either imperceptible to the human eye, or result in a loss of perceived quality that can be tolerated for a certain application • There are trade-offs between: • Image quality and compression ratio • Codec processing time and compression ratio • 4Mb before compression, 1Mb after = 4:1 compression ratio; • codec = compression-decompression • Preferable to have compression techniques where users can set parameters for compression ratio / loss of quality / codec processing time

41. “information explosion” • Further detail on compression, e.g.: • Run-length encoding: 2211333334333222  22 21 53 14 33 32 16 characters becomes 12 characters • Huffman coding: 12334123343212334  AABA 17 characters becomes 4 characters plus ‘map’ 12334 = A; 32 = B [Dunckley 2003: p. 32]

42. Catalogues 0 GENERALITIES 1 PHILOSOPHY. PSYCHOLOGY 2 RELIGION. THEOLOGY 3 SOCIAL SCIENCES 4 VACANT 5 NATURAL SCIENCES 6 TECHNOLOGY 7 THE ARTS 8 LANGUAGE. LINGUISTICS. LITERATURE 9 GEOGRAPHY. BIOGRAPHY. HISTORY (Around 60k categories) Catalogues elsewhere? “information explosion”

43. Set Exercise 1-2:what is metadata? Skim read* the following web pages and extract / summarise 2-3 definitions of what metadata is and 3-4 reasons why people say metadata is important. http://dublincore.org/documents/usageguide/#whatismetadata www.tasi.ac.uk/advice/delivering/metadata.html www.openp2p.com/pub/a/p2p/2001/01/18/metadata.html *You do NOT need to read these articles in detail

44. Part 2: Metadata Multimedia data coding formats represent the multimedia content to reproduce it, metadata represents information about the content • What are the issues and challenges in making and using metadata standards? • Describing multimedia at different levels of abstraction; adding metadata at different stages of the media production process • Manual creation (cost) / automation; formalisation / restrictiveness • Objectivity / subjectivity – metadata is context-sensitive, time-variant, culturally variant

45. What is metadata? 26 26 what? 26 days Full days or working days? • What assumptions have been made in this exchange?

46. Sources: Various

47. Why do we need metadata standards? • Short answer – interoperability • Longer answer - interoperability between people, services and systems, including the automatic composition of computer programs for undertaking novel tasks “At least, Pat should be able to use Pat’s metadata. Slightly better, Chris should be able to use Pat’s metadata. Even better, Chris’s computer should be able to use Pat’s metadata. At best, Chris’s computer and Chris should be able to use Pat’s and Pat’s computer’s metadata” Adapted from Davis (1995) – Davis wrote ‘annotations’ rather than ‘metadata’.

48. Why do we need metadata standards? • The Semantic Web: interoperability between people, services and systems. Increased coverage for XML content: XML meant for machines and machines like precision • “Standardized metadata” • Enable convergence of formats, systems and collections to improve reusability / interoperability • Systemic efforts in ISO • Standards-conformant metadata can improve management of data and facilitate scaling up of research. • Research results identified using finer-grained metadata become available for use beyond that originally intended, ethical considerations allowing. • The Multilingual Internet? - September 2005, ICANN approved the first “ccTLD” for a particular human language and culture: ‘.cat’ (Catalan): not just ASCII?

49. What is metadata? <p> <p>? ?? <para> An unknown or unexpected error has occurred.

50. Metadata in HTML • Keywords (search engines?): • <meta name="keywords" content=“Multimedia, Metadata"> • Description: • <meta name="description" content=“Lectures on Multimedia"> • Refresh every 10 seconds: • <meta http-equiv="refresh" content=“10"> • Retire your content • <meta http-equiv=“expires" content=“Tues, 23 Jan 2007 09:00:00 GMT">