T325: Technologies for digital media

1. T325: Technologies for digital media Secondsemester – 2011/2012Tutorial 4 – Seeing and Hearing: Multimedia Output and Input Arab Open University – Spring 2012

2. Introduction • Perception • Display technologies • Rasters and video signals • Graphics operations • Capturing the content • The sound channel Outline Arab Open University – Spring 2012

3. How we actually see things ? • How display data for real and virtual images is obtained and manipulated? • Technologies involved in presenting multi-media information • Technologies involved in capturing it from the outside world or conjuring it up from virtual objects in a computer • Sound channels Introduction Arab Open University – Spring 2012

4. Perception Arab Open University – Spring 2012

5. Why do you hear by perception ? Why do you think perception is important for multimedia technologies? Question Arab Open University – Spring 2012

6. Perception has been of interest to technologists from the earliest days of radio and television, largely because of the bandwidth issue. • Studies of human perception indicate what is important to send, and what can safely be discarded. • Human perception theory is an essential factor in communication systems standards Introduction Arab Open University – Spring 2012

7. Light is a form of electromagnetic radiation, like radio but at a much higher frequency • the colors of the rainbow are due to light at specific wavelengths • A color TV picture is made up of red, green and blue dots (RGB)(additive colors) and a color printer uses yellow, cyan, magenta and black (CYMK) (subtractive colors) inks. • There is no need to send data representing the entire range of visible wavelengths for each point in a scene; just three quantities will do. Light and Color Arab Open University – Spring 2012

8. The wavelength of visible light is between about 400nm and 700nm (1nm = 10-9 m). If the speed of light is 3 x108 m/s, what is the frequency range of visible light? (Frequency = speed/wavelength.) • Answer • At 700 nm the frequency is (3 x108)/(700 x10-9) Hz, which is approximately 430 x1012 Hz or 430 THz (terahertz). • At 400 nm the frequency is :(3 x108)/(400 x109) Hz = 750 x1012 Hz or 750 THz. Light and Color - Activity Arab Open University – Spring 2012

9. The eye as a camera has a light-sensitive surface, the retina, analogous to the film in a film camera or the detector in a digital camera. • A lensforms an image on the retina. • The iris, like a camera aperture, responds to sudden changes in brightness • The retina has specialized sensory cells which convert light to electrical signals by triggering nerve impulses Light and color Arab Open University – Spring 2012

10. Nerves carry signals in the form of trains of impulses to the regions of the brain that deal with sensory input. • Impulses are ‘all or nothing’; there are no half-size impulses. • There is a time gap between impulses there is a maximum rate at which nerves can conduct impulses. • Brain Processing : timescales of milliseconds • Silicon logic : nanosecond timescales • Vast numbers of interconnections between individual nerve cells or neuronsmakes the brain particularly adept at parallel processing operations: pattern recognition, that have been found difficult to implement on traditional computers with their serial architectures. Light and Color Arab Open University – Spring 2012

11. There are four types of light-sensitive cell in the retina: • S cones, which respond preferentially to shorter wavelength light. • M cones, which respond preferentially to medium wavelengths. • L cones, which respond preferentially to longer wavelengths. • Rods, which are very sensitive cells used in night vision. Light and color Arab Open University – Spring 2012

12. Color decision (eye and brain) relative responses of the three types of cone. • Displays with RGB dots stimulate L, M and S cones respectivelycan give the illusion of a wide range of colors simply by varying the relative brightness of the dots Responses of L, M and S cells Light and Color Arab Open University – Spring 2012

13. Cones do not respond instantly when the light falling on them changes. • The chain of chemical reactions behind their operation is very fast, but still takes time • A flash of light causes a response in a cone that builds to a maximum in about 50 ms and lasts about 200 ms, so images on the retina persist for a short time. • This ‘persistence of vision’ is relevant to television and film, where sequences of still images are perceived as smooth motion Light and color Arab Open University – Spring 2012

14. Alternative to the RGB representation • A light source can be characterized by its: • Luminance: a measure of how bright it is • Chromaticity: which identifies the particular shade of color • Shining a white light on an object increases its luminance without significantly affecting its chromaticity. • Luminance is related to the total of the red, green and blue components • Chromaticity depends on their relative proportions. Luminance &Chromaticity Arab Open University – Spring 2012

15. The points along the curved boundary correspond to light of single wavelengths(the rainbow spectrum). • There is a point in the middle of the diagram that corresponds to light of all wavelengths(perceived as white, or grey at lower levels of luminance). • The diagram shows one particular level of luminance. • You can imagine a series of such diagrams all similar but lighter or darker to cover the full range of possibilities. CIE chromaticity diagram Arab Open University – Spring 2012

16. The triangle shown corresponds to the colors that are available on a particular RGB display. • The three corners of the triangle represent the red, green and blue primary colors of the display. • Any color within the triangle or along its edges can be reproduced by the display, usingred, green and blue in the appropriate proportions CIE chromaticity diagram Arab Open University – Spring 2012

17. Some colors lie outside the triangle cannot be reproduced accurately with this display. • You would see a nearby color from the range that the display is capable of showing. • The range of colors available on a display is called its gamut. • Color printing also has a gamut. • Generally, the gamut does not cover the entire CIE chromaticity diagram • Colors outside the gamut are displayed as a near equivalent displayable color. CIE chromaticity diagram Arab Open University – Spring 2012

18. The human visual system is much better at resolving luminance information than chromaticity or chrominance. • The analogue system has a full-bandwidth luminance signal together withtwo chrominance signals with much lower bandwidth. • RGB signals, such as those from a color TV camera, can easily be converted to a luminance signal and two chrominance signals. • The eye is most sensitive to green, less so to red and less again to blue the components are given unequal weighting. • The luminance signal Y is given by : Y = 0.587G + 0.299R + 0.114B • The chrominance signals are also derived by combining the R, G and B signals. U = 0.492(B - Y) and V = 0.877(R - Y) Note: the formulas don’t need to be memorized Color in TV and video Arab Open University – Spring 2012

19. Photographs, paintings and TV pictures give us a two-dimensional representation of a three-dimensional world. • Our two eyes provide slightly different viewpoints of a three-dimensional scene, and this gives some clues about depth. • But we can still appreciate the depth in a picture with one eye closed. • There are several visual cues in an image that give us this feeling of depth. • They come across automatically when it is a picture of a scene taken with a camera. • Should be taken into consideration when creating a three-dimensional object in a computer that will look correct on a two-dimensional screen just as a painter does Depth in images Arab Open University – Spring 2012

20. Depth in images Arab Open University – Spring 2012

21. Display Technologies Arab Open University – Spring 2012

22. A display is an array of red, green and blue elements • Each element can individually be controlledin brightness so as to convert images from digital data to visible form. Introduction Arab Open University – Spring 2012

23. Transmissivemethods: where the picture elements pass or obstruct light. The light energy comes from a light source behind the array of picture elements. Example: Liquid-Crystal display (LCD) • Emissive methods: where each picture element emits light, through the conversion of electrical energy. Example: Cathode ray tube (CRT) and Plasma • Reflective methods: where the picture elements reflect more or less of the light that shines on them and thus appear light or dark. The light energy comes entirely from ambient lighting. Example: Electronic paper Classification of Display techniques Arab Open University – Spring 2012

24. Light-emitting diodes (LEDs) are semiconductor devices that convert electricity directly and efficiently into light. • The wavelength of light emitted is determined by the material used. • Red, green and blue types are available and can be put together in an array to make a display. • Very large emissive displays of this type are often seen at entertainment and sporting events. Light Emitting Diode (LED) Arab Open University – Spring 2012

25. How many LEDs would be needed to make a 640 x 480 pixel screen of this type, if a pixel consists of one LED of each color? • 640x480 = 307 200 of each of red, green and blue or • 3 x 307 200 = 921600 LEDs in total. • Assuming each LED occupies 1 cm2, what area would it cover? • 1m2 = 10^4cm^2, so the area is 921600/10^4 = 92 m2 approximately Display technologies – Activity 4.5 Arab Open University – Spring 2012

26. Liquid-crystal displays (LCDs) operate by controlling the passage of light. • Most LCDs (used television and computer monitors) are illuminated from behind with a uniform artificial backlight • Some types are viewable using only ambient light, by means of a reflective layer behind the display • Consume more power than LEDs: Although LCDs consume little power themselves, but the backlight needs power and this can be a consideration with battery equipment. Liquid-Crystal Display (LCD) Arab Open University – Spring 2012

27. LCDs depend on polarization of light, a property which is exploited in many sunglasses. • Light is a transverse wave, as with all types of electromagnetic radiation. • Electromagnetic wave:electric field component and a magnetic field componentat right angles to the direction of travel and also at right angles to each other. Liquid-crystal displays Arab Open University – Spring 2012

28. If the fields are predominantly in one direction then the radiation is said to be polarised. • Example: if the electric field is up and down in a wave moving horizontally • Many radio and TV transmitters use polarised waves, which result from the design of the antenna. • Most light sources are polarized, sending out waves of all orientations. • Polarising filters such as those used in sunglasses pass only light which is oriented in a particular direction. Liquid-crystal displays Arab Open University – Spring 2012

29. Light is twisted using liquid crystals • Liquid crystals are unusual substances. • The molecules move around quite freely • They do not have a fixed long-term structure like crystalline solids. Liquid Crystal Displays Arab Open University – Spring 2012

30. The molecules tend to align themselves in loosely ordered structures. • The molecules tend to line up in an electric field. • At and around surfaces, the molecules tend to follow features such as parallel grooves  This gives some control over the structure of the liquid crystal. • The ordered molecules can affect the direction of polarization of light. Liquid Crystal Displays Arab Open University – Spring 2012

31. Liquid crystals used in displays are held in a thin layer between two glass sheets. • In one common type, the two glass sheets are covered with a polymer surface with parallel grooves. • The grooves on one sheet are at right angles to the grooves on the other sheet. • This imposes a twisted structure on the liquid crystal molecules. • While the liquid crystal has this twisted structure, light shining through it has its polarization direction rotated through 90º Liquid Crystal Displays Arab Open University – Spring 2012

32. The assembly of liquid crystal held between the two glass sheets is placed between two polarizing filters. • Because of the twist, light can go through both filters. • So we have reversed the action of the filters. • To be useful as a display, though, we also need a way of returning it to the previous condition of stopping light. • This is simply done by applying an electric field, which lines up the liquid crystal molecules in a parallel arrangement which does not affect polarization. Liquid Crystal Displays Arab Open University – Spring 2012

33. How to control each element (pixel) on a LCD TV screen? Question Arab Open University – Spring 2012

34. Matrix arrangement of rows and columns individual pixels could be addressed by selecting the correct row and column • Example: an array of 1000 x 1000 pixels would only need 2000 conductors rather than a million. • The top electrodes of each column are connected together, and so are the bottom electrodes of each row. • Addressing a particular pixel apply a voltage between the conductors for the pixel’s row and column. Liquid Crystal Displays Arab Open University – Spring 2012

35. A device is needed at each intersection which will respond when the correct row and column signals are applied but not otherwise Use thin-film transistor (TFT). • A TFT, like other transistors, can be used as a semiconductor switch, turning on and off the voltage across the LCD elements. Liquid Crystal Displays Arab Open University – Spring 2012

36. Plasma displays fall in the emissive category. • Use similar physical principles to fluorescent lighting. • A display is made up of an array of small cells that can be controlled individually. • A plasma is a mixture of free electrons, atoms from which an electron has been removed (ions) which have a positive charge, and unaffected atomscan be caused by very high temperatures or by electric fields • Plasmas are produced in a special gas mixture in fluorescent tubes by applying a voltage between the two ends. Plasma displays Arab Open University – Spring 2012

37. Freely moving electrons in plasma  can carry an electric current. • Energy supplied to the plasma from the current source is turned into radiation at certain specific wavelengths. • This radiation may fall in the visible region • In practical, plasma radiation is converted to a different wavelength using a ‘phosphor’because much of the radiation from effective plasma sources is in the invisible UV region. Plasma displays Arab Open University – Spring 2012

38. Phosphor example : The white material that coats the inside surface of a fluorescent tube • A phosphor absorbs energy and then reradiates it at fixed visible wavelengths. • There is a well-established technology for producing efficient phosphors with bright colors. • The phosphor used in fluorescent tubes emits white light and phosphors in CRTs emit red, green and blue. • In CRTs, the energy is supplied to the phosphor by an electron beam rather than by UV radiation, but the effect is the same Plasma displays Arab Open University – Spring 2012

39. Electronic paper has been proposed as a reflective type of display • Visually more similar to reading print on paper than looking at a conventional computer screen. • Being reflective rather than backlit, they are thought to be easier on the eye. • Non-volatility feature: the image does not disappear when the power is switched off. • Some of the technologies use floating charged particles with sizes in the order of tens of micrometers, suspended between two flexible sheets. • The sheets have transparent electrode arrays which can attract or repel the particles, bringing them into or out of view. Electronic paper Arab Open University – Spring 2012

40. Rasters and Video Signals Arab Open University – Spring 2012

41. For TV, the picture arrives as a serial stream of analogue or digital information, in a standard format where exact timing is important. • The end product is three video signals representing red, green and blue, together with signals that are used for synchronization. • When computer monitors were introduced they followed TV practice Rasters and video signals Arab Open University – Spring 2012

42. Moving images are an illusion produced by displaying a series of still images or frames. • TV frames are built up by scanning: writing a line at a time and moving down to the next line when a line is complete, in the same way that we read text on a page. • The lines were created by scanning, thus covering the whole screen with a pattern called a raster. • In digital systems the line is a row of a finite number of pixels; for example, 720 pixels/line. Rasters and video signals Arab Open University – Spring 2012

43. The idea of raster scanning is deeply embedded in computer graphics. • Image data is routinely held in blocks of memory in the expectation that it will be read out in a raster scanned fashion. • In a computer, the video output is produced by repeatedly reading out the contents of a block of memory (sometimes called the ‘frame buffer’) and sending it to a digital-to-analogue converter. • Each memory location holds a coded value for the red, blue and green content of a pixel. Rasters and video signals Arab Open University – Spring 2012

44. With the demand for higher definition and a greater range of colors, the volume of data that needs to be sent to the display has increased dramatically • This led to introduction of new digital interface standards to replace the traditional analogue interface. • A high-definition TV data stream compressed for broadcast takes 19 Mbit/s, while the data rate off a Blu-ray disc is 48 Mbit/s. • HDMI1.3 specification offers 10.2 Gbit/s of bandwidth, to support a 48-bit rather than 24-bit color gamut • Last version of HDMI specification is 1.4a (March 2010) Rasters and video signals Arab Open University – Spring 2012

45. Graphics operation Arab Open University – Spring 2012

46. A picture can be built up by • reading out the contents of a memory, • with a one-to-one correspondence between memory locations and the pixels on a screen. • A two dimensional operations to move objects around the screen can be done just by changing where the data representing the object is in the memory. (See Activity 4.10) Two dimensional operations Arab Open University – Spring 2012

47. Graphics operations Arab Open University – Spring 2012

48. An object may be moved around the screen (translation) by simple addition of memory addresses. • Related techniques can be used to perform other operations, such as scaling (expansion or contraction) or rotation. • If there are a number of objects moving around, they may need to pass in front of or behind each other. • Remember that interposition or occlusion is an important factor in how things are perceived. Two-dimensional operations Arab Open University – Spring 2012

49. Speeding up two-dimensional operations by processing a whole block of data (in data transfer and logical operations) instead of a byte or word at a time • Introducing new set of computer instructions : bit block transfers (abbreviated to bitBLTor simply BLT). • Use of dedicated hardware in the graphics processors for BLT to relieve the main processor of this burdensome task • These operations are simple and repetitive but involve moving large amounts of data, • Carrying them out separately from the main processor can save a significant amount of computing time. Two-dimensional operations Arab Open University – Spring 2012

50. Much of the early interest in three-dimensional graphics was related to CAD (Computer-aided design). • A designer could create a virtual three-dimensional model in a computer, display it on a screen and look at it from all angles. • One of the first problems to be solved was how to represent a three dimensional object without needing to store and process an unfeasibly large mass of data. • Most three-dimensional objects could be represented by storing the coordinates of a relatively small number of points. • The rest of the object could be filled in later using a variety of algorithms to interpolate between the known points. Three-dimensional operations Arab Open University – Spring 2012