Introduction to Video on FPGA

1. Introduction to Video on FPGA Aaron Arenas

2. Agenda • Video Displays • Video Generators • Video Interfaces • VGA Lab Specific's • Lab Overview Video Generator Video Display Video Interface

3. Video Displays • How do displays display? • Technologies • CRT • Plasma • LCD • Color • Display Specifications

4. How do displays display? • Interlaced • Progressive

5. CRT (Cathode Ray Tube) (Aaron) • Monochromatic • Color • Comparison • Pros • First video technology • Cons • Form Factor • Price • Screen Size • Health Risks

6. Plasma • Pros • Better Color Quality • Bigger Screen Sizes (upwards of 100 in) • Faster Response Times • Cons • Power Hungry • Heavy • Longevity

7. LCD (Liquid Crystal Display) • LCD Panel Types • Twisted Nematic (TN) • In Plane Switching (IPS) • Pros • Lower weight & power • Longevity • Cons • Poor viewing angles • More expensive

8. Display Specifications (1 of 2) • Screen Size • Resolution • Aspect Ratio

9. Display Specifications (2 of 2) • Refresh Rate

10. Color Models • RGB • Red, Green, and Blue • YCbCr • Y = Brightness (Luma) • Cb = Blue – Luma (Blue Chromaticity) • Cr = Red – Luma(Red Chromaticity) • Application: • MPEG Video Compression

11. Color Models – YCbCr Example • Color Image • Y/Luma/Brightness • Cb • Cr

12. Color (1 of 2) • Color Depth • Number of shades a color can have • Color Space • Total number of colors that can be created

13. Color (2 of 2) • Contrast • Brightness

14. Video Generators • Computers • Graphic Processing Unit (GPU) • FPGA Display Adapters Display Cards Graphics Cards Video Card Graphics Adapter Video Controllers Display Controllers All refer to the same thing…

15. Computers Video Controllers • Computers have video controllers (aka video or graphics card and graphics or video adapters) • Key hardware component that allows computers to generate graphic information to video display devices. • Many modern computers using either integrated or external graphics processing units

16. Graphic Processing Unit (GPU) • Similar to CPU • Specialized electronic circuit designed to • Translate data into images • Perform complex mathematical calculations • They are embedded in mobile phones and game consoles • Designed by Nvidia, AMD, and soon Intel

17. FPGA (1 of 2) • Why use FPGA for Video? • Image Processing (IP) applications are computational demand • Pros of FPGA • Fully Custom End to End Solution • Reconfigurable • Greater Potential Performance • Cons of FPGA • Requires low level, hardware-orientated programming model

18. Video Interfaces • Many image processing algorithm can be described in terms of a Directed Acyclic Graph (DAG) • Common IP tasks can be classified in terms of the locality of their data access requirements into three categories: • Point • Neighborhood • Global

19. Video Interfaces • Overview • VGA • DVI • HDMI • DisplayPort

20. Video Interfaces • What's their purpose? • There are standardize medium to interface between a video source and a display

21. VGA (Video Graphics Array) • Created by IBM in 1987 • First introduced in IBM PS/2 line of computers • Color Transmission: Analog • Oldest Standardized Connector • Limited to RGB • Capable of doing 1080p resolution

22. DVI (Digital Visual Interface) • Created in 1999 by Digital Display Working Group • Intel is apart of this group • Color Transmission: Digital • Encodes color data for transmission • Limited to RGB • In dual link mode, • Can do 3840 x 2400 @ 30 Hz & 1080p @ 120 Hz

23. HDMI (High Definition Media Interface) • Created by 90 companies (7 founders and 83 in forum) in 2002 • Color Transmission: Digital (Serialized) • Proprietary • Include audio in transmission • Latest version is 2.1 (released in 2017) • Includes HDR and support for 10K (or 10240 x 4320) @ 120 Hz

24. DisplayPort • Created by VESA in 2006 • Color Transmission: Digital (Serialized) • Includes audio • Royalty-Free • Latest version is 1.4 (released in 2016) • Includes HDR and support for 8K (or 7680 x 4320) @ 30 Hz

25. VGA Lab Specific’s Overview • Signaling & Timings • Color • Lab Breakdown • Lab Block Diagram • Lab - Pattern Generator

26. Row VGA Lab Specific’s - Timing Pixel • Signaling & Timings Row Frame

27. VGA Lab Specific's - Color • Color • Depth = 16 (or 4-bit) per R, G, & B • Space = 4096 • Interface with an FPGA (DE10 Lite)

28. Lab Overview (Aaron) • Display to a Monitor with FPGA • Pattern Generator • Upgrade Resolution • Pattern Generator V2 • Pong Demo • Note: Sections 1-4 all have intentional problems

29. FPGA Outside World Block Diagram Sync Generator Video Display H Sync PLL CLK V Sync Counter CLK ROM ( w/ decoder) Pattern Generator Red Green ADDR & DATA Blue

30. Lab (Pattern Generator)

31. Questions

32. Thank you and Workshop Time!

33. END of SlideShow

34. History of Display Technology Color CRT became the standard No longer being made Prototypes & Development Commercially Available by RCA Idea Demonstrated CRT 1897 1953 1970 1940 1920 2007 Plasma LCD 1897

35. Video Generators Overview • Computer • A video controller, often referred to as a video or graphics card, is a key hardware component that allows computers to generate graphic information to any video display devices, such as a monitor or projector. They are also known as graphics or video adapters. Some modern computers do not include video cards, but rather have graphics processing units directly integrated into the computer's motherboard. • GPU • As the brain of a computer's motherboard is the CPU, video controllers have their own unique "centers," referred to as the graphics processing unit, although the GPU is also referred to as the visual processing unit. The GPU's specialized electronic circuit is designed specifically to translate data into graphic images and performs complex mathematical calculations in order to do so. GPUs are also embedded into mobile phones and game consoles. • FPGA • Many modern Image Processing (IP) applications (such as processing video and very large images) are so computationally demanding that special purpose hardware solutions need to be considered. Reconfigurable hardware in the form of FPGAs can offer the performance advantages of a custom hardware solution, while their inherent reprogram ability feature makes them multi-purpose and reusable. However, a big disadvantage is the low level, hardware-oriented programming model needed to fully exploit the FPGA’s potential performance.

36. FPGA (2 of 2) A high level model for IP operations Many image processing algorithm can be described in terms of a Directed Acyclic Graph (DAG), where vertices represent IP tasks, and the directed edges represent the data flow. • Nodes are typically simple tasks such as adding two input images, or an image convolution. Common IP tasks can be classified in terms of the locality of their data access requirements into three categories: • Point operations: The same operation is applied to each individual pixel of one or many source images to produce a corresponding result pixel in the new image. These include: relational operations (e.g. ‘≥’, ’≤’, ‘=’), arithmetic operations (e.g. ‘+’, ‘-‘, ’*’, ‘÷’), logical operations (e.g. ‘AND’, ‘OR’) and Look-Up tables. The operation could either be between two images or between an image and a scalar value. • Neighborhood operations: In neighborhood operations, a new pixel value is calculated using only the pixel values in the neighborhood of the original pixel and the weights in a window (e.g. convolution). This is done for all image pixels, and results in a new image. Neighborhood operations are completely defined by a local operation between corresponding pixels and window values (e.g. multiplication), a global operation (e.g. accumulation) which reduces the window of intermediate results to a single result pixel, and a window (with given shape and coefficients). • Global operations: These operations operate globally on the whole image. We can distinguish two common types of simple global operations: • Reduction to Scalar (RS): These operate on the whole image to produce a scalar as a result. Examples include count, global maximum, global minimum and global accumulation (Σ). • Reduction to Vector (RV): This operation operates on the whole image to produce a vector as a result.

37. FPGA (2 of 2) • Many image processing algorithm can be described in terms of a Directed Acyclic Graph (DAG), where vertices represent IP tasks, and the directed edges represent the data flow. • Nodes are typically simple tasks such as adding two input images, or an image convolution. Common IP tasks can be classified in terms of the locality of their data access requirements into three categories: • Point • Neighborhood • Global

38. Video Generator Video Display Video Interface

39. FPGA Outside World Sync Generator Video Display H Sync PLL CLK V Sync Counter CLK ROM ( w/ decoder) Pattern Generator Red Green ADDR & DATA Blue