Real-time Image Processing System

1. Real-time Image Processing System Peter Chun Dr. Lev Kirischian Dr. Vadim Geurkov

2. Image Processor & hardware controllers (FPGA) Data Acquisition device (CMOS digital camera) Communication Interface (USB, FireWire, Parallel Port) VGA interface (DAC) Memory (SRAM) Overview Peter Chun

3. Schematic Peter Chun

4. Topics on the table • Controllers on FPGA • Digital Camera (OV5017 and M4088) • SRAM (IDT71V416: 256K X 16-bit) • VGA (DAC board – THS8134) Peter Chun

5. Topics on the table (continue…) • Real-time Image Processing system • Step1 : VGA verification • Step2 : SRAM WRITE/READ verification using verified VGA • Step3: Digital camera data acquisition on verified VGA using verified SRAM • Static one-frame • Continuous video capture • Video capture with pixel update Step by Step Guide : How to construct one Peter Chun

6. VGA • Hardware Interface • DB15 • Number of pins you need: • 6 • Pin Descriptions • Hsync: Horizontal Sync • Vsync: Vertical Sync • R: red • G: green • B: blue • GND: ground Peter Chun

7. 5 4 3 2 1 10 9 8 7 6 15 14 13 12 11 Hardware Interface: DB15 Peter Chun

8. VGA (continue…) • VGA arrangement (0,0) (0, 0) Horizontal Synchronization Vertical Synchronization (640,480) (640, 480) Peter Chun

9. D Red, Green, Blue Horizontal Sync (Hsync) C E B A Timing Requirements • Horizontal Peter Chun

10. Timing Requirements • Vertical R Red, Green, Blue Horizontal Sync (Hsync) Q S P O Peter Chun

11. VGA (continue…) • Example • 20MHz Peter Chun

12. SRAM • Signals • Control signals • we, oe, cs, address(length..0), bhe, ble • Data signals • Data(depth..0) • NOTE: “depth” is how many bits can you access at one time. “length” is how many of data can you store, which is equivalent to 2length. Peter Chun

13. SRAM (continue…) • read Peter Chun

14. SRAM (continue…) • write Peter Chun

15. SRAM (continue…) • Example • 40MHz Peter Chun

16. Row Amplifier for Gain Control Digital Pixel Process Optical Sensor Array A/D converter Column Analog pixel region Digital pixel region hsync vsync pclk data[max..0] Digital Camera • Hardware Overview Peter Chun

17. Digital Camera (continue…) • Reference Signals • Hsync • indicates the duration of active horizontal pixel insertion. It remains high during valid line and reset while not valid. • Vsync • represents the duration of active frame. When valid, it remains low and asserts high briefly to indicate the start of new frame. • Pclk • establishes the availability of the pixels. Either at the rising or falling edge of pclk, the pixel data is ready to be outputted. Peter Chun

18. Digital Camera (continue…) • SetUp • Exposure control, AGC (auto gain control), Gamma correction, Frame rate, resolution control • Method • Memory Access • I2C protocol Peter Chun

19. 1 1 2 2 3 3 initial initial ready ready done done Digital Camera (continue…) • Memory Access =100ns =100ns =50ns =50ns =50ns =20ns Peter Chun

20. case y is -- Initial state after resets when initial => if(loop_num = num_mul) then loop_num <= 0; y<=HWCTL_initial; else loop_num <= loop_num + 1; end if; -- Horizontal Window size control when HWCTL_initial => y<=HWCTL_ready; when HWCTL_ready => if(loop_num = num_mul) then loop_num <= 0; y<=HWCTL_done; else loop_num <= loop_num + 1; end if; when HWCTL_done => y<=VWCTL_initial; process(y) begin -- initial and reset values oeb <= '0'; csb <= '1'; web <= '1'; a <= "0000"; cam_data <= "00000000"; if y = initial then oeb <= '0'; csb <= '1'; web <= '1'; elsif y = HWCTL_initial then a <= "0110"; cam_data <= "00000000"; oeb <= '1'; csb <= '0'; web <= '0'; elsif y = HWCTL_ready then a <= "0110"; cam_data <= "00000000"; oeb <= '1'; csb <= '0'; web <= '0'; elsif y = HWCTL_done then oeb <= '0'; csb <= '0'; web <= '1'; 1 1 initial initial 2 2 ready ready 3 3 done done SetUp Example Peter Chun

21. Digital Design Tips • Multiple processes • For reference signals (pclk, vsync, href) • Filtering unstable states of the signals Peter Chun

22. Digital Design Tips (continues…) • Address space division • Address setup time exceeds synchronous clock events • Long integer or std logic vector need attention • Flip image • Use decrement of address space Peter Chun

23. Digital Camera (continue…) • Pixel Timing Peter Chun

24. Digital Camera (continue…) • Horizontal Timing Peter Chun

25. Digital Camera (continue…) • Vertical Timing Peter Chun

26. Design evolution • Frame based • One memory • Pixel based • One memory • Internal memory Peter Chun

27. Examples • 40MHz • 66MHz Peter Chun

28. Edge Detection • Gradient based Peter Chun

29. Edge Detection (continue…) • An edge pixel is described using two important features • Edge strength, which is equal to the magnitude of the gradient • Edge direction, which is equal to the angle of the gradient. Peter Chun

30. Max(|d-a|, |b-c|) ,where a b c d Edge Detection (continue…) • Robert’s cross Operator 385 pixels 288 pixels Peter Chun

31. 1 0 0 -1 0 1 -1 0 Robert’s Cross Operation 385 pixels a1 b1 c1 d1 e1 f1 …………………………… x1 y1 z1 a2 b2 c2 d2 e2 f2 …………………………… x2 y2 z2 288 pixels ……………………………………………………………………………………… ……………………………………………………………………………………… ……………………………………………………………………………………… ……………………………………………………………………………………… ……………………………………………………………………………………… or A B C D E F …………………………… X Y Z Peter Chun

32. Example • Real-time image processing system with edge detection Peter Chun