Towards a Visual Notation for Pipelining in a Visual Programming Language for Programming FPGAs

1. Towards a Visual Notation for Pipeliningin a VisualProgramming Language for Programming FPGAs Chris Johnston Supervisors: Donald Bailey, Paul Lyons Institute of Information Sciences and Technology Massey University, Palmerston North New Zealand

2. Outline • Background to the problem • Real time image processing using FPGAs • Overview of VERTIPH • Visual Environment for Real Time Image Processing in Hardware • Visual notations for pipelining • Pipelining is required to meet timing constraints • Representing pipelining in a VPL • Summary CHINZ July 2006

3. I/O RAM CLB CLB CLB FPGA Architecture • Programmable hardware • Configurable logic blocks (arbitrary logic functions) • Programmable interconnects (routing switches) • I/O • RAM • (Multipliers) • (Processor cores) CLB CHINZ July 2006

4. Image processing Extracting information from or improving an Image CHINZ July 2006

5. Why FPGAs? • Image processing operations • data intensive • computationally expensive • To do more work • serial processors have to go faster • FPGAs can process information in parallel • FPGAs • build custom hardware for each operation • can be reconfigured as needs change CHINZ July 2006

6. Issues With Programming FPGAs • Hardware design is different from software design (!) CHINZ July 2006

7. Function block Input data Function block Register Function block Register Output data Input data Output data Function block A pipeline Parallel operations Function block Register Function block Register Output data Input data Function block Register Function block Register Parallelism Temporal Spatial • Temporal and spatial parallelism Hybrid CHINZ July 2006

8. Hardware Design Entry Schematic HDLs(Hardware description languages) HLLs(High-level languages) VHDL code entity HA is port( x,y:in bit; z,C:out bit ); End HA; architecture BEHAVIOURAL of HA is begin C<=x and y; z<=x xor y; end HA; Handel-C code unsigned 1 x,y,z,C; par { C = x & y; z = x ^ y; } Increasing abstraction Hard to reuse Not algorithmic Verbose Low level Very flexible Like softwareHigher levelHave to work to a design model CHINZ July 2006

9. Architecture view Frame Pixel Line Pixel While If If else Computational view Resource & scheduling VERTIPH • Three Views of the System CHINZ July 2006

10. Architecture view Camera Interface Frame Buffer Manager RAM1 Data flow Control flow RAM2 VERTIPH (Architectural) Three Views of the System Shows components, data and control flow Allows for encapsulation of data structures and processes Bilinear Interpolation Video Driver Barrel Correction Keyboard Interface CHINZ July 2006

11. Architecture view Frame Pixel Line Pixel While If If else Camera Interface Frame Buffer Manager Bilinear Interpolation Video Driver RAM1 Data flow Computational view Control flow RAM2 Resource & scheduling Barrel Correction Keyboard Interface VERTIPH • Three Views of the System Shows components, data and control flow Allows for encapsulation of data structures and processes CHINZ July 2006

12. Architecture view Frame Pixel Line Pixel While If If else Computational view Resource & scheduling VERTIPH (Resource & Scheduling) Three Views of the System • Local and Global scheduling • Avoids resource conflicts • Shows when operations can run CHINZ July 2006

13. While (true) If (videoScanX == VisibleCols) Architecture view xc * xc xc * xc sx sx Operators xadd: x + 1 sqrd: sx + 2x + 1 kru: (sx + sy) * k correctx: mag * x correcty: mag * y Operators xadd: x + 1 sqrd: sx + 2x + 1 kru: (sx + sy) * k correctx: mag * x correcty: mag * y xc xc x x y + 1 y + 1 y y sy sy sy+2y+1 sy+2y+1 Frame Pixel Line Pixel While x xadd If sx sqrd If sx Kru1 else Interpolated LUT mag k correctx y correcty Resource & scheduling Clock cycle VERTIPH (Computational) Three Views of the System While (true) If (videoScanX == VisibleCols) -3 Computational view CHINZ July 2006

14. Time While A B C If If B else C VERTIPH Three Views of the System • Computational view • shows concurrency, pipelining and latency • no obvious way to show a pipeline Sequential Concurrency Computational view Parallel CHINZ July 2006

15. VERTIPH Pipelining notations Diagonal Gantt illustrates the dataflow spacially inefficient shows that A, B & C run in separate clock cycles A B C CHINZ July 2006

16. VERTIPH Pipelining notations Staggered Gantt A A A B B • shows when all processors are active C CHINZ July 2006

17. B B B1 B1 VERTIPH Pipelining notations Staggered Gantt with Conditional Branching branching makes diagrams more complex cannot express multicycle pipelines A A A C CHINZ July 2006

18. C A B D E F C1 VERTIPH Pipelining notations Sequential Pipeline • A series of processors perform successive phases of a task • processor passes data “down the line” when its phase is complete • early processors accept more data while later ones handle early data • A slider to change data interarrival times • Notation is compact, but hides processor concurrency CHINZ July 2006

19. C A B D E F C1 VERTIPH Pipelining notations Sequential Pipeline With Staggered Bars • Combines the benefits of Sequential and Gantt • Bars indicate: • phase, throughput & time when processors have valid data. CHINZ July 2006

20. extend clock lines down C A B D E F C1 C A B D E F C1 VERTIPH Pipelining notations Staggered Sequential Pipeline With Staggered Bars Detailed bars show repeated hardware CHINZ July 2006

21. Summary • Staggered Gantt • used too much screen space • could not describe all options • Sequential • was not expressive enough • Adding Staggered Bars • gives an indication of data throughput • Using Detailed Bars • Balance between complexity and expressiveness. • Shows hardware, phase and throughput CHINZ July 2006

22. FIN CHINZ July 2006