SIFT, FPGAs and Forth

1. SIFT, FPGAs and Forth Dave Wyland Reasonable Machines © David C. Wyland 2006

2. Overview • SIFT algorithm for Object Recognition • SIFT: FPGA vs DSP • Forth Machine in FPGA • SIFT on the R1A1 - The R1A1 Robot Project • SIFT and R1A1 Status • Implications for Robotics and HBRC © David C. Wyland 2006

3. SIFT – What Is It? • An algorithm for object recognition • Developed by David Lowe of University of BC • Identifies objects using ordinary cameras (TV, webcam) • At any distance • For any orientation (pose) • For (almost) any lighting • With simple image distortions © David C. Wyland 2006

4. SIFT – What Good Is It? • Lets us recognize objects automatically • Gives bearing and approximate range with 1 camera • Gives pose of object: normal, tilted, etc. • Lets our robots see what we see • Lets us build robots to go to named objects, pick them up © David C. Wyland 2006

5. SIFT – How does it work? • Outline • Find keypoints using Difference of Gaussians • 2D Gaussian filters of image • Look at 16x16 patch around keypoint • Generate a 128 dimension feature vector per point • Record new or search for match among recorded © David C. Wyland 2006

6. SIFT – References • Lowe, David, “Distinctive Image Features from Scale Invariant Keypoints,” International Journal of Computer Vision, 2004. • Google for SIFT, Scale Invariant Feature Transform, David Lowe © David C. Wyland 2006

7. SIFT – How Much Computation? • Steps in Algorithm • Keypoint finding: 49.6 x 10^6 pixel operations • Keypoint descriptor generation: 1.4 x 10^6 pix ops • Keypoint search: 100 ref objects: 32 x 10^6 compares © David C. Wyland 2006

8. SIFT – FPGA vs DSP • 600 MHz ADI Blackfin DSP vs Xilinx Spartan 3 FPGA • Time for DoG on 640x480 Pyramid, 3 planes/level • Blackfin: 12 frames/second at 600 MIPs • FPGA: 41 frames/second at 50 MHz © David C. Wyland 2006

9. FPGA Characteristics © David C. Wyland 2006

10. Xilinx Spartan 3 Evaluation Board - $99 © David C. Wyland 2006

11. Xilinx Spartan 3 Evaluation Board - $99 © David C. Wyland 2006

12. FPGA: Housekeeping CPU • Need a CPU function for housekeeping, supervisor • Plenty of resources in FPGA for a CPU • Proposal: 32-bit CPU running Forth => Forth machine • 32-bits to address at least 1Meg of memory • Added benefit: Use primitives for FPGA custom functions • Launch & sequence keypoint processing steps • Control complex I/O, etc. © David C. Wyland 2006

13. Forth Machine in FPGA • One 512x36 block RAM holds data and return stacks • 512 words => 8 sets of stacks for multitasking • Use internal block RAM and external SRAM • Use 2+ block RAMs = 4 Kbytes for fast primitives • Have >200 Kbytes of external SRAM available for code, variables • Boot load from/save to 32 Kbyte serial EEPROM • Ref: Stack Computers by Philip Koopman © David C. Wyland 2006

14. Forth Machine Support Idea • Use browser in a PC/Laptop for an IDE • Provides a clean way for visualizing Forth • Small TCP/IP stack in FPGA to communicate with Net, browser • Use web page and HTML links for word display • Show colon defs as a list of links to the words in the def • Click on a link, and nest into sub word • Primitives show the Forth machine instructions © David C. Wyland 2006

15. Forth: Some Controversial Ideas • Use Data Stack for parameters, not calculation • Data stack ops are arcane: RPN calculators have lost the war • Use local variables + a “frame pointer” FP = SP saved at entry • Input parameters have fixed locations relative to FP • Have ops that reference stack values rel to FP + local vars • Use fixed size return stack “frames” for local variables • Push 8 words instead of 1 • 1 word for return addrs, 1 for loop count, 6 for local variables © David C. Wyland 2006

16. SIFT on the R1A1 David Wyland Reasonable Machines © David C. Wyland 2006

17. The R1A1 Project • A Prototype to Test a Robotic Architecture • Animal Training as a Robot Design Model • Platform: Zagros base, Mini-itx CPU, Lynxmotion arm, webcam • Test Task: Go get a named object and bring it back • David Wyland • Reasonable Machines © David C. Wyland 2006

18. R1A1 © David C. Wyland 2006

19. R1A1 Block Diagram • David Wyland • Reasonable Machines © David C. Wyland 2006

20. R1A1 Status • SIFT algorithm running on a PC • Demo at HBRC meeting • SIFT Gaussian filter running in FPGA • R1A1 elements to be done • Platform control: drive motors, arm servos, camera control • Dialog mgr (alicebot), Task mgr (RAP), Behavior mgr © David C. Wyland 2006

21. Robot Design References • Wyland, David, “Autonomy Without Independence: Animal Training as a Robot Design Model,” 2nd Workshop on Radical Agent Concepts (WRAC), NASA Goddard Spaceflight Center, September 2005. To be published by Springer-Verlag • Wyland, David, “Reasonable Machines: Analogical Reasoning in Autonomous Agent Design,” 1st Workshop on Radical Agent Concepts, NASA Goddard Spaceflight Center, January 2002. Springer-Verlag © David C. Wyland 2006

22. SIFT: Implications for Robotics • Robots are currently blind and deaf • Try walking around with your eyes closed + earplugs • With SIFT, robots can move around and pick up objects • Work in changing environments • With SIFT, we can make robots just like in the movies • Walk, talk and do things for us © David C. Wyland 2006

23. SIFT: Implications for HBRC • With a SIFT Camera, We can make robots that: • Walk around without bumping into things • Can go to named objects and pick them up • With a SIFT Camera, We Can Make Real Robots • Just like In the Movies (JLIM™) • With SIFT, we can make robots just like in the movies © David C. Wyland 2006

24. That’s All, Folks David Wyland Reasonable Machines © David C. Wyland 2006