Memorial University of Newfoundland Faculty of Engineering & Applied Science Engineering 6806

1. Memorial University of Newfoundland Faculty of Engineering & Applied Science Engineering 6806 Electrical & Computer Engineering Design Project INTRODUCTION TO MACHINE VISION II Prof. Nick Krouglicof Memorial University of Newfoundland

2. Presentation Outline • Machine vision systems for mechanical metrology • A systematic approach to the calibration of machine vision systems for industrial metrology • An automatic, machine vision based system for robot calibration • Industrial Applications of Machine Vision • High speed, line scan camera-based inspection system for the food processing industry • Vision based inspection of liquid crystal display (LCD) modules • CPLD based interfaces for sensors and actuators Memorial University of Newfoundland

3. A systematic approach to the calibration of machine vision systems for industrial metrology • In the field of machine vision, camera calibration refers to the experimental determination of a set of parameters which describe the image formation process for a given analytical model of the machine vision system. • Ideally, camera calibration is performed without specialized optical equipment, without modifications to the hardware, and without a priori knowledge of the vision system. • Most calibration techniques are based on the observation of planar (2D) targets with a large number of control points. Memorial University of Newfoundland

4. A systematic approach to the calibration of machine vision systems for industrial metrology The machine vision parameters which must be identified include : a) The scale factor b) The frame buffer coordinates of the image center c) The effective focal length of the lens-camera assembly d) The radial lens distortion coefficient e) The pose (position and orientation) of the camera Parameters a) through d) are classified as intrinsic, e) as extrinsic. Memorial University of Newfoundland

5. A systematic approach to the calibration of machine vision systems for industrial metrology Memorial University of Newfoundland

6. A systematic approach to the calibration of machine vision systems for industrial metrology Memorial University of Newfoundland

7. A systematic approach to the calibration of machine vision systems for industrial metrology • In general, most camera calibration techniques fail because: • They fail to account for the mathematical dependency between the image center and the X-Y position of the camera. • The determination of the intrinsic parameters assumes a priori knowledge of the extrinsic parameters (i.e. position and orientation of the camera). Therefore, the extrinsic parameters are generally determined first. • In order to accurately evaluate the intrinsic parameters, the extrinsic parameters must be know to a degree of accuracy which is practically unattainable (6 significant digits). Memorial University of Newfoundland

8. A systematic approach to the calibration of machine vision systems for industrial metrology In order to compensate for the deficiencies of existing calibration techniques, a special fixture was designed to evaluate the critical parameters (i.e. image center and effective focal length) before the remaining parameters. Memorial University of Newfoundland

9. A systematic approach to the calibration of machine vision systems for industrial metrology • To locate the edge with sub-pixel accuracy, the gradient in the vicinity of the edge is reconstructed using a continuous Gaussian function. • The coordinates of the pixel nearest to the edge are denoted by i and j. The location of the edge with sub-pixel accuracy is determined by calculating the offsets x and y which maximize the function s(x,y) below over a 5 by 5 kernel : Memorial University of Newfoundland

10. An automatic, machine vision based system for robot calibration Objective: To develop a low cost, non-contact robot calibration system that can be applied on a routine basis without significantly affecting production schedules. Based on a CCD camera mounted on the robot’s wrist. The camera is used to determine the position and orientation of a fixed, passive target in six degrees-of-freedom. The system employs the Hayati and Mirmirani modeling convention for closed kinematic chains and non-linear least squares analysis to determine the real kinematic parameters specific to a particular robot. Memorial University of Newfoundland

11. An automatic, machine vision based system for robot calibration Memorial University of Newfoundland

12. An automatic, machine vision based system for robot calibration Memorial University of Newfoundland

13. An automatic, machine vision based system for robot calibration Memorial University of Newfoundland

14. Industrial Machine Vision Applications Memorial University of Newfoundland

15. High Speed, Line Scan Based Inspection System Memorial University of Newfoundland

16. Line Scan Based Inspection System: Specifications • Objective: To remove defects (I.e. visible, dark particle larger than 0.007”) from apple sauce • System must be able to handle 12 metric tons per 8 hour shift • System must remove 95% of visible defects Memorial University of Newfoundland

17. High Speed, Line Scan Based Inspection System • 2 distinct challenges: • Detection • Removal Memorial University of Newfoundland

18. High Speed, Line Scan Based Inspection System Memorial University of Newfoundland

19. High Speed, Line Scan Based Inspection System Memorial University of Newfoundland

20. Line Scan Based Detection System • Detection system based on a high performance line scan camera; 4096 pixels per line at 4800 lines per second. • Image acquisition and processing functions implemented on a Complex Programmable Logic Device (CPLD) as opposed to a microprocessor or Digital Signal Processor (DSP). • The objective is to implement image processing functions with dedicated logic gates (i.e. hardware) for real-time performance. Memorial University of Newfoundland

21. What are CPLDs? • Complex Programmable Logic Devices (CPLDs) are a class of programmable logic device that are commonly used to implement complex digital designs on a single integrated circuit. • Applications of CPLDs in the field of computer engineering include the implementation of bus controllers, address decoders, priority encoder and state machines Memorial University of Newfoundland

22. Line Scan Camera-Based Detection System Memorial University of Newfoundland

23. Line Scan Camera-Based Detection System Memorial University of Newfoundland

24. Line Scan Camera-Based Detection System Typical section of apple sauce recorded with an area scan camera Typical Particle Memorial University of Newfoundland

25. Line Scan Camera-Based Detection System Memorial University of Newfoundland

26. Line Scan Camera-Based Detection System Memorial University of Newfoundland

27. Removal System Memorial University of Newfoundland

28. Removal System Memorial University of Newfoundland

29. Removal System: Flow Characterization • Rheological nomenclature and associated velocity profiles for steady flow through tubes with circular cross section. Memorial University of Newfoundland

30. Viscosity Measurement Memorial University of Newfoundland

31. Viscosity Measurement Velocity profile can be characterized by power law! Memorial University of Newfoundland

32. Flow Profile of Apple Sauce Memorial University of Newfoundland

33. Particle Removal Window Memorial University of Newfoundland

34. System Timing Diagram Memorial University of Newfoundland

35. Aspiration Valve Characterization Memorial University of Newfoundland

36. Aspiration Valve Characterization Memorial University of Newfoundland

37. Particle Detection Rate Versus Flowrate Memorial University of Newfoundland

38. High Speed, Line Scan Based Inspection System: Current Status • Industrial partner is currently developing the production version of the system. • Packaging of the principle components (i.e., lenses, cameras, electronics, light sources) remains a major challenge given the environment. • One possible solution is to integrate all the electronics in the camera enclosure. • Partner is anxious to explore applications in the pulp and paper industry. Memorial University of Newfoundland

39. Vision Based Inspection of Liquid Crystal Display (LCD) modules • Objective: to automate the inspection of LCD modules in order to improve quality control • One step in the implementation of a Six-Sigma Program (“3.4 defects per million opportunities”) • The inspection must be completed within 30 seconds for 10 predetermined LCD patterns • System can “learn” new LCD modules without modifying software Memorial University of Newfoundland

40. Vision Based Inspection of LCD modules: System Components • Pulnix camera with macro lens • High frequency fluorescent light sources • Coreco Bandit integrated image acquisition and VGA accelerator • Software developed using with WiT graphical programming environment in combination with Microsoft VB Memorial University of Newfoundland

41. Vision Based Inspection of Liquid Crystal Display (LCD) modules Memorial University of Newfoundland

42. Vision Based Inspection of Liquid Crystal Display (LCD) modules Original Image Showing Error in Alignment Memorial University of Newfoundland

43. Vision Based Inspection of Liquid Crystal Display (LCD) modules Thresholding Operation – Image Subtraction with respect to an image with no segments illuminated Memorial University of Newfoundland

44. Vision Based Inspection of Liquid Crystal Display (LCD) modules Blob Analysis – Reference Points are Identified Memorial University of Newfoundland

45. Vision Based Inspection of Liquid Crystal Display (LCD) modules Image Rotation and Translation Memorial University of Newfoundland

46. Vision Based Inspection of Liquid Crystal Display (LCD) modules Pixel by Pixel Image Subtraction from Reference Image – Thinning Operator Memorial University of Newfoundland

47. Vision Based Inspection of Liquid Crystal Display (LCD) modules Blob Analysis Memorial University of Newfoundland

48. Vision Based Inspection of Liquid Crystal Display (LCD) modules: Conclusions • Inspection system was installed at BAE Systems Canada Ltd. where it was used to test between 200 and 600 LCD displays per day. • Number of defective modules that passed inspection was basically reduced to zero. • Occasional “false positives” proved to be technical problems with the devices that previously went unreported. • Applications for this technology are numerous given the number of LCD displays produced annually. Blob Analysis Memorial University of Newfoundland

49. CPLD based interfaces for smart sensors and actuators Memorial University of Newfoundland

50. Introduction • Several novel “mechatronic” applications for CPLDs are presented. • CPLD basically provides reconfigurable digital I/O that permits the implementation of interfaces for smart sensors and actuators. • Applications that will be discussed include: • Stepper motor controller • Quadrature decoder/counter for optical encoder • Pulse-Width Modulation (PWM) motor drives • By incorporating a CPLD that supports In-System Programmability (ISP) the target device can be reprogrammed by the user for a variety of applications without removing it from the host system. Memorial University of Newfoundland