| ntelligent Space with Time Sensitive Applications

1. |ntelligent Space with Time Sensitive Applications Wai-Lun Danny Leung, Rangsarit Vanijjirattikhan, Zheng Li, Le Xu, Tyler Richards, Rachana A Gupta and Bulent Ayhan, Student Member, IEEE and Mo-Yuen Chow, Senior Member, IEEE Advanced Diagnosis, Automation and Control (ADAC) Lab Department of Electrical and Computer Engineering North Carolina State University Raleigh, NC 27695-7911, USA http://www.adac.ncsu.edu adac_lab@ncsu.edu

2. Overview of presentation • Introduction to Intelligent Space (iSpace) • iSpace @ ADAC, NCSU realization and development • Flow diagram • iSpace settings • Software • Hardware • Communication network • Graphic User Interface (GUI) • Current research projects related to iSpace ADAC, NC State University

3. What is Intelligent Space • A new concept to effectively use distributed sensors, actuators, robots, computing processors, and information technology over a physically and/or virtually connected space. For examples, a room, a corridor, a hospital, an office, or a planet. • It fuses global information within the space of interest to make intelligent operation decision such as how to move a mobile robot effectively from one location to another. Human-machine interaction in iSpace ADAC, NC State University

4. Examples of iSpace Space Tele-operation (Hubble telescope) Enterprise Main Computer Futuristic Nursing Homes ADAC, NC State University

5. Current research in iSpace • Research topics by other universities • Intelligent navigation and manufacturing (Lab for Comput. Sci., MIT) • Intelligent room (Artificial Intelligent Lab, MIT) • Distributed vision sensor fusion (Hashimoto Lab, U. of Tokyo) • Our work at ADAC • iSpace is A large Scale Network based Mechatronics system. • To realize and develop iSpace infrastructure to investigate research for time sensitive distributed network-based control, Real-time applications, tele-operation. ADAC, NC State University

6. Johnny6 plays fetch in iSpace • Prototyping project – Johnny6 plays fetch in iSpace • Demonstrate how iSpace makes superior decisions based on global information from distributed sensors • Game rules • The space with obstacles is continuously monitored by a webcam. • Johnny6 (an Unmanned Ground Vehicle, UGV) is to be commanded to go from its current location to a specific destination chosen by a user in a remote computing interface as quickly as possible while avoiding collisions. Johnny5 ADAC, NC State University

7. Structure of iSpace at ADAC • Hardware • Sensors • Webcam • Actuators • Motors on Johnny6 (UGV) • Computer Network (IP) • Wired and wireless connection • Remote computer controller • Software (main network controller) • Image acquisition • Image processing • Path generation • Path tracking controller • Graphic User Interface (GUI) • Any remote Computing interface in the world (Internet) ADAC, NC State University

8. Template Matching • Image acquisition • Top view • Image processing • Image Thresholding – Black and white • Template matching • Circles used for rotation invariance fB – Template image ADAC, NC State University

9. Path generation • Find a path from the starting point A to the end point B for the UGV • The path of the UGV should be as short as possible (minimize time) • The path of the UGV should not collide with any obstructions • Fast Marching Method (by J.A. Sethian, Dept. of Mathematics, UC Berkeley) • A numerical technique that counts the shortest distance from a point to the original point with a shortest distance update algorithm ADAC, NC State University

10. Path tracking • The path tracking algorithm runs in every control loop and adjusts the speed and turn rate of the UGV to track the generated path. • Calculate the closest point on the generated path from the current UGV position. • Pick a reference point on the generated path that is a set distance in front of the UGV • Calculate the speed and turn rate for the UGV to reach the reference point given its current position and orientation. ADAC, NC State University

11. Johnny6 communication interface • UGV receives speed and turn rate information from control software via 802.11b wireless network channel. • Speed and turn rate are converted to left and right linear wheel velocities on the UGV and applied to the motors. • As the UGV moves, a new image is captured by the webcam to perform the next calculation of speed and turn rate using the software. ADAC, NC State University

12. Hardware on Johnny6 • Main Components of UGV (Johnny6) • PC104 • Pentium 266MHz, 64MB SDRAM • 4MB Flash Array • 10GB External Hard-drive • 802.11b wireless module • 5V DC • Motor • 9V DC • Rated Torque=600g-cm • 165mA no load current • 415mA at 600g-cm load • Insulation R: 10M Ohm • Interface Board • 5V DC • Uses latches and logics to perform PWM • Communicate with PC104 via parallel port • Power Supply • 3 batteries • 3 voltage regulation circuits (5V,5V, 9V) Johnny6 ADAC, NC State University

13. Hardware on Ana - Next generation UV • Main Components of UGV (Ana) • PIC18F452 on board (Ana) • 4MB Flash Array • 10GB External Hard-drive • 5V DC • Motor • 5V Servo motor • Rated Torque=600g-cm • ER25 wireless module • PIC18F452 for wireless card • 802.11b wireless module • Ethernet and Serial port interface • Communicate with PIC18f452 on Ana using i2c protocol • Power Supply • Single 12 V battery • 2 voltage regulation circuits (5V for microcontroller and motor and 9 V for the wireless module) ADAC, NC State University

14. A new path planning method ADAC, NC State University

15. Laplacian of Gaussian 2G Zero Crossing Image I Edge E Threshold (C) Gradient of Gaussian G Magnitude Edge Detection • The zero crossing contours from Laplacian of Gaussian operator are taken as the candidate edges. The filtering adopted here is based on the strength of the edge, i.e. its contrast. • An approximated Gaussian first derivative gradient operator is used to produce an estimate of the contrasts. If the contrast of an edge is above a certain threshold, the edge is accepted as authentic; otherwise it is rejected as noise. ADAC, NC State University

16. Goal point chosen, shown as the small red dot A Model Based 2-D HPF • 2is the Laplace operator,  is the workspace of the UGV (2),  is the boundary of the obstacles (output of the edge detection stage), and is (xT, yT) the target point. The obstacles were represented by the repelling force and the point of destination was represented by the attractive force. • The potential function in closed contour of  will converge to a constant potential • The obstacle free path to the target is generated by traversing the negative gradient() i.e. . The normalized gradient at each point represents the directional guidance at that point in the workspace. ADAC, NC State University

17. Results with HPF Actual workspace images with different backgrounds, different obstacle shapes and sizes. Red line is actual path and white line is the ideal path. ADAC, NC State University

18. Main controller GUI • Remote user interface • Display window • Status display • Action buttons • Display options • Plot selections • Manual Commands ADAC, NC State University

19. Time delay issue • Capabilities: • Hard real-time data collection • Image acquisition time • Image processing time • Network delay between remote user interface and the UGV • Network-based control system ADAC, NC State University

20. Network delay and Processing delay Round-trip time between the remote user interface and the UGV over a wireless link. The mean is 0.129 second, the median is 0.01 second Actual network delay Histogram of network delay • Time for a image to be captured and saved in the hard disk • The average time for image acquisition ~ 0.2 seconds • Time required to extract position and orientation information of the objects and UGV • Non real-time processing takes ~1.2 seconds (initial) • Hard real-time processing takes ~0.02 seconds (remaining) ADAC, NC State University

21. Network-based control and GSM Research Challenges • Time delay alleviation • Path tracking control algorithms • Remote wireless control via Internet • Network delay is a critical issue in network-based control, as shown in the results of the iSpace project • Gain Scheduler Middleware (GSM) is a technology that allows the communication network to be transparent to controller and remote system, alleviating the adverse effects of network delay • GSM enables a conventional controller for network-based control purpose ADAC, NC State University

22. GSM results ADAC, NC State University

23. Conclusion • Introduction to Intelligent Space (iSpace) • iSpace at ADAC realization and development • iSpace structure (A large Scale Mechatronics System) • Software • Image acquisition and processing • Path generation and tracking • Hardware • Communication network • Main Controller GUI • Infrastructure for research • Network-based control system • Hard real time data Analysis • Path planning and Path tracking • Experimental results • Future research • Network delay alleviation, Path planning, Security ADAC, NC State University

24. ADAC Members http://www.adac.ncsu.edu adac_lab@ncsu.edu ADAC, NC State University

25. ADAC, NC State University