1 / 57

Zhigang Zhu Visual Computing Laboratory Department of Computer Science

Explore the pioneering PRISM Center for enriched education blending robotics, vision, and multimedia in a pervasive computing environment. Learn about cutting-edge research, multimedia virtualized classrooms, automatic media capture, and intelligent integration for personalized presentations.

krogmane
Download Presentation

Zhigang Zhu Visual Computing Laboratory Department of Computer Science

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. C CVC L Pervasive Computing Experience in Planninga Center forPerceptual Robotics, Intelligent Sensors & Machines Zhigang Zhu Visual Computing Laboratory Department of Computer Science City College of the City University New York http://www-cs. ccny.cuny.edu/~zhu/ John (Jizhong) Xiao Robotics and Intelligent Systems Lab Department of Electrical Engineering City College of the City University New York Website: http://134.74.16.73

  2. Outline • Problems and Our Goal • Vision/Robotics Research • The planned PRISM Center • MVC: Multimedia Virtualized Classroom • Automatic multimedia capture (Audio, video, text…) • Intelligent media integration and indexing • User-customized presentation • RISE-NET: Robotized Intelligent Sensor Networks • Stereo mosaics: Airborne, ground & under-vehicle inspection • Wall-climbing robots /smart miniature robots • Summary Pervasive Computing

  3. Outline • Problems and Our Goal • Vision/Robotics Research • The planned PRISM Center • MVC: Multimedia Virtualized Classroom • Automatic multimedia capture (Audio, video, text…) • Intelligent media integration and indexing • User-customized presentation • RISE-NET: Robotized Intelligent Sensor Networks • Stereo mosaics: Airborne, ground & under-vehicle inspection • Wall-climbing robots /smart miniature robots • Summary Pervasive Computing

  4. IntroductionCCNY & PRISM Pervasive Computing

  5. CUNY System w/ Vision and Robotics CUNY City College Graduate Center Brooklyn Hunter Queens … Flagship campus HSI / MI (PhD granting institution*) SOE * Site of PhD programs in Engineering ME CS EE PRISM Center Planned Center for Perceptual Robotics, Intelligent Sensors and Machines under NSF MII planning program Pervasive Computing

  6. Vision & Robotics People at CCNY • Prof. Izidor Gertner • Image processing, data fusion, parallel processing • Prof. Michael Grossberg • Physics-based vision, machine learning • Prof. Esther Levin • Spoken language HCI, machine learning • Prof. Jie Wei • Vision, Multimedia Digital Library • Prof. George Wolberg • Vision, graphics • Prof. Jizhong (John) Xiao • Robotics, control • Prof. Zhigang Zhu • Vision, video computing, robotics, HCI Pervasive Computing

  7. PRISM: a Center for Vision and Robotics • NSF Planning Project (Grant No. CNS-0424593) • Planning a CCNY PRISM Center • PIs: Jizhong Xiao, Zhigang Zhu • Research Center of Excellence in • Distributed Perceptual Robotics (RISE-NET) • Intelligent Sensors (MM) • Smart Machines (HCI) • Education Environment • For PhD, MS, undergrad and high school students • Serving underrepresented students • CCNY- MI and HSI Pervasive Computing

  8. PRISM: a Center for Vision and Robotics • Infrastructure in Place or in Plan • Airborne and Indoor/Outdoor Mobile Platforms • UAVs, outdoor robots, indoor robots • Heterogeneous Sensor Packages (AFRL) • Miniature and high-end cameras, acoustic, LDV • GPS, INS , ladars, online processors • Fast Prototyping Machine (ARO) • For fabricating wall-climbing robots • Various Levels of Vision Processors (NSF MRI) • Compact, high-end, low-cost, low power • Wireless Sensor Networks (CCNY Samsung Lab) • Practical for imaging sensors and robots Pervasive Computing

  9. Outline • Problems and Our Goal • Vision/Robotics Research • The planned PRISM Center • MVC: Multimedia Virtualized Classroom • Automatic multimedia capture (Audio, video, text…) • Intelligent media integration and indexing • User-customized presentation • RISE-NET: Robotized Intelligent Sensor Networks • Stereo mosaics: Airborne, ground & under-vehicle inspection • Wall-climbing robots /smart miniature robots • Summary Pervasive Computing

  10. Multimedia Integration for a Virtualized Classroom • Collaborators • Rick Adrion, Ed Riseman, Al Hanson, Jim Kurose, (UMass) • Parviz Kermani, IBM / PINY • Student: Weihong Li, Hao Tang, Chad McKittrick • Supported by • CUNY CISDD, NSF EIA (UMass) Pervasive Computing

  11. Multimedia Virtualized Classroom • Goal: automating e-learning in MVC • Automatic capture (A, V, WB, BB) • Intelligent media integration • User-customized presentation • Problem and Promise • Joint proposals with UMass to NSF • MVC is a smart room, part of the effort in building a PRISM center for research and learning • Many good examples of advance MM classrooms • Georgia Tech, UIUC, UMass, … Pervasive Computing

  12. conversion index PPT Capture Add-on registration Mimio Virtual Ink video transformation Remote Reality Omnidirectional Camera MVC: Automatic MM capture Pervasive Computing

  13. video PPT slides Handwriting pages S V1 H V2 combined MVC: Media Integration

  14. MVC: Media Integration Panoramic mosaic from video Pervasive Computing

  15. MVC: Media Integration Panoramic mosaic from video Pervasive Computing

  16. Active WWW link Synthetic Hi-Res digital display Slide combined with video Panoramic mosaic from video Annotated and active instructor MVC: Media Integration Toward an immersive VC interface Question Window, ToC, etc Pervasive Computing

  17. MVC: Media Integration Instructor Extraction…and integration video image shadow contour Pervasive Computing

  18. User Customized Presentation • Virtualized Classroom Presentation System (VCPS) in Java, consists of • An authoring tool –VCPS creator • An presentation interface - VCPS player • Features • Customized sizes and locations • Pop up ToC for indexing • Synchronized play, next, last… Pervasive Computing

  19. User Customized Presentation • PPT+ panoramic view + popup ToC Pervasive Computing

  20. User Customized Presentation • PPT+ Mimio + Combined + images +New Buttons Pervasive Computing

  21. Outline • Problems and Our Goal • Vision/Robotics Research • The planned PRISM Center • MVC: Multimedia Virtualized Classroom • Automatic multimedia capture (Audio, video, text…) • Intelligent media integration and indexing • User-customized presentation • RISE-NET: Robotized Intelligent Sensor Networks • Stereo mosaics: Airborne, ground & under-vehicle inspection • Wall-climbing robots /smart miniature robots • Summary Pervasive Computing

  22. RISE-NET: Robotized Intelligent Sensor Network in 3Dfor surveillance and security • Joint Work with • George Wolberg, John Xiao @ CCNY • Bob Haralick @ CUNY Graduate Center • Ed Riseman, Al Hanson, Rod Grupen @ UMass • Tom Huang @ UIUC • Supported by NSF, DARPA,AFRL, ARO, NYSIA, ACTI, etc Pervasive Computing

  23. Goal • Dream: • transform the present 2-D world of mobile rovers into a new 3-D universe –air, ground, under-object, and wall-climbing • move on ground, climb walls, walk on ceilings, transit between surfaces. • Applications: • Urban warfare applications: surveillance and reconnaissance, weapon delivery, guiding perimeter around a building, etc • Security and counter-terrorist applications: intelligence gathering about a hostile situation within a building, etc. • Inspection and maintenance applications: routine inspection of buildings, nuclear containment domes, and other hard-to-reach places, inspection of aircraft, sand blasting of ship hulls, etc. • Other Civilian applications: assistance in firefighting, search and rescue operations, etc. Pervasive Computing

  24. Some Real Examples • City/Urban modeling (NSF/AFRL/NYSIA) - airborne video: far-view (1000 ft) • Robot Navigation (DARPA, ARO)- ground video: medium range (~100ft) • Facade inspection (ARO) – climbing robots: near-view (inches?) • Under-vehicle inspection (ACTI) - car drives over camera: near-view (< 8 in) Pervasive Computing

  25. Objectives • Input: 2D array of cameras • multiple viewpoints; motion parallax • Output: Content-based representation • WFOV, Stereo, Occlusion Rep. -> 3D and Moving targets Pervasive Computing

  26. Airborne Stereo Mosaics:What is the Goal? • Joint Work with • George Wolberg, Robert Haralick, Hao Tang (CUNY) • Allen Hanson, Edward Riseman (UMass) • Dr. Jeff Layne (AFRL) • Supported byAFRL, NSF and NYSIA • Publications: ICCV01, VideoReg03, PAMI04, A3DISS05 Pervasive Computing

  27. Objectives • Rapidly create large FOV image mosaics that are • geo-referenced, • with 3D (stereo) viewing and • with all moving targets identified • could be dynamically updated • As a light UAV flights over an area Pervasive Computing

  28. Objectives • Rapidly create large FOV image mosaics that are • geo-referenced, • with 3D (stereo) viewing and • with all dynamic targets identified • As a light UAV flights over an area Pervasive Computing

  29. Region-Based Stereo Matching Left mosaic – Full Depth Map Pervasive Computing

  30. Region-Based Stereo Matching • under development… Left mosaic – closeup Pervasive Computing

  31. Region-Based Stereo Matching • under development… Left mosaic – Depth and Motion Pervasive Computing

  32. Objectives • Rapidly create large FOV image mosaics that are • geo-referenced, • with 3D (stereo) viewing and • with all dynamic targets identified • As a light UAV flights over an area Pervasive Computing

  33. Ground 3D Panoramas:autonomous vehicle navigation • Joint Work with • Allen Hanson, Edward Riseman (UMass) • Roderic Grupen (UMass) • Supported by DARPA • Publications : CVPR99, ICCV01, CVIU04, IJCV05 Pervasive Computing

  34. An Indoor Example • Video under translation • Varying depths • Wide FOV stereo mosaics and • Mosaic-based virtual walk-through Pervasive Computing

  35. Under-Vehicle Stereo Mosaics:3D vehicle inspection • Joint Work with • Allen Hanson, Edward Riseman,Howard Schultz(UMass) • P. Dickson, J. Li (UMass), Gary Whitten ( ACTI) • Supported by ACT • Publications : WACV02, AIPR03, ICIP05 Pervasive Computing

  36. Camera Geometry • Car drives over a 1D array of cameras… Pervasive Computing

  37. One camera to N stereo mosaics • Very near range view • Distortion removal • Motion estimation • Stereo mosaicing • anomaly detection Pervasive Computing

  38. UVIS: Dynamic Stereo Mosaics Anti-Terrorism: Under-Vehicle Inspection 5 mosaics with changing viewing directions Stereo mosaics: Each pair of 5 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Pervasive Computing

  39. UVIS: Dynamic Stereo Mosaics Anti-Terrorism: Under-Vehicle Inspection 5 mosaics with changing viewing directions Stereo mosaics: Each pair of 5 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Pervasive Computing

  40. UVIS: Dynamic Stereo Mosaics Anti-Terrorism: Under-Vehicle Inspection 5 mosaics with changing viewing directions Stereo mosaics: Each pair of 5 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Pervasive Computing

  41. UVIS: Dynamic Stereo Mosaics Anti-Terrorism: Under-Vehicle Inspection 5 mosaics with changing viewing directions Stereo mosaics: Each pair of 5 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Pervasive Computing

  42. UVIS: Dynamic Stereo Mosaics Anti-Terrorism: Under-Vehicle Inspection 5 mosaics with changing viewing directions Stereo mosaics: Each pair of 5 mosaics is a stereo pair Dynamic mosaics: Look around objects even without stereo Pervasive Computing

  43. Sensor Network in 3Dfor wall-climbing robots • Joint Work with • John Xiao (EE @ CCNY), Ali Sadegh (ME @ CCNY) • Yi Feng (GC @ CUNY) • Supported by NSF, ARO • Related Publications: OmniVision00, CVIU04, RAM04 Pervasive Computing

  44. Existing Technologies and Robots CMU gecko inspired climber • Existing wall-climbers: JPL-Stanford rock climber MSU “Flipper” & “Crawler” Avionic Instruments Inc. Vortex attraction technique iRobot’s Mecho-Gecko Pervasive Computing

  45. Wall Climber: Adhesive Mechanism • Design alternatives: • vacuum pumps (MSU climber) • vortex attraction device • vacuum rotor package Pervasive Computing

  46. Wall Climber: Vacuum Chamber Seal • Inflated Tube Skirt Seal • Flexible Bristle Skirt Seal attraction force is so strong that it anchored the device to wall surfaces trade-off between sealing and mobility Pervasive Computing

  47. Wall Climber: Selected Design • Selected Design • vacuum rotor package • flexible bristle skirt seal • differential drive • pressure force isolation rim (re-foam) • improves mobility, & enhances sealing by reducing the deformation of the skirt Pervasive Computing

  48. Wall Climber: Transition Mechanism • Transition Mechanism • Modular Design Four wall-climber modules are configured to form a larger wall-climbing robot which can carry heavy payload Pervasive Computing

  49. Vision for Cooperative Climbing Robots • View planning in 3D • with robots on ground, walls and ceilings • Scene and object understanding • to recognize floor, walls, obstacles, ceilings • by taking advantage the climbing feature of multiple wall-climbers in 3-D environments Pervasive Computing

  50. Robot Localization and View Planning • Mutual awareness • Heterogeneous sensors • Active Sensing • Robust Features • for tracking and matching • SIFT Pervasive Computing

More Related