1 / 42

A Framework for Multi-resolution and Multi-touch Systems

A Framework for Multi-resolution and Multi-touch Systems . 多重解析度及多重觸控互動系統之開發. Presenter: 胡婷婷. Advisor: 洪一平 教授. Outline. Introduction Related Work i -m- Top System i -m- Top SDK Application Conclusion & Future Work. Outline. Introduction Related Work i -m- Top System

hisa
Download Presentation

A Framework for Multi-resolution and Multi-touch Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Framework for Multi-resolution and Multi-touch Systems 多重解析度及多重觸控互動系統之開發 Presenter: 胡婷婷 Advisor: 洪一平 教授

  2. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Application • Conclusion & Future Work

  3. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Application • Conclusion & Future Work

  4. Introduction • Motivation • Multi-resolution • Variable-resolution nature of human vision • Cost-effective large high-res display • Multi-touch • Intuitive interaction • Computer-vision based approach • Software Toolkit • Hide engineering challenges

  5. Introduction • i-m-Top • interactive multi-resolution tableTop • interactive: multi-touch • multi-resolution: fovea + peripheral projectors • i-m-Top SDK • Toolkit for rapid prototyping applications • Featured on multi-resolution/multi-touch

  6. Introduction • Overview Cams Image Processing Applications i-m-Top SDK Projectors Touch Detection Pan-tilt Unit Multimedia Other sensors User Interface

  7. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Application • Conclusion & Future Work

  8. Related Work • Multi-Resolution Display • Fovea Plus Context • LCD screen (fixed) + projector Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01

  9. Related Work • Multi-Resolution Display • Fovea-Tablett • Tablets as high-res displays • Tracked with visual marker • Multiple high-res is allowed • Cons: physical boundary J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea-tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007

  10. Related Work • Software Development Toolkit • DiamondTouch SDK (C++) • Multi-touch, multi-user Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002

  11. Related Work • Software Development Toolkit • DiamondSpin(Java): • Around-the-table interaction • Real-time polar to Cartesian transformation rotatable circular tabletop public + personal work areas Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004

  12. Related Work • Software Development Toolkit • T3 (Java) • Multiple projectors → single high-res display • Mixed-presence collaboration Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.

  13. Outline • Introduction • Related Work • i-m-Top System • Hardware Configuration • Touch Detection • i-m-Top SDK • Application • Conclusion & Future Work

  14. i-m-Top System 96 ppi 24 ppi 56” Mirror Pan-tilt Unit Fovea Projector IR Camera Peripheral Projector IR Illuminator

  15. i-m-Top System

  16. i-m-Top System • Touch Detection finger palm Opening Background Subtraction Connected Component PCA Analysis Palm-finger Association Finger Fusion Finger Tracking - - Background Subtraction Connected Component PCA Analysis Palm Association Opening Opening Down- scale Connected Component Palm Tracking -

  17. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Design Principles • Architecture • Implementation • Application • Conclusion & Future Work

  18. i-m-Top SDK • Design Principles • Efficient prototyping • Hide engineering details • Use metaphors • Provide reusable components • Touch-detection module independent • Vision-based & electronic-based approaches • Real-time requirement • Touch event handling + UI rendering • Extensibility • Integrate with other components

  19. i-m-Top SDK • Architecture Virtual Director Scene Scene Scene Scene Node Node View View Node Node Viewport / Virtual Camera Global Coordinate Position / Textures / Event Handlers / Animation Effects Node Virtual Camera (Fovea View) Node Virtual Camera (Peripheral View) Local Coordinate

  20. i-m-Top SDK • Architecture Application Projector Virtual Director Touch Detection Projector Message Parser Scene Device Control Socket Event queue Node View PTU Window Message Speaker Node Cam Other Sensors COM LED Light OpenGL c#

  21. i-m-Top SDK • Loop of virtual director Time spent 0 Affiliated Timer Functions Check Inputs Update States T Render Scenes Multimedia Output 1/fps-T Sleep 1/fps

  22. i-m-Top SDK • Sample Code • A scene with a photo • InitScenes( ){ • oScene scene = newoScene(); //construct a scene • director.AddScene(scene); //add scene to director • oView view = newView(0, 0, 1280, 720); //construct a view • scene.AddView(view); //add view to scene • oPhoto photo = newoPhoto(0,0,-100,640,480,”sun.png”); //construct a photo • scene.AddNode(photo); //add to scene • }

  23. i-m-Top SDK • Implementation • Multi-resolution Handling • Touch-event Processing • Reusable Components

  24. Implementation • Multi-resolution Handling • Calibration • Homography of fovea & peripheral projections • Run-time warping • Masking • Steerable fovea projection • Region/Object oriented • Multi-presented Information

  25. Implementation • Calibration • Homography of fovea & peripheral projections • Provide tools for manual calibration • Save homography/ center point /pan-tilt of PTU P2 P1 V1 V2 Y Y F1 F1 F2 F2 V1 V2 X V1 V2 X V3 V4 V3 V4 F3 F3 F4 F4 V3 V4 P3 P4

  26. Implementation • Calibration • Run-time warping • Adapt OpenGL rendering pipeline Warping Homography Matrix Model-View Matrix Projection Matrix Perspective Division Viewport Transformation Local coordinate Global coordinate Eye coordinate Clipping coordinate Window coordinate Normalized device coordinate

  27. Implementation • Masking fovea peripheral fovea

  28. Implementation • Masking • Find maximum region in fovea • Utilize Stencil buffer in OpenGL Apply arbitrary masks Set ‘0’ in mask regions Invert pixels in fovea region Set ‘1’ in the fovea region

  29. Implementation • Steerable Fovea Projection • Region-oriented • Spotlight on a specific region • Steered by finger gestures • Object-oriented • Spotlight on a specific object • Move with object • Real-time update • Steer the mirror • Retrieve homography • Apply mask

  30. Implementation • Multi-presented Information • Different content in fovea and peripheral regions • Eg. Google Map • Automatically change oRectanglemulti_presentation =newoRectangle(0, 0,-100, 200, 200); multi_presentation.LoadTexture(“context.png");//load low-resolution image multi_presentation.LoadHighTexture(“detail.png");//load high-resolution image

  31. Implementation • Touch-event Processing Touch events Touch Detection i-m-Top SDK Message Parser Virtual Director Node Receive Pass Check Dispatch Handle (by event handler)

  32. Implementation • Touch-event Processing • Receive & parse • Run on background threads • Pass to director • Gather all events in one detection frame before pass • if rendering rate < detection rate • Keep vital events: ‘Up’, ‘Down’ • Discard other events: ‘Move’ i-m-Top SDK Message Parser Virtual Director Node Pass Receive

  33. Implementation • Touch-event Processing • Check • ‘Down’: Test if hit some node • By SELECTION mode in OpenGL • If yes: add to TargetTable (event id → node) • ‘Move’, ‘Up’: Check if there is a target • Hit TargetTable by event id • ‘Up’: Remove from TargetTable • Dispatch • Dispatch events to each node i-m-Top SDK Message Parser Virtual Director Node Dispatch Check

  34. Implementation • Touch-event Processing • Define event handler • Node decides how to act • Default handler • Move/Scale/Rotate under ‘Move’ events • Redefine handler • InitScenes( ){ • //…… • //add new event handlers to the photo object • photo.MultiFingerEvent += newMultiFingerHandler(MyMultiFingerHandler); • photo.FingerClick += newFingerClickHandler(MyFingerClickHandler); • } i-m-Top SDK Message Parser Virtual Director Node Handle

  35. Implementation P3 P3 P1 P2 P4 P4 Translation: T Scaling: S P3 P3 P1 P34 Rotation: R P12 P2 P4 P4

  36. Implementation • Reusable components • Multimedia • Video/audio • Camera capture • UI components • Virtual keyboard • Text-pad • Button, scrollbar…

  37. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Application • Conclusion & Future Work

  38. Application • Map Browser • Different content in the fovea & peripheral • Photo Browser • Object/Region oriented fovea projection • Emotion Player • Interactive art c

  39. Outline • Introduction • Related Work • i-m-Top System • i-m-Top SDK • Application • Conclusion & Future Work

  40. Conclusion • i-m-Top • Multi-resolution • Accommodating to human vision • Cost-effective large high-res display • Multi-touch • Intuitive interaction • i-m-Top SDK • Hide engineering challenges • Rapid prototyping applications • Investigating new interfaces / interactions

  41. Future Work • Attentive computing • Combine with gaze-tracking • 3D interaction • Surface to space • Auto calibration • Vision-based • Continuous pan-tilt / homographymap

  42. Thanks for your listening

More Related