Make my desktop app great with touch, mouse, and pen

2. Make my desktop app great with touch, mouse, and pen Nick Waggoner Senior Program Manager 4-022

3. Demo: Photo annotation

4. Agenda • Input and hit-testing • Windows touch language • Panning and zooming • Pen and ink

5. Where we are today • Deep and mature • Diverse ecosystem • Primarily mouse and keyboard

6. PC Form Factorshave evolved.

7. Key platform APIs Desktop Applications Win32 apps Input and Interaction Rendering and composition Input and interaction DirectManipulation DirectComposition DirectManipulation Remainder of Win32 InteractionContext Direct2D, DirectWrite Realtime Stylus Ink Recognition Dynamic DPI InteractionContext Realtime stylus Ink recognition WM_POINTER Family Direct3D WM_Pointer family Windows kernel services

8. Input and hit-testing

9. WM_POINTER • Unified messages for touch, mouse, and pen. • EnableMouseInPointer(…) • GetPointerInfo(…) / GetPointerInfoHistory(…) • GetPointerTouchInfo(…) / GetPointerPenInfo(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

10. Touch hit testing Control C Button 1 • RegisterTouchHitTestingWindow(…) • WM_TOUCHHITTESTING • EvaluateProximityToRect(…) or EvaluateProximityToPolygon(…) • PackTouchHitTestingProximityEvaluation(…) Button 2 DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

11. Demo: Moving to WM_POINTER & WM_TOUCHHITTESTING

12. Windows touch language

13. Windows touch language DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

14. InteractionContext • Initialization • CreateInteractionContext(…) • SetInteractionConfigurationInteractionContext(…) • RegisterOutputCallbackInteractionContext(…) • Feeding input • AddPointerInteractionContext(…) • ProcessPointerFramesInteractionContext(…) • Handling output DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family voidINTERACTION_CONTEXT_OUTPUT_CALLBACK( _In_opt_  void *clientData, _In_      const INTERACTION_CONTEXT_OUTPUT *output );

15. InteractionContext – Output structures typedefstruct INTERACTION_CONTEXT_OUTPUT { INTERACTION_ID     interactionId; INTERACTION_FLAGS  interactionFlags; POINTER_INPUT_TYPE inputType; float              x; float              y; union { INTERACTION_ARGUMENTS_MANIPULATION manipulation; INTERACTION_ARGUMENTS_TAP          tap; INTERACTION_ARGUMENTS_CROSS_SLIDE  crossSlide; } arguments; } INTERACTION_CONTEXT_OUTPUT;

16. InteractionContext – Output structures typedefstruct INTERACTION_ARGUMENTS_MANIPULATION { MANIPULATION_TRANSFORM   delta; MANIPULATION_TRANSFORM   cumulative; MANIPULATION_VELOCITY    velocity; MANIPULATION_RAILS_STATE railsState; } INTERACTION_ARGUMENTS_MANIPULATION;

17. Demo: Integrating the touch language

18. Panning and zooming

19. DirectManipulation • Drives panning and zooming in the system • Slide and pinch • Encapsulates the personality • Inertia • Bounce effects • Snap-points DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

20. DirectManipulation • Pre-declare your behaviors/interactions DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family Harness the power of Windows with DirectComposition

21. DirectManipulation – Key interfaces/messages • Initialization • IDirectManipulationManager • IDirectManipulationViewport • IDirectManipulationContent • Hit-testing • SetContact() • WM_POINTERDOWN / DM_POINTERHITTEST • Manipulation begins • WM_POINTERCAPTURECHANGED DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

22. Adding panning and zooming

23. DirectManipulation - threading • Best performance by integrating DirectManipulation with a 3 thread model: • UI thread • Delegate thread • Independent hit-test thread • IDirectManipulationManager::RegisterHitTestTarget(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

24. Pen and ink

25. RealTimeStylus • Independent of the UI thread • Handwriting recognition DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

26. RealTimeStylus- Key interfaces/methods • Initialization • IRealTimeStylus • AddStylusSyncPlugin(…) • Processing Input • IStylusSyncPlugin • StylusDown(…), Packets(…), StylusUp(…) • IStrokeBuilder • BeginStroke(…), AppendPackets(…), EndStroke(…) • Handwriting Recognition • InkRecognizerContext • Recognize(…), BackgroundRecognize(…), • BackgroundRecognizeWithAlternates(…) DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

27. RealTimeStylus- Key interfaces/methods IRealTimeStylus DirectManipulation IStylusSyncPlugin InteractionContext Realtime stylus Ink recognition IStrokeBuilder InkRecognizerContext WM_Pointer family

28. Adding ink and recognition

29. Performance is critical for both inking as well as panning and zooming experiences.

30. Recap

31. Takeaways • Use WM_POINTER for touch, mouse, and pen. • Use the InteractionContext to incorporate the Windows touch language into your app. • Use DirectManipulation for great panning and zooming. • Differentiate your app by making ink pop with the RealTimeStylus APIs. DirectManipulation InteractionContext Realtime stylus Ink recognition WM_Pointer family

32. Resources • Related talks • 3-017 - Power up your desktop app with Windows • 4-021 - DirectComposition: Smooth composition and animation for desktop apps • 4-9147 – Exploring customized touch interactions for your Windows store app • Reference documentation • WM_POINTER - http://msdn.microsoft.com/en-us/library/windows/desktop/hh454916(v=vs.85).aspx • InteractionContext - http://msdn.microsoft.com/en-us/library/windows/desktop/hh437192(v=vs.85).aspx • DirectManipulation - http://msdn.microsoft.com/en-us/library/windows/desktop/hh446972(v=vs.85).aspx • RealTimeStylus - http://msdn.microsoft.com/en-us/library/windows/desktop/ms701683(v=vs.85).aspx • InkRecognizerContext - http://msdn.microsoft.com/en-us/library/windows/desktop/ms696371(v=vs.85).aspx • Process touch input with DirectManipulation sample • http://go.microsoft.com/fwlink/?LinkID=309852&clcid=0x409 • Design guidelines • Touch target sizes - http://msdn.microsoft.com/en-us/library/windows/apps/hh465415.aspx#touch_targets

33. Acer Iconia W3, Surface Pro, and Surface Type Cover Get your goodies Device distribution starts after sessions conclude today (approximately 6:00pm) in the Big Room, Hall D. If you choose not to pick up your devices tonight, distribution will continue for the duration of the conference at Registration in the North Lobby.

34. Required Slide *delete this box when your slide is finalized Your MS Tag will be inserted here during the final scrub. Evaluate this session • Scan this QR codeto evaluate this session and be automatically entered in a drawing to win a prize!