1 / 50

Connect with your

Connect with your. Hi I’m Justin Weinberg I’m a Carnegie Mellon Graduate Manager Consultant at Sogeti Founder of the North Texas Silverlight User Group. Wow!. 3D Depth Sensors. Microphone Array. Motorized Tilt. RGB Camera. That’s not USB?. You’ll need this. http ://openkinect.org/.

misae
Download Presentation

Connect with your

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Connect with your

  2. Hi I’m Justin Weinberg • I’m a Carnegie Mellon Graduate • Manager Consultant at Sogeti • Founder of the North Texas Silverlight User Group

  3. Wow!

  4. 3D Depth Sensors Microphone Array Motorized Tilt RGB Camera

  5. That’s not USB? You’ll need this..

  6. http://openkinect.org/ http://www.openni.org/ http://research.microsoft.com/kinectsdk/

  7. http://www.microsoft.com/express http://research.microsoft.com/kinect/

  8. Color Camera Streams Depth / Depth with Player Streams Skeletal Tracking Audio Functionality http://research.microsoft.com/kinect/

  9. Demo 0 Hello Tilt

  10. Kinect NUI Big Picture Runtime nui = newRuntime(); nui.VideoFrameReady += ... nui.VideoStream.Open(...) void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)    {                         //Process e.ImageFrame.   }

  11. ImageFrame.Image.Bits With camera data you might expect Byte[,] 1 3 2 4 6 7 8 5 12 9 10 11 What you actually get is Byte[]: 16 13 14 15 1 3 5 7 2 4 9 11 6 8 13 15 10 12 14 16

  12. BGR32 Format width in pixels 4 Bytes Per pixel Stride: Bytes Per Row Bytes per pixel * Width height in pixels Byte Array:

  13. Demo 1 Hello Colorful World

  14. Depth data 500mm 500mm 500mm 500mm 500mm 500mm 500mm 2500mm 2000mm 2000mm 2030mm 2010mm

  15. Depth data Max resolution for raw depth data is 640 x 480 2 bytes per pixel of depth data 11011110 11101010 00001110 00000101 00001010 11101010 pixel pixel pixel 11011110 00001010 distance(0,0) = byte0 | byte1 << 8

  16. Demo 2 Depth

  17. Depth + Player Data player 1 player 1 player 1 player 1 500mm 500mm 500mm 500mm player 1 player 1 player 1 500mm 500mm 500mm 2500mm 2000mm 2000mm 2030mm 2010mm

  18. Depth + Player Data Max resolution is 320 x 240 Still 2 bytes per pixel 11011001 00001011 But player is last 3 bits in first byte Player: byte0 & 7 00000111 11011001 00000001 & Depth(0,0) = byte0 >> 3 | byte1 << 5 11011 001 00001011

  19. Depth + Player Index 11 bits per pixel distance In mm distance In mm distance In mm distance In mm 11011001XXXX0111 player player player player 11011001XXXX0101 distance In mm distance In mm distance In mm distance In mm 11011001XXXX0111 player player player player distance In mm distance In mm distance In mm distance In mm 000 = No player 001 = Player 1 player 010 = Player 2 player player player

  20. Demo 3 Depth + Player Data

  21. Skeletal Tracking

  22. from skeleton inskeletonFrame.Skeletons whereskeleton.TrackingState == SkeletonTrackingState.PositionOnly skeletonFrame.Skeletons skeletonFrame.Skeletons whereskeleton.TrackingState == SkeletonTrackingState.Tracked

  23. hand_right hand_left head wrist_right wrist_left shoulder_center elbow_left elbow_right shoulder_left shoulder_right Skeleton.Joints[JointIdenum] spine hip_right hip_left hip center Joint.Position (X,Y,Z,W) Skeletons are mirrored X, Y, Z in left handed cartesian system +Z is out from sensor. W always returns 1 at this moment. knee_right knee_left ankle_right ankle_left foot_left foot_right

  24. Demo 3 Skeletal Tracking

  25. Demo 4 Putting Visuals Together

  26. Kinect Audio

  27. Kinect Audio API Beamforming and Sound Localization Features Microphone Array DMO Implementation

  28. Beam Forming

  29. Beam Forming 0.875 radians 50 degrees -0.875 radians -50 degrees .0175 radians 10 degrees

  30. MicArrayMode SoundSourcePosition BeamChanged Event SoundSourcePositionConfidence

  31. Accousting Echo Cancellation

  32. AEC • intEchoLength – Sets an adaptive filter to the AEC algorithm (in MS). • Recommended values are 128, 256, 512, 1024. For most offices and homes, use 256ms • boolCenterClip// After AEC, some echo residuals will remain. Setting to true removes. • boolNoiseFill// Clip removes residual audio but leaves gaps. This fills with noise if true • intFrameSize// Size of a frame in samples • Allowed values are : 80, 128, 160, 240, 256, 320. • System default for microphone array only is 320 • All other modes default is 160 • intSpeakerIndex – Device index of speaker apply AEC process to

  33. Noise Suppression

  34. Noise Supression • boolNoiseSuppression – True enables, false disables.

  35. Automatic Gain Control

  36. Automatic Gain Control • boolAutomaticGainControl– True enables, false disables.

  37. Demo 5 Recording Audio

  38. Audio @!%!#!! Gotchas Must run in an MTA AEC SystemModes crash if no speaker output. Use Microsoft.Speech not System.Speech DevicePairGuid / Qualitymetrics= crash Should run in ThreadMode Priority Highest Must plug Kinect into Wall

  39. Demo 6 Speech recognition

  40. Demo 7 Kinect Effect (Thanks SoulSolutions!)

  41. Creative Commons Acknowledgments • Flickr by Ian Lewis • Flickr by major nelson • Flickr by markodesign • Flickr by stephantom • Flickr by Abstract Machine • Flickr By Aixo.co.kr • Flickr By Andresmh • Flickr by Antony Bennison • Flickr by Ars Electronica • Flickr By bm.iphone • Flickr by campuspartymexico • Flickr by Chris Jagers • Flickr By ci_polla Paola Sucato • Flickr by creepyhalloweenimages • Flickr by cvander • Flickr by IMAL.org • Flickr by Institut Douglas • Flickr by Joshua and AmberFlickr by Logon Sakai • Flickr by openexhibits • Flickr by Paul_Clarke • Flickr by petyr.rahl • Flickr by pt • Flickr by Stilldavid • Flickr by y3rdua • Flickr by yesyesnono • Flickr by yum9me • Flickr by Jeff Howard • Flickr by jhscreenwm • Flickr by Kyle McDonald • Flickr by Morten Rand-Hendriksen • Flickr by Stuart Herbert • Flickr by Visiophone • Flickr by orkomedix • Michael Zöllner and Stephan Huber at the University of Konstanz • Openclip by J_Alves • Openclip by jetxee • Openclip by molumen • Openclip by shokunin • Openclip by Jhnri • Openclip by Machovka • Adafruit image of Hector Martin Thanks!!

More Related