EEC-492/592 Kinect Application Development

1. EEC-492/592Kinect Application Development Lecture 7 Wenbing Zhao wenbing@ieee.org

2. Outline Depth data stream How depth measurement is done Capture and process depth data Classes related to depth data Getting the Distance from a Particular Pixel

3. Depth Data Stream • Depth data stream consists of a sequence of depth image frames • Each depth image frame is in the form of a 16-bit grey scale image • Viewable range: 43 degrees vertical, 57 degrees horizontal • A depth pixel contains the following • (x,y) coordinates represent the location of the object in the depth sensor view • The value of the pixel at (x,y) contains the distance between the object and the Kinect sensor in millimeters

4. Depth Data Stream • Depth data stream supports • 640x480 at 30fps • 320x240 at 30fps • 80x60 at 30fps • Kinect depth sensing range • Up to 13.1 feet (4 meters) • As close as 40cm

5. How the Depth Data is Obtained

6. Stereo Triangulation • Stereo triangulation is an analysis algorithm for computing the 3D position of points in an image frame • In general stereo triangulation, two images are used to obtain the two different views on a scene, in a similar fashion to human binocular vision. • By comparing these two images, the relative depth information is calculated • Kinect stereo triangulation uses one real (IR) image and a virtual image (more accurately, the line of each projected point) • http://www.youtube.com/watch?v=uq9SEJxZiUg

7. Depth Measurement • When there is a pixel in any of the diagonal views of the sensor, it internally draws a line that is perpendicular to the sensor and then calculates the distance directly from there

8. Capturing and Processing Depth Data • Enable the depth stream channel with the type of depth image format • Attach the event handler to the stream channel • Process the incoming depth frames • Render the frames on UI this.sensor = KinectSensor.KinectSensors[0]; this.sensor.DepthStream.Enable(); sensor.DepthFrameReady += depthFrameReady); void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { }

9. Processing Depth Data void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) { return; } imageFrame.CopyPixelDataTo(depthPixels); // Write the pixel data into our bitmap this.depthBitmap.WritePixels( new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight), this.depthPixels, stride, 0); } } Very similar to the code for ColorImage Processing: color => depth

10. Classes Related to Depth Data

11. Classes Related to Depth Data

12. Depth Data and Distance • The DepthImageStream and DepthImageFrame classes have the MaxDepth and MinDepth properties, which return the maximum and minimum depth ranges for that particular stream or captured image frame, in millimeters • This range value returns the best range where distance can be measured properly • These values will change automatically based on the selection of DepthRange for the stream

13. Depth Range • The Range property sets the viewable range for the Kinect sensor, which is a type of DepthRange enumeration • A DepthRange enumeration has the following two values: • Default • Near sensor.DepthStream.Range = DepthRange.Near;

14. Special Depth Range Values • DepthImageStream defines three additional read-only properties • TooFarDepth: It provides the depth point where the object is beyond the range of the sensor • TooNearDepth: It provides the depth point where the object is too close to the sensor • UnknownDepth: There are instances when a depth is completely indeterminate, and the value will be zero

15. Default Depth Range

16. Depth Data Representation • The Kinect sensor returns 16-bit raw depth frame data • The first three bits are used to represent the identified players and the remaining 13 bits give you the measured distance in millimeters • To retrieve the 13-bit depth data, perform a bitwise shift operation >> to move the bits to the beginning int depth = depthFrame[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;

17. Depth Data Representation

18. Getting the Distance from a Particular Pixel • Add feature to our DepthCam app • Very important: make the image control exactly the same size as the resolution chosen • Image should be 320x240 if Resolution320x240Fps30 • Image should be 640x480 if Resolution640x480Fps30 • Use mouse to select a pixel in the image => display the depth for that pixel • Need to add event handler for MouseDown Point currentPoint = e.GetPosition(depthImageControl); int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frame.Width)); int distancemm = this.pixelData[pixelIndex] >> DepthImageFrame. PlayerIndexBitmaskWidth;

19. Build the DepthCam App • Create a new WFP project named DepthCam • Draw the GUI for the app • Modify xaml file to add WindowLoaded and WindowClosing as before • Add Microsoft.Kinect reference • Add member variables for displaying depth images • Start Kinect sensor, enable depth data stream, register depth stream handler, connect image control to the bitmap for display • Process each depth image • Add code for displaying depth data for pixel clicked

20. GUI Design • Image control • Group control for depth information • Two sets of labels and two sets of textboxes • Use meaningful names for the textboxes • Group control for pixel information • Four sets of labels and four sets of textboxes

21. Adding Code KinectSensor sensor; private WriteableBitmap depthBitmap; private short[] depthPixels; private int frameWidth; // to calculate depthIndex • Add member variables: • WindowLoade method (WindowClosing() same as before): private void WindowLoaded(object sender, RoutedEventArgs e) { if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30); this.depthPixels = new short[this.sensor.DepthStream.FramePixelDataLength]; this.depthBitmap = new WriteableBitmap(this.sensor.DepthStream.FrameWidth, this.sensor.DepthStream.FrameHeight, 96.0, 96.0, PixelFormats.Gray16, null); this.image1.Source = this.depthBitmap; this.sensor.DepthFrameReady += this.depthFrameReady; this.sensor.Start(); } else { MessageBox.Show("No device is connected!"); this.Close(); } } }

22. Adding Code void depthFrameReady(object sender, DepthImageFrameReadyEventArgs e) { using (DepthImageFrame imageFrame = e.OpenDepthImageFrame()) { if (null == imageFrame) { return; } this.frameWidth = imageFrame.Width; this.maxDepthField.Text = "" + imageFrame.MaxDepth; this.minDepthField.Text = "" + imageFrame.MinDepth; imageFrame.CopyPixelDataTo(depthPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel; this.depthBitmap.WritePixels( new Int32Rect(0, 0, this.depthBitmap.PixelWidth, this.depthBitmap.PixelHeight), this.depthPixels, stride, 0); } } • Event handler for depth frames:

23. Adding Code • Add mouse down event on image control • Find the MouseDown line, double click it to get event handler template • Event handler for mouse click: private void image1_MouseDown(object sender, MouseButtonEventArgs e) { Point currentPoint = e.GetPosition(image1); this.pixelXField.Text = currentPoint.X.ToString(); this.pixelYField.Text = currentPoint.Y.ToString(); int pixelIndex = (int)(currentPoint.X + ((int)currentPoint.Y * this.frameWidth)); this.depthIndexField.Text = "" + pixelIndex; int distancemm = this.depthPixels[pixelIndex] >> depthImageFrame.PlayerIndexBitmaskWidth; this.depthField.Text = "" + distancemm; }