Kinect Color Image Retrieval and Manipulation

1. EEC-492/592Kinect Application Development Lecture 5 Wenbing Zhao wenbing@ieee.org

2. Outline Important Classes on Color Image Retrieving/Setting Color Image Format Capturing Frames on Demand Frame Number Frame rate Saving frames Extend the KinectCam app with more controls

3. Important Classes on Color Image • We have dealt with two classes related to Kinect color images • ColorImageStream: handles a stream of ColorImageFrame • ColorImageFrame • ColorImageStream • this.sensor.ColorStream.Enable(); • ColorStream is of ColorImageStream type • ColorImageFrame • ColorImageFrame imageFrame = e.OpenColorImageFrame()

4. ColorImageStream Sealed class: no other class can inhert a sealed class

5. ColorImageFrame

6. More on ColorImageFrame

7. Retrieving/Setting Color Image Format • Can retrieve the color image format for the current image frame using the ImageFrame.Format property • public ColorImageFormat ImageFormat {get;set;} • Retrieve format • this.ImageFormat = imageFrame.Format; • Setting format • this.sensor.ColorStream.Enable(RgbResolution640x480Fps30);

8. Capturing Frames on Demand • Polling: request an image frame on demand • ColorImageStream::openNextFrame() • OpenNextFrame() accepts one parameter specifying how long the sensor should wait before it sends the image frame • If there is no frame ready within the provided time, the method will return null int millisecondsWait = 10; if (this.sensor.ColorStream.IsEnabled) { ColorImageFrame colorImageFrame = this.sensor.ColorStream. OpenNextFrame(millisecondsWait); }

9. Frame Number • Frame number: Every frame has a number to identify the frame • This number is incremented with each new frame generated • It is read only in image frame • int frameNumber=imageFrame.FrameNumber;

10. Frame Rate Calculation private int TotalFrames; private DateTime lastTime = DateTime.MaxValue; private int LastFrames; int currentFrameRate = 0; private int GetCurrentFrameRate() { ++this.TotalFrames; DateTime currentTime = DateTime.Now; var timeSpan = currentTime.Subtract(this.lastTime); if (this.lastTime == DateTime.MaxValue || // first time to run timeSpan >= TimeSpan.FromSeconds(1)) // average 1 second { currentFrameRate = (int)Math.Round((this.TotalFrames - this. LastFrames) / timeSpan.TotalSeconds); this.LastFrames = this.TotalFrames; this.lastTime = currentTime; } return currentFrameRate; }

11. Saving Color Image private void SaveImage() { using (FileStream fs = new FileStream(string.Format("{0}.jpg", Guid.NewGuid().ToString()), System.IO.FileMode.Create)) { BitmapSource imageSource = (BitmapSource)image1.Source; JpegBitmapEncoder jpegEncoder = new JpegBitmapEncoder(); jpegEncoder.Frames.Add(BitmapFrame.Create(imageSource)); jpegEncoder.Save(fs); fs.Close(); } }

12. Saving Color Images Periodically • Call the SaveImage () method on a Tick event of DispatcherTimer • Automatic periodic image saving. 4 steps. • Step1: Define the DispatcherTimer object under the System.Windows.Threading namespace: • private DispatcherTimer timer = new DispatcherTimer();

13. Saving Color Images Periodically • Step 2: Write the start method with an interval of 10 seconds and attach the Tick event handler: public void StartTimer() { this.timer.Interval = new TimeSpan(0, 0, 10); this.timer.Start(); this.timer.Tick += handleTickEvent; } // handleTickEvent: event handler delegate to be implemented

14. Saving Color Images Periodically • Step 3: On the Tick event handler, call the SaveImage() method: public void handleTickEvent(object sender, object e) { if (this.sensor.IsRunning && this.sensor.ColorStream.IsEnabled) { this.SaveImage(); } } • Step 4: To launch the timer thread, call timer.StartTimer();

15. Saving Color Images Periodically • To stop timer public void StopTimer() { this.timer.Stop(); this.timer.Tick -= this.handleTickEvent; }

16. Adjusting Elevation Angle • Kinect elevation angle can be changed between • MaxElevationAngle (27 degrees) • MinElevationAngle (-27 degrees) • Adjusting angle private void SetSensorAngle(int angleValue) { if(angleValue > sensor.MinElevationAngle || angleValue < sensor.MaxElevationAngle) { this.sensor.ElevationAngle = angleValue; } }

17. Build KinectCam2.0 • Extend the KinectCam we built last time • Revise GUI design • Adding image controls • Add code to handle image controls

18. Revised GUI

19. Settings • Changing frame format on the fly • RgbResolution640x480Fps30, enable this by default • RgbResolution1280x960Fps12 • Customize the checkbox names and the event handlers this time private void ChangeToHighResolution(object sender, RoutedEventArgs e) { if (this.sensor != null) { this.normalresolution.IsChecked = false; this.sensor.ColorStream.Enable( ColorImageFormat.RgbResolution1280x960Fps12); this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; } }

20. Settings • Changing frame format on the fly • Enable one, and disable the other • Need to reset the colorPixels array length for different formats private void ChangeToNormalResolution (object sender, RoutedEventArgs e) { if (this.sensor != null) { this.highresolution.IsChecked = false; this.sensor.ColorStream.Enable( ColorImageFormat.RgbResolution640x480Fps30); this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength]; } }

21. Settings • Displaying frame rate • Calculated and displayed for each frame received • At the end of void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) • textBox1.Text = ""+GetCurrentFrameRate(); • Changing elevation angle private void IncrementAngle(object sender, RoutedEventArgs e) { int angleValue = this.sensor.ElevationAngle + 1; if (angleValue < sensor.MaxElevationAngle) { this.sensor.ElevationAngle = angleValue; this.AngleBox.Text = "" + angleValue; } } private void DecrementAngle(object sender, RoutedEventArgs e) { int angleValue = this.sensor.ElevationAngle - 1; if (angleValue > sensor.MinElevationAngle) ……

22. Settings • Save image on demand (Save Image button) private void SaveImage(object sender, RoutedEventArgs e) { SaveImage(); } private void SaveImage() { using (FileStream fileStream = new FileStream(string.Format("{0}.jpg", Guid.NewGuid().ToString()), System.IO.FileMode.Create)) { BitmapSource imageSource = (BitmapSource)image1.Source; JpegBitmapEncoder jpegEncoder = new JpegBitmapEncoder(); jpegEncoder.Frames.Add(BitmapFrame.Create(imageSource)); jpegEncoder.Save(fileStream); fileStream.Close(); } }

23. Settings • Save image periodically (via checkbox) • Make sure you designate event handler for both checked and unchecked event!

24. Settings private void PeriodicSavingChanged (object sender, RoutedEventArgs e) { if (checkBox1.IsChecked == true) { StartTimer(); } else { this.timer.Stop(); } } private DispatcherTimer timer = new DispatcherTimer(); public void StartTimer() { this.timer.Interval = new TimeSpan(0, 0, 10); this.timer.Start(); this.timer.Tick += this.handleTickEvent; } public void StopTimer() { this.timer.Stop(); this.timer.Tick -= this.handleTickEvent; } public void handleTickEvent(object sender, object e) { if (this.sensor.IsRunning && this.sensor.ColorStream.IsEnabled) { this.SaveImage(); } } • Save image periodically (via checkbox)