EEC-693/793 Applied Computer Vision with Depth Cameras

1. EEC-693/793Applied Computer Vision with Depth Cameras Lecture 2 Wenbing Zhao wenbing@ieee.org

2. Outline Components of Kinect sensor Kinect for Windows SDK Building first Kinect application 2

3. Components of Kinect for Windows Color camera Infrared (IR) emitter IR depth sensor Tilt motor Microphone array LED 3

4. The Color Camera Red, blue, green color images (RGB) Frame rate: 30 FPS at 640x480 12 FPS at 1280x960 Viewing range: 43 degrees vertical by 57 degrees horizontal 4

5. IR Emitter & IR Depth Sensor IR emitter emits IR light in a pseudo-random dot pattern Dotted light reflects off different objects IR depth sensor reads reflected light => depth info Depth stream resolution 640x480 320x240 80x60 5

6. Depth Data Processing 6

7. Tilt Motor • Used to change the camera and sensor’s angles to get the correct position of the human skeleton • Can tilt vertically (upwards/downwards) up to 27 degrees 7

8. Microphone Array 4 different microphones: locate the direction of the audio wave, not just capture the sound Enhanced noise suppression, echo cancellation, beam-forming 8

9. LED Placed between camera and IR emitter Used for indicating the status of the Kinect Green indicates Kinect drivers loaded properly 9

10. Kinect for Xbox vs. Kinect for Windows Kinect for Windows Near mode: as close as 40cm away from Kinect Can be used for commercial applications Can be used in a Win7 virtual machine: HyperV, VMWare, Parallels For both Track motion up to 12 feet (4 meters) away Identical resolution 10

11. Kinect for Windows SDK Supported operating systems Windows 7, Windows 8, Windows Embedded 7 Hardware requirements Dual core 2.66 GHz or faster CPU Dedicated USB 2.0 bus 2GB or higher RAM Software requirement Microsoft Visual Studio 2010 or higher Kinect for Windows SDK: current version 1.8 11

12. Kinect for Windows SDK 12

13. Kinect for Windows SDK 13

14. How Applications Interact with Kinect 14

15. Classification of Kinect SDK APIs 15

16. Kinect Driver The Kinect driver controls the camera, depth sensor, audio microphone array, and motion Data passes between the sensor and the app in the form of data streams Color data stream Depth data stream Audio data stream Infrared data stream (for low light environment) Accelerameter data 16

17. The Near Mode Track a human upper body as close as 40 cm Only Kinect for Windows supports the near mode 17

18. Tracking Human Skeleton and Joint Movement Kinect SDK does the magic to extract the skeleton and joint positions from the data streams received Can track up to two users at the same time with full joint positions Each skeleton consists of 20 joints 18

19. Building First Kinect App Modified KinectInfoBox app Steps Create a new project: C# WPF application Adding Kinect reference Draw the GUI Modify MainWindow.xaml Adding code 19

20. Creating a New Project 20

21. Adding Kinect Libraries 21

22. Create User Interface 22

23. Modify MainWindow.xaml 23

24. Adding Code (MainWindow.xaml.cs) Setting up name space using Microsoft.Kinect; Setting up state variable for the Kinect sensor An instance of Microsoft.Kinect.KinectSensor class It represents the complete runtime pipeline for the Kinect sensor during life cycle of the app public partial class MainWindow : Window { KinectSensor sensor; // remaining code goes here } 24

25. Adding Code private void WindowLoaded(object sender, RoutedEventArgs e) { if (KinectSensor.KinectSensors.Count > 0) { this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) { this.sensor.Start(); displayInfo(); } } else { MessageBox.Show("No device is connected with system!"); this.Close(); } } private void WindowClosing(object sender, System.ComponentModel.CancelEventArgs e) { if (this.sensor != null && this.sensor.IsRunning) { this.sensor.Stop(); } 25

26. Adding Code Double click “Increase” button, the template for the button1_Click() method will be automatically created Double click “Decrease” button, the template for the button2_Click() method will be automatically created private void displayInfo() { this.textBlock1.Text = this.sensor.DeviceConnectionId; this.textBlock2.Text = this.sensor.Status.ToString(); this.textBlock3.Text = this.sensor.ElevationAngle.ToString(); } private void button1_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle + 1; displayInfo(); } private void button2_Click(object sender, RoutedEventArgs e) { this.sensor.ElevationAngle = this.sensor.ElevationAngle - 1; displayInfo(); } 26

27. More In-depth Stuff Starting Kinect sensor: sensor.Start() In the Start() method First check the status of Kinect, if no sensor connected, runtime will throw an InvalidOperationException object with the KinectNotReady message Init the sensor if connected Init options None: default option UseDepthAndPlayerIndex UseColor UseSkeletonTracking UseDepth UseAudio 27

28. More In-depth Stuff Open data streams ColorImageStream: this.sensor.ColorStream.Enable(); DepthImageStream: this.sensor.ColorStream.Enable(); SkeletonStream: this.sensor.SkeletonStream.Enable(); Stopping the Kinect sensor this.sensor.Stop(); Stops the color, depth, and skeleton data stream Stops audio source, if open Kills all threads spawned Shuts down Kinect sensor and sets init option to None 28