EEC-492/592 Kinect Application Development

1. EEC-492/592Kinect Application Development Lecture 10 Wenbing Zhao wenbing@ieee.org

2. Outline Human skeleton tracking (part III)

3. Skeleton Smoothing • Skeleton data can be smoothed by registering a set of smooth parameters • As a parameter to SkeletonStream.Enable(….); // create the smooth parameters var smoothParameters = new TransformSmoothParameters { Correction = 0.1f, JitterRadius = 0.05f, MaxDeviationRadius = 0.05f, Prediction = 0.1f, Smoothing = 1.0f }; // Enable the skeleton stream with smooth parameters this.sensor.SkeletonStream.Enable(smoothParameters);

4. Smoothing Parameters • Correction: specifies the amount of correction needed for the raw data. Range [0.0, 1.0] • The value must be within the range of 0 to 1.0 and the default value is 0.5. • With lower values more correction is applied, the raw data is corrected, and the data looks smoother • Smoothing: determines the amount of smoothing applied while processing. Range [0.0, 1.0] • The value must be within the range of 0 to 1.0 and the default value is 0.5. • The larger value, the smoother skeleton data, however, it increases the latency • Zero value => you will get the raw skeleton data

5. Smoothing Parameters JitterRadius: limit the radius value for jittery data, >=0 Measured in meters and the default value is 0.5. If the position of a jitter is outside the set radius, it is corrected to be positioned at the radius MaxDeviationRadius: max limit of the deviation that is allowed to be considered for determining a jitter, >=0 points outside of the MaxDeviationRadius range are not considered as jitter default value is 0.04 meters Prediction: number of frames predicted into the future, >=0 default value: 0.0 A value greater than 0.5 will likely lead to overshoot when the data changes quickly

6. Kinect SDK Smoothing Algorithm • Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data • The smoothing algorithm applies to each set of data and calculates a moving average based on the previous set of data • During the calculation of moving average, it uses the values passed by the smoothing parameter.

7. Getting Data Frames Together • We have covered three (color, depth, skeleton) types of data streams that are returned by the sensor • For a real application, we often need all three types of data streams • We can use a single event AllFramesReady, which will do the job for all three of them • The AllFramesReady event fires when new frames are available for the color, depth, and skeleton streams this.sensor.AllFramesReady+=sensor_AllFramesReady; void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { }

8. Getting Data Frames Together

9. Each skeleton frame contains a floor-clipping-plane vector, which contains the coefficients of an estimated floor-plane equation The equation is normalized so that the physical interpretation of D is the height of the camera from the floor, in meters The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players Floor Determination Ax + By + Cz + D = 0

10. How to Use FloorClipPlane In Your App • FloorClipPlane can be used to calculate the height of each joint with respect to the floor • Can be used for validation study, as well as fall detection • How to get FloorClipPlane parameters: • How to calculate the height of a joint: float A = frame.FloorClipPlane.Item1; float B = frame.FloorClipPlane.Item2; float C = frame.FloorClipPlane.Item3; float D = frame.FloorClipPlane.Item4; float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;

11. Joint Orientation • The bone orientation is provided in two forms: • A hierarchical rotation based on a bone relationship defined on the skeleton joint structure • An absolute orientation in Kinect camera coordinates • The orientation information is provided in form of quaternions and rotation matrices for use in different animation scenarios • BoneOrientation Class, public properties: • StartJoint: Gets the skeleton joint where the bone starts • EndJoint: Gets the skeleton joint where the bone ends • HierarchicalRotation: Gets or sets the rotation of a bone relative to its parent bone, of type BoneRotation • AbsoluteRotation: Gets or sets the rotation of the bone relative to camera coordinates, of type BoneRotation

12. BoneRotation Class • BoneRotation has two properties • Matrix: Gets or sets a matrix representation of the bone rotation • Quaternion: Gets or sets a quaternion representation of the bone rotation • Vector4 struct: has four properties • W, X, Y, Z • Matrix4 struct: has 17 properties • Identity, of type Matrix4 • M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33, M34, M41, M42, M43, M44 public Matrix4 Matrix { get; set; } public Vector4 Quaternion { get; set; }

13. Access Joint Orientation private void DrawSkeletonsWithOrientations() { foreach (Skeleton skeleton in this.skeletonData) { if (skeleton.TrackingState == SkeletonTrackingState.Tracked) { foreach (BoneOrientation orientation in skeleton.BoneOrientations) { // Display bone with Rotation using quaternion DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint, orientation.AbsoluteRotation.Quaternion); // Display hierarchical rotation using matrix DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix) } } } }

14. Build ShapeGame App • Create a new C# WPF project with name ShapeGame • It is a much simplified app from the Kinect ShapeGame • Only one shape, i.e., a ball is used • You are limited to use your righthand to hit the ball • Ball only drops from a designated spot down, i.e., no X velocity • Add Microsoft.Kinect reference • Design GUI • Added WindowLoaded() method in xaml file • Adding code

15. GUI Design Image control Canvas

16. Adding Code KinectSensor sensor; Skeleton[] totalSkeleton = new Skeleton[6]; WriteableBitmap colorBitmap; byte[] colorPixels; Skeleton skeleton; Thing thing = new Thing(); // a struct for ball double gravity = 0.017; • Add member variables: • WindowLoaded(): • Enable both ColorImageStream and SkeletonStream • Register event handler for both ColorFrameReady and SkeletonFrameReady events • Initialize the shape (i.e., ball)

17. Adding Code The Thing Struct. Make it a private struct inside the MainWindow class private struct Thing { public System.Windows.Point Center; public double YVelocity; public double XVelocity; public Ellipse Shape; public bool Hit(System.Windows.Point joint) { double minDxSquared = this.Shape.RenderSize.Width; minDxSquared *= minDxSquared; double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y); if (dist<= minDxSquared) { return true; } else return false; } }

18. Adding Code private static double SquaredDistance(double x1, double y1, double x2, double y2) { return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1)); } private void WindowLoaded(object sender, RoutedEventArgs e) { // same as before …. // new code for ball initialization thing.Shape = new Ellipse(); thing.Shape.Width = 30; thing.Shape.Height = 30; thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255)); thing.Center.X = 300; thing.Center.Y = 0; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); canvas1.Children.Add(thing.Shape); }

19. Modify Event Handler for Skeleton Frames void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { canvas1.Children.Clear(); advanceThingPosition(); canvas1.Children.Add(thing.Shape); // remaining code same as before ….. Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position); if(thing.Hit(handPt)) { this.thing.YVelocity = -1.0*this.thing.YVelocity; } }

20. Shape Animation Code void advanceThingPosition() { thing.Center.Offset(thing.XVelocity, thing.YVelocity); thing.YVelocity += this.gravity; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); // if goes out of bound, reset position, as well as velocity if (thing.Center.Y >= canvas1.Height) { thing.Center.Y = 0; thing.XVelocity = 0; thing.YVelocity = 0; } }

21. Challenge Task • For advanced students, improve the shape game in the following ways: • Keep the hit count, count should reset if ball falls to the bottom • Display the hit count at the ball • Consider the angle of the hit to the ball, and adjust the x velocity accordingly • Allow other joints to hit the ball too, such as left hand, shoulder, etc. • Add smoothing to skeleton data EEC492/693/793 - iPhone Application Development