Part 2 – Getting started with Microsoft Kinect

Hi Folks,

In the first Part, I discussed about the fundamentals of Kinect hardware as well as technical details of the camera & microphone array.
Today in this part of my “Programming with Kinect” series I am going to guide you about how to setup the development environment for Kinect and writing code to get started with development using Kinect.

Note: I’ll be using “Kinect for Xbox 360” hardware with “Kinect for Windows SDK” in my posts until & unless I am not using specific features of Kinect for Windows hardware.

Installation of SDK

The Kinect SDK is a development toolkit that allows software developers to build applications using Kinect and expand the possibilities of application and user experience. Kinect Software development kit (SDK) provides interface to interact with Camera, Motors, Microphone array etc. It allows user to build applications through Visual C#, Visual Basic.Net and C++ language using Microsoft Visual Studio IDE.

Recent release of Kinect SDK (1.6) supports Visual Studio 2012 and Windows 8.

Prerequisites

Before you begin with the installation, make sure you have the following components

• Microsoft Kinect Sensor
• USB cable of Kinect Sensor.

Make sure your Kinect sensor is unplugged and previous SDK of Kinect (if any) is uninstalled properly.

Installation

The installation procedure is very simple and wizard base.

• Go to Kinect for Windows developer’s page and download the latest SDK (v1.6) and Kinect Developer toolkit (I’ll discuss about it later).
• Install the SDK you just downloaded.
• After that Install the Kinect Toolkit.

After successful installation of both setup files you just downloaded, it’s time to connect the Kinect sensor with your PC.
Plug in your Kinect sensor with you PC and wait till the Windows install its driver.

Note: You may skip the “Searching Driver on Windows Update” part by clicking “Skip obtaining driver from Windows Update” while Windows is installing the driver. It will expedite the installation process and use the driver files that comes with the SDK you just downloaded.

Following components will be available after successful installation of Kinect SDK.

• Kinect Developer Browser.
• Kinect Studio.
• Kinect Drivers.
• Kinect Runtime.
• Kinect Speech Recognition Language Pack (en-US).

Kinect for Windows Architecture

Below is the architecture diagram of Kinect for Windows, You can access Kinect microphones array using your standard audio API.

I am not going to discuss this in details, but just for a quick view, we have

• Kinect Sensor (Physical Kinect Hardware)
• Kinect Driver
• Audio & Video Components
• Direct X Media Object
• Kinect API

More information regarding architecture can be found at http://msdn.microsoft.com/en-us/library/jj131023

Development Using Kinect

After we have our environment ready for software development using Kinect, let’s move towards using Kinect in our Project.

I’ll be using Windows Presentation Foundation (WPF) application type throughout my demos, but you can also use Win-Forms with Kinect.

Go ahead and start your Visual Studio.

• Create a new WPF Application and name it appropriately (in my case it’s UmerKinectBlogDemo1). 
• Add “References” of “Microsoft.Kinect” by right clicking on References folder in solution explorer and click “Add Reference”.
• Expand the “Assemblies” list on your left and select “Extensions”.
• Scroll down until you find “Microsoft.Kinect”, apply check on that and click OK.

After adding Kinect reference;

• In your XAML code, add the following attribute in your Window markup code.Loaded=”MainWindow_Loaded”

  

• Drag/Insert an Image Control from the Toolbox into your application, name & resize it appropriately as this image control will be used for showing the camera feeds from Kinect.

   

• Go to your Code Behind file of Main page (MainWindows.xaml.cs) and add the following namespaces into your code behind fileusing Microsoft.Kinect;
  using System.Windows.Media.Imaging;

• Create a global reference of “KinectSensor” class, this will hold the instance of our connected Kinect sensorKinectSensor myKinectSensor;

• In your MainWindow_Loaded method, fetch the connected Kinect sensor object using the following code;if (KinectSensor.KinectSensors.Count != 0)
  {
  if (KinectSensor.KinectSensors[0].Status == KinectStatus.Connected)

myKinectSensor = KinectSensor.KinectSensors[0];
else
MessageBox.Show(“Kinect sensor is not yet ready”);
} else
MessageBox.Show(“Kinect sensor is not connected”);

The “KinectSensors” collection holds all the connected Kinect sensor to the machine, since we have only one sensor connected to our machine, we are directly fetching the one at “zero” index. I am also doing a little error handling that if the status of the connected sensor is not ready, notify the user.

Note: For better experience, you should use StatusChanged event of KinectSensors Collection.

 Getting RGB (Color Camera Stream)

The color stream from Kinect sensor can be used to display the camera view of Kinect and you can also use the same stream to apply different Image Processing algorithms to achieve your specific output.

Kinect provides different resolutions and Formats for Color stream and you can select the one that best suits your need.

Now as we have our Kinect sensor object, let’s move and add an event handler of “ColorFrameReady” and start the “ColorStream” of our Kinect sensor to get the Color Stream from Kinect.

myKinectSensor.ColorFrameReady += myKinectSensor_ColorFrameReady;
myKinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

The “ColorFrameReady” event handler is used to handle the RGB images coming from our Kinect Sensor. We have the option to enable specified Video Stream (i.e. RGB, depth data etc.) in Kinect; for instance we are enabling the Color Stream and defining the Image Format of the stream in parameter.
Kinect supports different image formats including “RGB” and “YUV” and with different FPS (frame per second) and resolutions as well. We are using RGB Format of 640X480 resolution with 30 FPS. (Ideal FPS is 30, lower FPS results in slow image propagation).

• Insert the following code snippet into your ColorFrameReady event handler.void myKinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
  {
  using (ColorImageFrame es = e.OpenColorImageFrame())
  {
  if (!(es == null))
  {
  byte[] bits = new byte[es.PixelDataLength];
  es.CopyPixelDataTo(bits);
  image1.Source = BitmapSource.Create(es.Width, es.Height, 96, 96,PixelFormats.Bgr32, null, bits, es.Width * es.BytesPerPixel);
  }
  }
  }

  The image data is encapsulated in ColorImageFrame, we have to copy the image data into array of byte to use it, the “PixelDataLength” property of that ColorImageFrame has the length of pixel data buffer of the current frame.

  After that we have to create a bitmap image from the byte array (image data) so that can show it in our Image Control.

  BitmapSource.Create method is used to do that, the parameters of the methods are;

  • pixelWidth:        The width of the bitmap.
  • pixelHeight:       The height of the bitmap.
  • dpiX:                     The horizontal dots per inch (dpi) of the bitmap.
  • dpiY:                     The vertical dots per inch (dpi) of the bitmap.
  • pixelFormat:     The pixel format of the bitmap. (BGR, CMYK, Black etc)
  • Palette:               The palette of the bitmap.
  • Pixels:                  An array of bytes that represents the content of a bitmap image.
  • Stride:                  The stride of the bitmap. (Number of bytes allocated for one scan line of
    the bitmap).

After we create a bitmap image, assign that bitmap image object as the “Source” of Image Control we added in previous step.

Let the show begin

Our minimum code to display the camera stream from Kinect sensor is almost complete.

Before we run our project, there is one that that we need to do. We have to start the Kinect itself so that we can get the streams from the Kinect; to do that, add the following line at the end of your MainWindow_Loaded method.

myKinectSensor.Start();

Now run the project by pressing F5 to see the results.

Here is the result of my Kinect sensor.

It is my table where most of my stuff is found (I am bad at organizing stuff on my table 🙂 )

Summary

In this post, I’ve discussed the installation procedure, interfacing Kinect with our application and getting Color stream from the Kinect camera.

Don’t forget to do proper exception handling into your project, especially with KinectSensor object.

The complete project for your reference can be found here.

What’s next?

In my next post, I’ll be discussing about

• Skeleton Tracking

And more.

If you have any suggestions on topics, have questions, feedback or want to help me out, feel free to contact me by posting your comments below this post and I’ll try to help you out!

Muhammad Umer