Blog Archives

Part 2 – Getting started with Microsoft Kinect

Hi Folks,

In the first Part, I discussed about the fundamentals of Kinect hardware as well as technical details of the camera & microphone array.
Today in this part of my “Programming with Kinect” series I am going to guide you about how to setup the development environment for Kinect and writing code to get started with development using Kinect.

Note: I’ll be using “Kinect for Xbox 360” hardware with “Kinect for Windows SDK” in my posts until & unless I am not using specific features of Kinect for Windows hardware.

Installation of SDK

The Kinect SDK is a development toolkit that allows software developers to build applications using Kinect and expand the possibilities of application and user experience. Kinect Software development kit (SDK) provides interface to interact with Camera, Motors, Microphone array etc. It allows user to build applications through Visual C#, Visual Basic.Net and C++ language using Microsoft Visual Studio IDE.

Recent release of Kinect SDK (1.6) supports Visual Studio 2012 and Windows 8.

Prerequisites

Before you begin with the installation, make sure you have the following components

  • Microsoft Kinect Sensor
  • USB cable of Kinect Sensor.

Make sure your Kinect sensor is unplugged and previous SDK of Kinect (if any) is uninstalled properly.

Installation

The installation procedure is very simple and wizard base.

  • Go to Kinect for Windows developer’s page and download the latest SDK (v1.6) and Kinect Developer toolkit (I’ll discuss about it later).
  • Install the SDK you just downloaded.
  • After that Install the Kinect Toolkit.

After successful installation of both setup files you just downloaded, it’s time to connect the Kinect sensor with your PC.
Plug in your Kinect sensor with you PC and wait till the Windows install its driver.

Note: You may skip the “Searching Driver on Windows Update” part by clicking “Skip obtaining driver from Windows Update” while Windows is installing the driver. It will expedite the installation process and use the driver files that comes with the SDK you just downloaded.

Following components will be available after successful installation of Kinect SDK.

  • Kinect Developer Browser.
  • Kinect Studio.
  • Kinect Drivers.
  • Kinect Runtime.
  • Kinect Speech Recognition Language Pack (en-US).

Kinect for Windows Architecture

Below is the architecture diagram of Kinect for Windows, You can access Kinect microphones array using your standard audio API.

I am not going to discuss this in details, but just for a quick view, we have

  • Kinect Sensor (Physical Kinect Hardware)
  • Kinect Driver
  • Audio & Video Components
  • Direct X Media Object
  • Kinect API

More information regarding architecture can be found at http://msdn.microsoft.com/en-us/library/jj131023

Development Using Kinect

After we have our environment ready for software development using Kinect, let’s move towards using Kinect in our Project.

I’ll be using Windows Presentation Foundation (WPF) application type throughout my demos, but you can also use Win-Forms with Kinect.

Go ahead and start your Visual Studio.

  • Create a new WPF Application and name it appropriately (in my case it’s UmerKinectBlogDemo1). 
  • Add “References” of “Microsoft.Kinect” by right clicking on References folder in solution explorer and click “Add Reference”.
  • Expand the “Assemblies” list on your left and select “Extensions”.
  • Scroll down until you find “Microsoft.Kinect”, apply check on that and click OK.

After adding Kinect reference;

  • In your XAML code, add the following attribute in your Window markup code.Loaded=”MainWindow_Loaded”

  • Drag/Insert an Image Control from the Toolbox into your application, name & resize it appropriately as this image control will be used for showing the camera feeds from Kinect.

     

  • Go to your Code Behind file of Main page (MainWindows.xaml.cs) and add the following namespaces into your code behind fileusing Microsoft.Kinect;
    using System.Windows.Media.Imaging;
  • Create a global reference of “KinectSensor” class, this will hold the instance of our connected Kinect sensorKinectSensor myKinectSensor;
  • In your MainWindow_Loaded method, fetch the connected Kinect sensor object using the following code;if (KinectSensor.KinectSensors.Count != 0)
    {
    if (KinectSensor.KinectSensors[0].Status == KinectStatus.Connected)

myKinectSensor = KinectSensor.KinectSensors[0];
else
MessageBox.Show(“Kinect sensor is not yet ready”);
} else
MessageBox.Show(“Kinect sensor is not connected”);

The “KinectSensors” collection holds all the connected Kinect sensor to the machine, since we have only one sensor connected to our machine, we are directly fetching the one at “zero” index. I am also doing a little error handling that if the status of the connected sensor is not ready, notify the user.

Note: For better experience, you should use StatusChanged event of KinectSensors Collection.

 Getting RGB (Color Camera Stream)

The color stream from Kinect sensor can be used to display the camera view of Kinect and you can also use the same stream to apply different Image Processing algorithms to achieve your specific output.

Kinect provides different resolutions and Formats for Color stream and you can select the one that best suits your need.

Now as we have our Kinect sensor object, let’s move and add an event handler of “ColorFrameReady” and start the “ColorStream” of our Kinect sensor to get the Color Stream from Kinect.

myKinectSensor.ColorFrameReady += myKinectSensor_ColorFrameReady;
myKinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

The “ColorFrameReady” event handler is used to handle the RGB images coming from our Kinect Sensor. We have the option to enable specified Video Stream (i.e. RGB, depth data etc.) in Kinect; for instance we are enabling the Color Stream and defining the Image Format of the stream in parameter.
Kinect supports different image formats including “RGB” and “YUV” and with different FPS (frame per second) and resolutions as well. We are using RGB Format of 640X480 resolution with 30 FPS. (Ideal FPS is 30, lower FPS results in slow image propagation).

  • Insert the following code snippet into your ColorFrameReady event handler.void myKinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
    {
    using (ColorImageFrame es = e.OpenColorImageFrame())
    {
    if (!(es == null))
    {
    byte[] bits = new byte[es.PixelDataLength];
    es.CopyPixelDataTo(bits);
    image1.Source = BitmapSource.Create(es.Width, es.Height, 96, 96,PixelFormats.Bgr32, null, bits, es.Width * es.BytesPerPixel);
    }
    }
    }

    The image data is encapsulated in ColorImageFrame, we have to copy the image data into array of byte to use it, the “PixelDataLength” property of that ColorImageFrame has the length of pixel data buffer of the current frame.

    After that we have to create a bitmap image from the byte array (image data) so that can show it in our Image Control.

    BitmapSource.Create method is used to do that, the parameters of the methods are;

    • pixelWidth:        The width of the bitmap.
    • pixelHeight:       The height of the bitmap.
    • dpiX:                     The horizontal dots per inch (dpi) of the bitmap.
    • dpiY:                     The vertical dots per inch (dpi) of the bitmap.
    • pixelFormat:     The pixel format of the bitmap. (BGR, CMYK, Black etc)
    • Palette:               The palette of the bitmap.
    • Pixels:                  An array of bytes that represents the content of a bitmap image.
    • Stride:                  The stride of the bitmap. (Number of bytes allocated for one scan line of
      the bitmap).

After we create a bitmap image, assign that bitmap image object as the “Source” of Image Control we added in previous step.

Let the show begin

Our minimum code to display the camera stream from Kinect sensor is almost complete.

Before we run our project, there is one that that we need to do. We have to start the Kinect itself so that we can get the streams from the Kinect; to do that, add the following line at the end of your MainWindow_Loaded method.

myKinectSensor.Start();

Now run the project by pressing F5 to see the results.

Here is the result of my Kinect sensor.

It is my table where most of my stuff is found (I am bad at organizing stuff on my table 🙂 )

Summary

In this post, I’ve discussed the installation procedure, interfacing Kinect with our application and getting Color stream from the Kinect camera.

Don’t forget to do proper exception handling into your project, especially with KinectSensor object.

The complete project for your reference can be found here.

What’s next?

In my next post, I’ll be discussing about

  • Skeleton Tracking

And more.

If you have any suggestions on topics, have questions, feedback or want to help me out, feel free to contact me by posting your comments below this post and I’ll try to help you out!

       Muhammad Umer

Part 1 – Introduction to Microsoft Kinect Sensor

In this very first blog post of my Kinect programming series, I am going to talk about the fundamentals of Kinect sensor followed by other posts in which I’ll be discussing Kinect from a developer perspective.

In my spare time I like to develop & play with the Kinect. Kinect for Windows is a technology that is not very much adopted in my country, specifically Academia side. Most people see it as a toy but I see it as opportunities for mankind, making our lives easier. For example, doctors use it to help people that are disabled, make life more enjoyable for them and so on.

Building computer vision based applications had always been a difficult task for majority of application developers, since it requires lots of mathematics & similar algorithm information that researchers use in Computer vision, Signal processing and other fields of technology. Microsoft Kinect reduces a lot of development and hardware restriction that developers faces in past but still “What to do” & “How to do” purely depends on the developer.

You will see a lot of stuff on internet regarding Kinect integration with other systems (i.e. Arduino Platform) & using Kinect with other computer vision frameworks & libraries, I’ll try to discuss as much as possible in my future posts, but for now let’s get started to have basic understanding of Microsoft Kinect sensor.

Background

Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs. Based around a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox 360 without the need to touch a game controller, through a natural user interface using gestures and spoken commands.

The project is aimed at broadening the Xbox 360’s audience beyond its typical gamer base. A version for Windows was released on February 1, 2012.

After selling a total of 8 million units in its first 60 days, the Kinect holds the Guinness World Record of being the “fastest selling consumer electronics device”. 18 million units of the Kinect sensor had been shipped as of January 2012.

Microsoft released Kinect software development kit for Windows. This SDK will allow developers to write Kinect enabled apps in C++/CLI, C#, or Visual Basic .NET.

The Sensor

The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot and is designed to be positioned lengthwise above or below the video display. The device has two versions i.e. Kinect for Xbox 360 and Kinect for Windows (for commercial purpose).1

The device features

  • RGB camera.
  • Depth sensor (IR).
  • Multi-array microphone.
  • Motor to adjust camera angle.

In addition to the above features, Kinect for Windows offer few extra features i.e.

  • Facial recognition
    enables to track multiple points in your face like Skeleton Tracking.
  • Near Mode
    enables the camera to see objects as close as 40 centimeters in front of the device without losing accuracy or precision, with graceful degradation out to 3 meters.
  • Seated or 10 Joints Mode
    skeletal tracking which provides the capability to track the head, neck and arms of either a seated or standing user.

RGB Camera

The default RGB video stream uses 8-bit VGA resolution (640 × 480 pixels) with a Bayer color filter, but the hardware is capable of resolutions up to 1280×960 (at a lower frame rate) and other formats such as UYVY

Depth Sensor (IR)

The depth sensor consists of an infrared laser projector combined with a monochrome CMOS sensor, which captures video data in 3D under any ambient light conditions. The sensing range of the depth sensor is adjustable, and the Kinect software is capable of automatically calibrating the sensor based on gameplay and the player’s physical environment, accommodating for the presence of furniture or other obstacles.2

The monochrome depth sensing video stream is in VGA resolution (640 × 480 pixels) with 11-bit depth, which provides 2,048 levels of sensitivity. The Kinect sensor has a practical ranging limit of 3.9 – 11 ft. distance when used with the Xbox software.

Field View

The area required to play Kinect is roughly 6 m2, although the sensor can maintain tracking through an extended range of approximately 2.3 – 20 ft.

The horizontal field of the Kinect sensor at the minimum viewing distance of ~0.8 m (2.6 ft.) is therefore ~87 cm (34 in), and the vertical field is ~63 cm (25 in), resulting in a resolution of just over 1.3 mm (0.051 in) per pixel.

Microphone Array

The microphone array features four microphone capsules and operates with each channel processing 16-bit audio at a sampling rate of 16 KHZ.

What’s Next?

In my next post, I’ll be discussing about

  • Installation of SDK
  • Beginning with Kinect programming
  • RGB camera stream
  • Skeleton Tracking and more.

If you have any suggestions on topics, have questions, feedback or want to help me out, feel free to contact me by posting your comments below this post and I’ll try to help you out!