Part 2 – Getting started with Microsoft Kinect

Hi Folks,

In the first Part, I discussed about the fundamentals of Kinect hardware as well as technical details of the camera & microphone array.
Today in this part of my “Programming with Kinect” series I am going to guide you about how to setup the development environment for Kinect and writing code to get started with development using Kinect.

Note: I’ll be using “Kinect for Xbox 360” hardware with “Kinect for Windows SDK” in my posts until & unless I am not using specific features of Kinect for Windows hardware.

Installation of SDK

The Kinect SDK is a development toolkit that allows software developers to build applications using Kinect and expand the possibilities of application and user experience. Kinect Software development kit (SDK) provides interface to interact with Camera, Motors, Microphone array etc. It allows user to build applications through Visual C#, Visual Basic.Net and C++ language using Microsoft Visual Studio IDE.

Recent release of Kinect SDK (1.6) supports Visual Studio 2012 and Windows 8.

Prerequisites

Before you begin with the installation, make sure you have the following components

• Microsoft Kinect Sensor
• USB cable of Kinect Sensor.

Make sure your Kinect sensor is unplugged and previous SDK of Kinect (if any) is uninstalled properly.

Installation

The installation procedure is very simple and wizard base.

• Go to Kinect for Windows developer’s page and download the latest SDK (v1.6) and Kinect Developer toolkit (I’ll discuss about it later).
• Install the SDK you just downloaded.
• After that Install the Kinect Toolkit.

After successful installation of both setup files you just downloaded, it’s time to connect the Kinect sensor with your PC.
Plug in your Kinect sensor with you PC and wait till the Windows install its driver.

Note: You may skip the “Searching Driver on Windows Update” part by clicking “Skip obtaining driver from Windows Update” while Windows is installing the driver. It will expedite the installation process and use the driver files that comes with the SDK you just downloaded.

Following components will be available after successful installation of Kinect SDK.

• Kinect Developer Browser.
• Kinect Studio.
• Kinect Drivers.
• Kinect Runtime.
• Kinect Speech Recognition Language Pack (en-US).

Kinect for Windows Architecture

Below is the architecture diagram of Kinect for Windows, You can access Kinect microphones array using your standard audio API.

I am not going to discuss this in details, but just for a quick view, we have

• Kinect Sensor (Physical Kinect Hardware)
• Kinect Driver
• Audio & Video Components
• Direct X Media Object
• Kinect API

More information regarding architecture can be found at http://msdn.microsoft.com/en-us/library/jj131023

Development Using Kinect

After we have our environment ready for software development using Kinect, let’s move towards using Kinect in our Project.

I’ll be using Windows Presentation Foundation (WPF) application type throughout my demos, but you can also use Win-Forms with Kinect.

Go ahead and start your Visual Studio.

• Create a new WPF Application and name it appropriately (in my case it’s UmerKinectBlogDemo1). 
• Add “References” of “Microsoft.Kinect” by right clicking on References folder in solution explorer and click “Add Reference”.
• Expand the “Assemblies” list on your left and select “Extensions”.
• Scroll down until you find “Microsoft.Kinect”, apply check on that and click OK.

After adding Kinect reference;

• In your XAML code, add the following attribute in your Window markup code.Loaded=”MainWindow_Loaded”

  

• Drag/Insert an Image Control from the Toolbox into your application, name & resize it appropriately as this image control will be used for showing the camera feeds from Kinect.

   

• Go to your Code Behind file of Main page (MainWindows.xaml.cs) and add the following namespaces into your code behind fileusing Microsoft.Kinect;
  using System.Windows.Media.Imaging;

• Create a global reference of “KinectSensor” class, this will hold the instance of our connected Kinect sensorKinectSensor myKinectSensor;

• In your MainWindow_Loaded method, fetch the connected Kinect sensor object using the following code;if (KinectSensor.KinectSensors.Count != 0)
  {
  if (KinectSensor.KinectSensors[0].Status == KinectStatus.Connected)

myKinectSensor = KinectSensor.KinectSensors[0];
else
MessageBox.Show(“Kinect sensor is not yet ready”);
} else
MessageBox.Show(“Kinect sensor is not connected”);

The “KinectSensors” collection holds all the connected Kinect sensor to the machine, since we have only one sensor connected to our machine, we are directly fetching the one at “zero” index. I am also doing a little error handling that if the status of the connected sensor is not ready, notify the user.

Note: For better experience, you should use StatusChanged event of KinectSensors Collection.

 Getting RGB (Color Camera Stream)

The color stream from Kinect sensor can be used to display the camera view of Kinect and you can also use the same stream to apply different Image Processing algorithms to achieve your specific output.

Kinect provides different resolutions and Formats for Color stream and you can select the one that best suits your need.

Now as we have our Kinect sensor object, let’s move and add an event handler of “ColorFrameReady” and start the “ColorStream” of our Kinect sensor to get the Color Stream from Kinect.

myKinectSensor.ColorFrameReady += myKinectSensor_ColorFrameReady;
myKinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

The “ColorFrameReady” event handler is used to handle the RGB images coming from our Kinect Sensor. We have the option to enable specified Video Stream (i.e. RGB, depth data etc.) in Kinect; for instance we are enabling the Color Stream and defining the Image Format of the stream in parameter.
Kinect supports different image formats including “RGB” and “YUV” and with different FPS (frame per second) and resolutions as well. We are using RGB Format of 640X480 resolution with 30 FPS. (Ideal FPS is 30, lower FPS results in slow image propagation).

• Insert the following code snippet into your ColorFrameReady event handler.void myKinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
  {
  using (ColorImageFrame es = e.OpenColorImageFrame())
  {
  if (!(es == null))
  {
  byte[] bits = new byte[es.PixelDataLength];
  es.CopyPixelDataTo(bits);
  image1.Source = BitmapSource.Create(es.Width, es.Height, 96, 96,PixelFormats.Bgr32, null, bits, es.Width * es.BytesPerPixel);
  }
  }
  }

  The image data is encapsulated in ColorImageFrame, we have to copy the image data into array of byte to use it, the “PixelDataLength” property of that ColorImageFrame has the length of pixel data buffer of the current frame.

  After that we have to create a bitmap image from the byte array (image data) so that can show it in our Image Control.

  BitmapSource.Create method is used to do that, the parameters of the methods are;

  • pixelWidth:        The width of the bitmap.
  • pixelHeight:       The height of the bitmap.
  • dpiX:                     The horizontal dots per inch (dpi) of the bitmap.
  • dpiY:                     The vertical dots per inch (dpi) of the bitmap.
  • pixelFormat:     The pixel format of the bitmap. (BGR, CMYK, Black etc)
  • Palette:               The palette of the bitmap.
  • Pixels:                  An array of bytes that represents the content of a bitmap image.
  • Stride:                  The stride of the bitmap. (Number of bytes allocated for one scan line of
    the bitmap).

After we create a bitmap image, assign that bitmap image object as the “Source” of Image Control we added in previous step.

Let the show begin

Our minimum code to display the camera stream from Kinect sensor is almost complete.

Before we run our project, there is one that that we need to do. We have to start the Kinect itself so that we can get the streams from the Kinect; to do that, add the following line at the end of your MainWindow_Loaded method.

myKinectSensor.Start();

Now run the project by pressing F5 to see the results.

Here is the result of my Kinect sensor.

It is my table where most of my stuff is found (I am bad at organizing stuff on my table  )

Summary

In this post, I’ve discussed the installation procedure, interfacing Kinect with our application and getting Color stream from the Kinect camera.

Don’t forget to do proper exception handling into your project, especially with KinectSensor object.

The complete project for your reference can be found here.

What’s next?

In my next post, I’ll be discussing about

• Skeleton Tracking

And more.

If you have any suggestions on topics, have questions, feedback or want to help me out, feel free to contact me by posting your comments below this post and I’ll try to help you out!

Muhammad Umer

Part 1 – Introduction to Microsoft Kinect Sensor

In this very first blog post of my Kinect programming series, I am going to talk about the fundamentals of Kinect sensor followed by other posts in which I’ll be discussing Kinect from a developer perspective.

In my spare time I like to develop & play with the Kinect. Kinect for Windows is a technology that is not very much adopted in my country, specifically Academia side. Most people see it as a toy but I see it as opportunities for mankind, making our lives easier. For example, doctors use it to help people that are disabled, make life more enjoyable for them and so on.

Building computer vision based applications had always been a difficult task for majority of application developers, since it requires lots of mathematics & similar algorithm information that researchers use in Computer vision, Signal processing and other fields of technology. Microsoft Kinect reduces a lot of development and hardware restriction that developers faces in past but still “What to do” & “How to do” purely depends on the developer.

You will see a lot of stuff on internet regarding Kinect integration with other systems (i.e. Arduino Platform) & using Kinect with other computer vision frameworks & libraries, I’ll try to discuss as much as possible in my future posts, but for now let’s get started to have basic understanding of Microsoft Kinect sensor.

Background

Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs. Based around a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox 360 without the need to touch a game controller, through a natural user interface using gestures and spoken commands.

The project is aimed at broadening the Xbox 360′s audience beyond its typical gamer base. A version for Windows was released on February 1, 2012.

After selling a total of 8 million units in its first 60 days, the Kinect holds the Guinness World Record of being the “fastest selling consumer electronics device”. 18 million units of the Kinect sensor had been shipped as of January 2012.

Microsoft released Kinect software development kit for Windows. This SDK will allow developers to write Kinect enabled apps in C++/CLI, C#, or Visual Basic .NET.

The Sensor

The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot and is designed to be positioned lengthwise above or below the video display. The device has two versions i.e. Kinect for Xbox 360 and Kinect for Windows (for commercial purpose).

The device features

• RGB camera.
• Depth sensor (IR).
• Multi-array microphone.
• Motor to adjust camera angle.

In addition to the above features, Kinect for Windows offer few extra features i.e.

• Facial recognition
  enables to track multiple points in your face like Skeleton Tracking.
• Near Mode
  enables the camera to see objects as close as 40 centimeters in front of the device without losing accuracy or precision, with graceful degradation out to 3 meters.
• Seated or 10 Joints Mode
  skeletal tracking which provides the capability to track the head, neck and arms of either a seated or standing user.

RGB Camera

The default RGB video stream uses 8-bit VGA resolution (640 × 480 pixels) with a Bayer color filter, but the hardware is capable of resolutions up to 1280×960 (at a lower frame rate) and other formats such as UYVY

Depth Sensor (IR)

The depth sensor consists of an infrared laser projector combined with a monochrome CMOS sensor, which captures video data in 3D under any ambient light conditions. The sensing range of the depth sensor is adjustable, and the Kinect software is capable of automatically calibrating the sensor based on gameplay and the player’s physical environment, accommodating for the presence of furniture or other obstacles.

The monochrome depth sensing video stream is in VGA resolution (640 × 480 pixels) with 11-bit depth, which provides 2,048 levels of sensitivity. The Kinect sensor has a practical ranging limit of 3.9 – 11 ft. distance when used with the Xbox software.

Field View

The area required to play Kinect is roughly 6 m², although the sensor can maintain tracking through an extended range of approximately 2.3 – 20 ft.

The horizontal field of the Kinect sensor at the minimum viewing distance of ~0.8 m (2.6 ft.) is therefore ~87 cm (34 in), and the vertical field is ~63 cm (25 in), resulting in a resolution of just over 1.3 mm (0.051 in) per pixel.

Microphone Array

The microphone array features four microphone capsules and operates with each channel processing 16-bit audio at a sampling rate of 16 KHZ.

What’s Next?

In my next post, I’ll be discussing about

• Installation of SDK
• Beginning with Kinect programming
• RGB camera stream
• Skeleton Tracking and more.

If you have any suggestions on topics, have questions, feedback or want to help me out, feel free to contact me by posting your comments below this post and I’ll try to help you out!

 

Popcorn Time – Windows 8

Few months ago, i published the first windows 8 application from Pakistan.
Although i didn’t spent much time in coding but i really spent alot of time to build a better User Experience.

The Response from Windows 8 Store is very well as compared to the users & reach up till now.

Below is the actual blog post from “Microsoft Pakistan”

I feel really proud to share that first Windows 8 application from Pakistan is now in Windows Store. The application name is Popcorn time. It’s an ultimate movies database, which gives you exclusive information about the
latest Hollywood & Bollywood movies along with their insight reviews, ratings, trailers and movie clips.

The main features include; Latest Hollywood & Bollywood Movies, Movies Description, Budget, Studios and other relevant information along with trailers.

Please click on the below link to have the Popcorn experience:

Popcorn Time

http://apps.microsoft.com/webpdp/en-US/app/popcorn-time/9aea85a0-0c0f-45a8-bbfe-34f4304669c1

The application is developed by a young talented student of Pakistan; Muhammad Umer (https://www.facebook.com/umer.rajput?ref=ts). He is also a Microsoft Community Speaker and contributes in the IT community as Microsoft ambassador. 

Microsoft TechTalk on Kinect Programming

Dear Recipient,   It is with great pleasure that I would like to invite you to the Microsoft TechTalk on KINECT Programming. Following are the details of the event.   About TechTalk Microsoft TechTalk is another unique initiative of Microsoft Innovation Center – Pakistan with the objective to highlight the creativity and talent of students and professionals in the local software economy. These sessions are not only focused towards the technology discussions but also showcase the achievements (demonstration) of talented students and professionals.   Date, Time, Location & Registration   Date:  Friday 8 June 2012 Time:  4:00pm – 6:00pm Location:  Microsoft | Innovation Center – Pakistan Fast National University of Computer and Emerging Sciences (City Campus) 22 G, Block 6, P.E.C.H.S., Karachi.   For Registration kindly email at: anas_kidwai@hotmail.com with your Name, University, Email Address and Mobile Number by Thursday 7th June 2012.   Note: The registration of this level 200 session is based upon first come first basis and will be confirmed by the MIC-Team. Speakers Profile Muhammad Umer, is the runner-up of Imagine Cup 2012 Pakistan Final. He is an intelligent student from Sir Syed University of Engineering and Technology (SSUET) and Microsoft Community Speaker. His skillset include Windows Phone Development, Game Development and Kinect Programing.   Session Outline ·         Introduction to Microsoft Kinect. ·         Kinect Hardware Foundation. ·         Evolution of UI’s. ·         Programming with Microsoft Kinect. o   Kinect Architecture. o   RGB Camera. o   Skeletal Tracking. o   Joint’s. o   Depth Data using IR Camera. Showcasing ·         Introduction to “Kenection”. ·         The Imagine Cup Experience. ·         Demo of Kenection.   I look forward towards your kind participation in this event.     Regards   Haseeb Shaukat MIC Business Manager | Microsoft Pakistan

My Events Planner – Windows Phone 7

 

My another app “My Events planner” is now at marketplace for download.

My Events planner is an event planning application for windows phone 7, now you can create, plan and manage your events with your team’s member’s right from your windows phone device. My events planner stores your event details into a private cloud hosted by developer. Interested users may host their own private cloud for their organizations by contacting the developer. My events planner synchronizes all your team members of the event by providing critical updates, to-do’s, happenings etc to all your members. Application Features • Private Credentials. • Manage Multiple Events. • Manage multiple team groups. • Event day calendar. • Event specific To-do. • Event specific time to time Happenings. • Assigning specific task to team member. • Call team member directly from event page. • Critical notifications to team members (Task assigned, Group Added, Event Created etc). • Ability to connect to custom private my events planner server.

[DOWNLOAD FROM MARKETPLACE]

Windows Phone 7 Seminar – Muhammad Ali Jinnah University

I was really busy since last 2 weeks, so now uploading the pics of Windows phone 7 seminar that took place 2 weeks ago.

[Although the Lighting condition was poor ]

 

 

 

 

 

 

 

 

 

 

 

Game Development Seminar – 2

Here is the source code of the DEMO i’ve developed at today’s game development interlocutory seminar.
Hope you’ll find it useful.
Do write your comments & suggestions.

[DOWNLOAD HERE]

Game Development Seminar – Source Code

I am attaching the source code of the INCOMPLETE game i developed at the MIC seminar for your reference.

feeling sad because of shortage of time, otherwise i had several others things & demo’s planned for the day.
But yeah, better luck next time..

[DOWNLOAD HERE]

Funkey Munkey Lite – Windows Phone 7

Waiting for another game?
its finally here, well it is also a lite version, i was playing with codes and accidently this game is developed.
Hope u remember the Nintendo and SEGA monkey game, well its the same one, but ported to WP7,
The development time was 8 Hours at MIC.
looking forward to release its pro version very soon.

Funkey Munkey is a game about a tribe of monkeys who are hungry for some long time, and now they are ready to collect anything that would look like eatable material.

You job is to control your monkey and let him collect as much banana’s and coconuts’ as he can, before his tribe dies of hunger.

Beware of bombs that are falling from the sky, they will make your food poisonous and your monkey can die sooner.

This is the “LITE” version of this game, its full version contains multiple level with multiplayer support, so that you and your friends can play together to collect food for their monkey tribes through one resource center.

Game Features
• Two Different Levels.
• Multiple Collectable items.
[Download From Marketplace]

 

 

 

 

 

My Baby Toys – Windows Phone 7

Here is another WP7 application,
well the whole idea was to transform our WP7 phone into a baby toy, so whenever our young ones hold the phone, they actually enjoy using it.
It contains rattle, talking doll. etc etc