Creating a simple Flash app with Kinect

lucky larry flash kinect

First of all read up on how to set the Kinect up on your computer, I’m using Windows XP here.

Getting the Kinect setup on your PC: OpenNI / Primesense

Interfacing Kinect with Flash & AS3

Now you’ve read those (If you needed to), for this tutorial you’ll also need Flash CS4 or later, which you can download a demo from Adobe for free. Alternatively you can use the opensource actionscript compilers – I’m only using Flash to quickly draw some objects because I can’t be bothered to code them, so you can take my actionscript file and add to it if you don’t want to use the Flash IDE. One note though, you’ll need to compile and run this as an executable (.exe).

You can download ALL my files here

I say *ALL* because it seems that everyone’s more than happy to show their video of this working but no one shares their code, at the very least they share the .exe files. So here it all is and it’s very simple to follow, making use of the excellent AS3 server and libraries from as3kinect.org.

lucky larry flash kinect files

To quickly test this out, plug in the Kinect, download and unzip my folder, start as3-server.exe and then start LuckyLarryFlashKinect.exe. Perform the Cornholio pose and it should start tracking your right hand. Push your hand out to press a button.

Hopefully that worked albeit a bit glitchy. So how was it done? I’ll walk through this assuming you have some Flash knowledge but essentially it’s just a rehash of the demo on as3kinect.org

First get that demo of Flash, I tried CS3 but the AS3 kinect libraries are a bit buggy, so I’d recommend grabbing the latest demo from Adobe’s site, I’m using CS5.5 which works fine for this, but CS4 works just as well.

The LuckyLarryFlashKinect.fla file literally just has the buttons, hand icon and a link to an external actionscript file, if you really wanted to you could just draw that with actionscript code and compile the file using an open source alternative – I’m just being lazy.

If you open that up, you’ll see I have a movieclip called all with an instance name of all

actionscript instance name

That contains the movie clip buttons and 6 instances of the button movieclip

flash kinect buttons

The button movieclip has 2 frames in there, one for on and another for the off state, you could easily change the colour etc… in actionscript but I wanted to specfically show the nextFrame function in my code.

flash kinect hand

The hand movieclip is called Right, this is the hand icon which has another movieclip called handhit. I have this so that I can have a much smaller, more precise hit test bounding area. Under the properties for each movieclip on the stage you’ll see that they have instance names to link them to actionscript. Other than that, the files publishing settings are set to Actionscript 3.0, and to create a SWF and EXE file.

That’s it. Everything else happens in the actionscript file which I’ll walk through in more detail.

Looking at LuckyLarryFlashKinect.as actionscript file now…

First we create our package and import the necessary libaries

package 
{
	// load relevant libraries
	import flash.events.Event;
	import flash.display.MovieClip;
	// AS3 kinect 
	import org.as3kinect.*;
	import org.as3kinect.as3kinectWrapper;
	import org.as3kinect.events.as3kinectWrapperEvent;
	// Greensock for programmatic tweening for rollovers etc...
	import com.greensock.*;
	import com.greensock.easing.*;

Now we create our package class extending the movieclip object and declare our private variables


	public class LuckyLarryFlashKinect extends MovieClip
	{
		// declare new instance of the AS3 wrapper
		private var as3w:as3kinectWrapper;
		// configure a 'depthLimit' limit in millimetres, you need to stand further back than this
		private var depthLimit:Number = 800; 
		// store the depthLimit of the right hand
		private var right_z:Number;
		// store if something has been pushed
		private var isPush:Boolean;

The next function creates the link to the skeleton data via the libary files and then registers 3 event listeners, 1 for the skeletal data and the others for Enter Frame

		// declare the event listeners for skeleton tracking and enter frame
		public function LuckyLarryFlashKinect()
		{
			as3w = new as3kinectWrapper();
			as3w.addEventListener(as3kinectWrapperEvent.ON_SKEL, on_skeleton);
			addEventListener(Event.ENTER_FRAME,EnterFrame);
			addEventListener(Event.ENTER_FRAME,interaction);
		}

On the EnterFrame function we get the skeletal data and check for the depth of your hand, adding or removing the interaction event listener depending upon the distance. In this function, I also handle the rollover, since it’s present every time irrelevant of depth, I’ll explain that for loop a bit later on.

		private function EnterFrame(event:Event)
		{
			// get skeleton information to retrieve right hand info
			as3w.getSkeleton();
			//Depth detection event listener - set it to work at a minimal depthLimit
			if ((right_z < depthLimit)){
				removeEventListener(Event.ENTER_FRAME, interaction);
			}else if ((right_z > depthLimit)){
				addEventListener(Event.ENTER_FRAME, interaction);
			}

			/* Rollover
			 * Because we're not detecthing depthLimit and just x and y, this function lives
			 * outside of the interaction function it loops through all the items within 
			 * the buttons movie clip and do hit detection on each one, instead of declaring 
			 * each individual possible hit test we handle it in a much smaller for loop
			 */
			var i:int = 0;
       		for(i; i < all.buttons.numChildren; ++i){
				if (right.handhit.hitTestObject(all.buttons.getChildAt(i))){
					// use greensock to make the buttons bigger/smaller
					TweenMax.to(all.buttons.getChildAt(i), 0.1, {scaleX:1.10, scaleY:1.10, ease:Cubic.easeIn});
				} else {
					TweenMax.to(all.buttons.getChildAt(i), 0.1, {scaleX:1, scaleY:1, ease:Cubic.easeIn});
				}
			}

		}

After this we declare the skeleton function to get the data for the right hand and map it to our movieclip named right. For the x and y data I alter this to translate the tracked data to the screen, without this you can only control a very small space inside the movieclip. Unlike the demo files, here we're also looking at z index/ distance from the sensor, this is the real difference between the demos.

		private function on_skeleton(event:as3kinectWrapperEvent):void
		// gets hand tracking and data to build into gestures
		{
			var skel:Object = event.data;
			// right hand position, fetched from the AS3kinect wrapper/ skeleton data
			// I add a multipler (*4) and an offset (-400) to compensate for using right hand only, saving me stretching!
			right.x = (skel.r_hand.x*4)-400;
			right.y = (skel.r_hand.y*3)-400;
			right_z = skel.r_hand.z;
		}

In the final function, interaction, I just observe the right hand depth (z) and if that's less than my depth variable I then execute the hit tests dynamically to figure out which button to click/ animate. So I'm assuming that you stand further back than my depth limit, when the hand is less than this, you're pushing a button. I guess instead of calculating the distance of your hand from the sensor, we could calculate the distance of your hand from your shoulder, head or neck allowing you stand as close or far away from the sensor as you want as your hand depth then becomes relative to your body and not the sensor.

		private function interaction(e:Event)
		/* function that does all the button clicking /gestures
		 * in this case we're concerned on the detected depthLimit to figure out if
		 * a button has been 'pushed'
		 */
		{
			// if right hand is at required depthLimit for 'pushing'
			if (right_z < depthLimit){
				removeEventListener(Event.ENTER_FRAME, interaction);	
				/* Same as the rollover loop, detect the hit but also register that
				 * something has been 'pushed'
				*/

By setting a depth element (depthLimit), it's this that creates the push to click gesture. However there is an issue, because we're always calculating the z index of your hand in the application file, there's quite a bit of lag and when you want to use logic loops such as 'while' then this can cause you some memory issues in Flash, especially since actionscript is not a multithreaded language.

        		var i:int = 0;
        		for(i; i < all.buttons.numChildren; ++i){
			        if(right.handhit.hitTestObject(all.buttons.getChildAt(i))&& isPush==false){
						isPush = true;
						all.buttons.getChildAt(i).nextFrame();
						// greensock shrink the button
						TweenMax.to(all.buttons.getChildAt(i), 0.06, {scaleX:1, scaleY:1, ease:Cubic.easeIn});
						if(all.buttons.getChildAt(i).name == "myMovieClipName") {
							//Do something for an individual button
						}
            		} else if (right.handhit.hitTestObject(all.buttons.getChildAt(i))&& isPush==true){
						isPush = false;				
						all.buttons.getChildAt(i).prevFrame();
						TweenMax.to(all.buttons.getChildAt(i), 0.06, {scaleX:1, scaleY:1, ease:Cubic.easeIn});
						if(all.buttons.getChildAt(i).name == "myMovieClipName") {
							//
						}
					}
				}
			}
		}
	}
}

The final part of the interaction function is the hit test which then loops through each button and skips to it's next frame if it's the one hit, I also left in some logic to get the name, in case you wanted specific functions. I use the for loop rather than write a whole series of if statements, it works by getting the number of child movieclips in the parent movieclip and then iterates through this number to get the index of any child movieclip. With that we can also get the name, frames etc... of that clip. You can read more here about Dynamic hit tests in Flash & AS3

In this loop we record whether something has been pushed so we can skip back and forward between frames - if you were loading new content/ new screen on a button press then you wouldn't need this logic, it's just there as a quick hack.

Finally, be warned, if you introduce a lot of complex actionscript, then expect it to drastically slow down, the tween animations being used are actually happening much slower than they should. This is because of the way in which we're grabbing the Kinect data. If you were to do this with the Microsoft Kinect SDK on Windows 7 you'd still have the same issue due to the amount of middleware needed to get this to run. Looking forward however, my hope is that from Windows 8 onwards we could have native Kinect like the Xbox, I doubt Microsoft will ever support any none Microsoft platform or even their own older OS.

Here's the video of me using the app just to prove that it does work although the push gesture is a bit glitchy...

4 Comments

  • If I was more clever, had a computer, had a kinect box and half a brain I would give that a go. I read your post somehow expecting to be able to follow it. But you know me hey…..

    • It’s all the start of forthcoming headaches.

      I should have just left it alone and kept playing Battlefield instead of tinkering.

  • Hi Larry!

    I wanna give you so much thanks for this tutorial. I do have a peoblem with the sample files you shared though, I think the as3-server.exe file is having a problem running. Please let me know what I have been missing.

    And also, is the as3-server.exe necessary to all projects?

    Thanks and more power

    Marvin

  • l have download your sample.
    Your ‘s TweenMax have some little error.
    thank for your sample:)

You must be logged in to post a comment.