How it Works

Combining a pico projector, microcomputer, drone, leap motion, and the OptiTrack IR tracking system, our system provides the user with a seamless natural user interface on any surface.

How it all works

There are quite a few moving parts (literally and figuratively) with our system. The literal moving part is the drone. Our system uses a Parrot AR 2.0 Drone Elite Edition as a vehicle for our projector and microcomputer. While the drone has an internal control system for stability and a packaged phone app, we wanted more control over the movement of the drone, so using a C# SDK and an external tracking system in the form of the OptiTrack system we can actually create missions for the drone to follow certain markers recognized by the OptiTrack system. An external control system using the data of the absolute location tracked by OptiTrack actually allows for finer tuning of movements. This allows for the drone to be semi-autonomous, waiting on inputs from the user on where to go, and proceeds to travel there autonomously.


The OptiTrack system consists of up to 6 IR cameras set up around the room and small reflective markers that can be tracked by the system. Thus by placing markers on a wrist band and the drone, two distinct (or more) objects can be tracked and that data can then be streamed (wired or wirelessly) into our backend server running the Unity 3d game engine which sends the actual commands to the drone via Wifi. However, Unity has to receive the commands from somewhere. That’s where the Leap motion comes in.


The leap motion gesture recognition system is attached to the user via a flexible wood wristband and 3D printed encasing. The different preprogramed gestures allow for drone commands such as takeoff, turn towards user, go forward to wall, and land. When these gestures are triggered, the Unity server which is running on a machine connected to the drone’s built in Wifi network, sends the command to the drone. However, the leap motion gestures trigger more than just drone movement. The gestures are also mapped to interactions with the user interface that is the main crux of our system.

Before describing how the leap motion interfaces with the projector and microcomputer, let’s clarify how the projector and microcomputer work. The two devices are separate, but have plug and play pairing that can be configured via SSHing into the microcomputer to run the appropriate commands. The projector is a Digital Light Processing (DLP) projector evaluation module from Texas Instruments The demo application loaded onto the microcomputer is a music player with play, pause, volume control, and next song functionality. The audio is routed through a USB hub with a USB audio card and Bluetooth transmitter add-on. However, this is all somewhat useless if the device needs to be SSH’ed the entire time, so a startup script takes care of everything including connecting to the correct Wifi network in order to receive the leap motion gesture triggers to interact with the interface.

In order for this all to be portable, a crucial component is powering the projector module. This was a critical challenge for us, as the overall circuitry can consume quite some power (particularly the projector). The projector was designed to be powered using a 5 volt 3 Amp power supply from an outlet. For our application we designed a power management system. Together the microcomputer and projector can consume over 3 Amps at 5 volts. To provide that much current we would need a strong battery. We decided to use a Lithium Polymer (LiPo) battery. On top of being able to power the module, we also needed to be wary of weight constraints. Our drone has limited payload capabilities and we needed to keep our weight low. Thus in a voltage to weight tradeoff, we settled on using 7.7 Volt batteries. We ruled out linear rregulators to step down the voltage because of inefficiency, as well as heat dissipation and current requirements. Instead, we turned to a Buck- Converter, a non-linear step down circuit to reduce the voltage from 7.7 to 5 volts. Yet we could still provide the high amounts of current needed at very high efficiency. Putting this together we got the connectors needed to wire everything together, and a charger to recharge our LiPo Batteries. With that we had designed a portable power supply for our projector and microcomputer which provides power to the system for over 45 minutes! The efficiency and strength of our power supply surprised even us.


Putting everything together means having a lipo battery powered microcomputer and projector with Bluetooth audio capabilities that responds to leap motion controls, all mounted on a drone that also responds to leap motion controls with external controlling and tracking via the OptiTrack system with all communication happening over the drone’s Wifi network. However, putting everything together isn’t always as clean as we want. In our case the main breakdown in the chain came at the point of mounting the payload to the drone.


The weight was simply too much for the drone to effectively take off and fly the way we wanted. Thus our demonstrations showcase everything integrated on either side of this breakdown. We feel that even more weight reductions are possible. With a precisely outlined schematic we can custom design more components and reduce weight significantly. Additionally due to budget constraints we purchased the cheapest drone with payload capability. While the drone nearly lifts our entire weight, we are certain that more expensive and powerful drones would have no trouble lifting our 200 gram module. In the future, we hope to modify this further and have both parts of our system integrated.

In the end we built a proof of concept prototype which showed that it is possible to build a portable, light weight, and wireless projector module. We were able to create natural interactions and show why portable projection is useful (even if hand held). With the drone, we built a semi-autonomous drone system that could intuitively travel to different parts of your room on command. Coupling these two would give us an intelligent system that would provide a digital interface to users when needed, but disappear when not in use. Drone’s Eye View: A New Perspective

Checkout Our Timeline

Checkout our timeline as we built this project over 13 weeks.

Visit Timeline