As a student in the Global Business and Digital Arts program at the University of Waterloo, I am always looking for opportunity to learn various technologies, and mashing them up in new and interesting ways. This is why I was extremely excited when my friend and classmate Filip Jadczak shared with me his experience working as a REAPster with REAP. When I learned that I would be paid to experiment with and learn about new technologies, I jumped at the chance to get involved.
Part 1 – My First Sprint
I was allowed to choose the focus of my first sprint, and since I am interested in the potential future applications of virtual reality, I wanted to explore programs that I could use to create a smooth platform when combining the Oculus Rift DK2 and the Microsoft Kinect.
Currently, I have seen users constrained in their interaction in virtual reality by merely moving their hands or by tilting their head. I believe that if the Kinect was connected to the Oculus Rift you would be able to move freely around in a virtual world, pick up virtual objects and interact with them. By combining the Oculus Rift and Kinect, there could potentially be more opportunities available for developers.
Part 2 – What I Have Been Doing
After I researched what I needed to combine the DK2 and the Kinect, I tried to make a very simple scenario using Unity3D, a game development platform, which thankfully is very simple and easy to learn. Here is a screenshot of a room I made:
It was very easy to learn the SDK (Software Development Kit) and how to bring new assets into Unity. I first started by only building a room for the Oculus to make sure the Oculus component would work. This is where I ran into my first problem; I could see it on the computer but not in the Oculus. I learned that the Oculus needed to be in direct display mode and not in extended mode. Changing these settings is easy through the Oculus Configuration Utility, however, I ran into trouble again as I kept getting an error message saying the graphics driver was out of date. I checked for updates but was told there were no updates available, and ended up going to the AMD website to find and download the driver update the computer said I needed.
Despite updating the driver and rebooting the system, the Oculus Configuration Utility still showed he same error message. I knew it wasn’t a hardware issue as the DK2 demo scene worked. Eventually, I realized that a different Runtime version was required but not the newest version. Runtime 0.8 was installed and the one that worked was Runtime 0.6. The graphics driver update error was misleading and not the actual issue. The issue was that applications built against SDKs prior to Runtime 0.6 will not work with Runtime of 0.7 or higher.
Finishing that, I moved onto the Kinect. It was harder to figure out what I needed for the Kinect to run on Unity, because there was no integration package specifically for Unity on Microsoft’s website, as there had been on Oculus’ website. When I brought in the Kinect’s SDK to Unity I kept receiving error messages with the assets. As such, I started to read what other people have done to troubleshoot this and discovered that many people used Zigfu’s Itegration Package. This worked fairly well and I received less error messages from their SDK. One thing I learned was that the Kinect’s toolkit works a lot better on Windows 8 rather than 7 and the SDK was a lot easier to use with Unity 5.
I did some further research on what has been done and found that there are some fairly good assets in Unity’s Asset Store which are better at copying players real life movements. The main issue I found with what has been done with the Microsoft Kinect is that it can’t detect exact figure movements. Unfortunately, this limits the possibilities of combining the Kinect with a Virtual Reality or Augmented Reality product. As such, I suggest looking into other options such as the Leap Motion.
Part 3 – What I Learned
Throughout this sprint I realized why there hasn’t been more content created for the Oculus Rift and Microsoft Kinect. I ran into many technical issues and it was very time consuming to troubleshoot them, all while trying to learn how to use the Oculus Rift DK2, Kinect and Unity individually and together. If there had been a list of which SDK and Runtime versions worked best with the version of Unity that would have helped the process a lot.
The people who successfully managed to combine the DK2 with the Kinect had made changes to some of the drivers with their own development skills. As a designer and not a developer I would have loved to see an online community designed for VR and AR support where people write posts on what has and hasn’t worked for them. Through this community, it would be both helpful and encouraging for anyone new to the technical side of working with these technologies, and would help to speed up the creation of marketable content and overall development of augmented reality and virtual reality applications.