Marching Forward

where you at, engage 3D?

engage 3D development has been in full swing here the last 4-6 weeks. Really since the mozilla award at the beginning of February I've been teetering between excited and terrified for delivering on this projet. 

We've been extremely busy making connections here in and outside of Chattanooga, TN for the development of our use case. We've had a great response from local schools, both the STEM School and Normal Park Middle, here in Chattanooga. Also, the TN Aquarium is giving us access to their facility to begin streaming from there for testing. As of this week, EPB has agreed to enable the gigabit connection to the Aquarium so that we can stream from there at our maximum available rate. 

new backend

We've moved from a python backend to c++, using openframeworks and the ofxKinect addon. For those of you out there wanting to experiment with Kinect Development, this is a great starting point, and an easy way to manage all the libraries and dependencies needed when developing on OS X in particular. 

Here's a video of streaming calibrated data from the new backend to our old point cloud viewer (thanks Allison for unknowingly volunteering for this!)

new front-end

Also in the works is a new front end in webGL, which will enable surfaces (instead of point clouds) to be rendered. We're pretty psyched on this new addition. Look this to be live by the end of the month!

MAKERDAY!!!!

Ok, if you're in Chattanooga this next weekend, go to this. We'll be demoing the bleeding edge of our work and printing out armies of plastic dinosaurs.

Mozilla Development Round 3

A little post announcement celebration

Ok, so we were awarded again in the MozillaIgnite contest. This time, $20k, which is simply awesome. I've been so excited about this project, and knew that we were doing well, but it has all of a sudden gained much more momentum and attention than I had ever really imagined. Quite amazing for a group of friends who met at a hackathon six months ago....

Here was my pitch, please disregard the smart-ass opening comment:

Team Update:

So now we're focused on business. I've hired two additional people to work on this project. The first is a local code wizard by the name of Craig Tanis. Big surprise here: another simcenter student. Craig has an extensive background in programming and is acutally an adjunct professor at UTC in Computer Science. Craig will be working the web front end, primarily webGL. Also recently hired is James McNut. James received an MS in education a year ago from UT and did teaching rotations in the Chattanooga area with a heavy emphasis on STEM education. His job is  coordination between team engage3D and the local community in further defining our use case, finding a venue, and locating the ideal audience. James has an undergraduate degree in physics, and he has already proven to be invaluable to the team in the one week that he has been here.

Coding, Coding, Coding

Wrapping up weekend #1

We're camping out at co.lab most every weekend throughout the remainder of round 3. There really is little room for error on this, I feel as if we simply must deliver. We are so grateful for Company Lab's support and the use of their facilities, the least that we can do hold up our end of the deal.

We've identified a new framework that we'll be using on the back end, OpenFrameworks. This is a very powerful platform with many great features out of the box, and plenty of addons to keep any developer busy. It seems to be a staple technology to many artists and designers in the video space, we feel as if we have struck gold. It will not only help organize the structure of our code, but it gives us a common foundation between all the OS's we are managing between developers. (Not everyone has a mac unfortunately!)

Use Cases

We are in the process of pursuing relationships with roughly half a dozen different academic organizations for piloting our work. Details will be posted once we get some firm commitments! This project has gained a lot of attention since this last award and people are coming out in droves to support us. Chattanooga rocks.

Partners (look here as we add more to the list)

• Chattanooga Public Library
• Company Lab

Potential Partners

• PLAY program at USC Annenberg Innovation Lab

Brainstorming: Use Cases

One of the most exciting aspects of our project so far has been finding a niche to apply this technology. 3D video conferencing is great. We get to play with fun cameras, the Microsoft Kinect, and work with emerging web technologies such as webGL and webRTC. So what are we going to do with this?

We see a lot of applications of this technology in interactive live events: concerts, stage performances, sporting events. But the point of the Mozilla Ignite competition isn't how to best offer sports events streaming over the web (however mind-blowing that would be), it is how to leverage these ultra-fast networks and come up with applications which are beneficial to the country and will affect people. What we want to do is show everyone how this gigabit/programmable networking technology can change lives.

There is currently a huge push in the U.S. Educational system to promote the education and outreach of Science, Technology, Engineering and Mathematics, dubbed STEM. Our local university, UTC, has a full time STEM Outreach Coordinator, and Chattanooga is opening a new school geared entirely towards STEM. We believe that engage3D will be a powerful educational tool for both illustrating the power of new web technologies and a highly effective means of communicating educational material.

Enter the Nautilus

A week ago, Friday, Will Barkis  one of the program managers from Mozilla Ignite, landed us a conference call with some folks from The Oceans Exploration Trust. This is a very high profile group working with NOAA, National Geographic, the Sea Research Foundation, etc. All very established groups in research and educational outreach. They travel with the Nautilus, and give 5-7 video conferences A DAY to students all across the country. Take a minute and look through their websites to understand the magnitude of the projects they are involved in. Essentially, they take HD video and laser scans of famous shipwrecks and broadcast their findings all for STEM. 

This was such an amazing call and they showed a sincere desire to be involved with a 3D telepresence app, displaying interactive content from atop one of their ships. Their response to "would this be of value to you for your educational work?" was "OF COURSE!" I could go on and on about this, but I'd rather not jinx it... So we're going to pursue this relationship, and see if we could be of value to them in the future, once our application is more mature.

And, back to reality

Anyway, what I've taken from this conversation is that there are really valuable use cases for our engage3D project, especially in educadtion. Mozilla sees the value, The Oceans Exploration Trust sees the value, so we're going to gear this tool as an app used to give lectures or tutorials where 3D information is beneficial. We just pitched back to Mozilla last night for our next round of funding, and Will had many useful suggestions on building out our use case, such as looking at stereo 3D as well as the point clouds. I am also trying to find a way to justify the purchase of a small hologram projector... I think that we're in store for a lot of fun with this project in the upcoming months.

Mozilla Ignite, Moving Forward! - Development Round #2

Family fun on Main St. Chattanooga, TN

engageCV had a very exciting month of December. After we found out that we were funded by Mozilla to continue development on our 3D video conferencing application, Andor and myself immediately made plans to get him back down to chattanooga for a hack-a-thon down at the co.lab in Chattanooga, TN.

New Team Members: David Collao & Forrest Pruitt

Knowing that we were going to need additional resources and talent, we recruited two new team members from the UTC SimCenter. David Collao, a PhD student in computational engineering, and Forrest Pruitt an assistant systems administrator for the SimCenter. David is tasked to work on camera calibration, and 3D geometry processing, while Forrest is in charge of webRTC integration and systems administration. Forrest, having a strong interest in game development and programming, wants to add skills in openGL/webGL to his resume through his engagement with our project.

Hacking...

The majority of the hacking happened the week leading up to christmas. Forrest and David were both free from their normal course load and Andor was on vacation from his regular 9-5. The group camped out at co.lab all day from wednesday until sunday, with myself joining in the evenings. We went solid through the week, all up until Christmas eve. Overall it was a great time, we have a very strong, motivated and interesting team.

What progress!?

Great thanks to George MacKerron for his related work on kinects and webGL. We were able to adapt many valuable bits of information from his work here: http://blog.mackerron.com/2012/02/03/depthcam-webkinect/

• We dropped the compression routines and other features in order to allow faster rendering, we're currently transmitting and rendering this data around 135Mbps
• 3D Colored Point cloud transmitted over LAN and rendered in browser with WebGL
• WebRTC mic integration
• Researched camera calibration and how to interface with multiple cameras
• explored various networking topologies, unfortunately we were not able to gain access to the local GENI networking resources, they have recently changed hands from EPB to UTC and were currently off line
• discussed potential use cases for this application and have many ideas, most focused around education

Work is moving more from research to implementation, we believe that we'll be able to start building a robust/enterprise level system here shortly.  We will focus our next efforts on research still, with the final coding sprint to our system beta ready.

Here's a short video for your viewing pleasure:

Our RGB camera is still out of sync with the point cloud, this will be resolved shortly. Also, after taking a hard look at this, I'm going to allow the user to select different rendering modes, offering a triangulation of the scene along with background subtraction.

I'm still in the process of cleaning the code, but look for the project up on github here soon.

You can now view the code at: https://github.com/bbrock25/engage3D

Next Steps

• InstaGENI Rack integration
• Apply calibration routines to allow for multiple cameras
• develop new algorithms to support multiple point clouds
• Allow visualization of triangulated surfaces out of point cloud, point cloud data, however very flexible, might not be the best medium for a video conference