Half Real was a live action, interactive theatre production I worked on with The Border Project back in 2011. A case study article on Half Real was published at the 2012 ISMAR conference. This post is a more human readable summary of my work.

Half Real is based on a murder investigation, where the live audience votes on how they investigation proceeds. On stage, actors are immersed into a virtual world projected onto the set. Actors were tracked in 3D on set, and the projections reacted to their actions and movements. I built the software that drove the projected content.

I worked on Half Real, building the projection software over a four month period. The system builds on technology I was developing at the University of South Australia during my PhD. The code was all in C++ with OpenGL, running on Linux.

The Stage as a 3D Environment

The projection system for Half Real needed to track actors as they moved about on stage and have projections appear attached to the actors. Simple 2D projection, which is common in performance art, was not going to cut it. Instead, the entire set is modeled as a 3D scene. A calibration process figures out the position and orientation of the projectors, and creates a perspective correct projection in OpenGL. Rather than creating unique content for each projector, the content is created for the scene, and then we figure out what the projectors can see.

This means art assets only need to be created once, regardless of the number of projectors. In addition, more projectors can be easily added. Pre-show setup is simpler, as projectors only need to be roughly aligned; the calibration algorithm takes care of the rest.

3D model of the set

3D model of the set
Projected content on the physical set
Projected content in real life

Actor Tracking

Projected text attached to the performer

Projected text attached to the performer

We used a Microsoft Kinect with OpenNI for tracking actors on set. Kinect works well for theatre; it’s IR and isn’t affected by stage lighting, and it’s cheap. However, there are a few limitations. The tracking isn’t precise enough to project onto actor, so we project onto the set near them. Also, the resolution is low and it can’t track really small objects.

While the Kinect is quite good at tracking people, it is not able to reliably identify them. Actors enter and exit the set many times throughout each performance. We needed the system to automatically attach the correct information to the correct actor. Through the course of rehearsals, catch areas were identified in each scene. These catch areas are regions on the set that an actor would always walk through. Once the tracking system registered an actor passing through a catch area, that actor was associated with the correct virtual information. In addition to catch areas, dead areas that never associated tracked objects in the system were needed.

Pre-show projector calibration

Pre-show projector calibration

The set of Half Real was not simply a static scene, it contained a door and window that actors could use and walk or climb through, and a chair that was moved about on stage. The detection algorithm in the tracker would sometimes incorrectly register these objects as actors. By marking these areas as dead areas, the system would ignore these objects when associating virtual information. As with catch areas, the dead areas were specifically defined for each scene, as in some scenes actors moved into the range of the window or door when the virtual information needed to appear.

The only real problem I found with the Kinect was a bug in Open NI which caused a segfault. Segfaults are always bad, but when your entire set is a projected environment they are even worse. Luckily, I was able to do a quick fix before opening night.

Interactivity

All the possible paths through the show

All the possible paths through the show

Half Real was an interactive murder investigation. The audience were asked “Who killed Violet Vario?” Each scene would uncover clues about the murder, and the audience voted on what to investigate next. This meant that there was around 6 hours of scenes for an hour long show. A graph of the show is shown to the right.

Each member of the audience had a ZigZag controller, developed by Matthew Gardiner, in their hand. During a scene vote options would appear in the projected world. At the end of the scene the audience would vote on where the investigation would go.

 

This slideshow requires JavaScript.

There was one way communication from the ZigZag voting system to the projection system. The ZigZag system was responsible for communicating with the devices, tallying votes, and keeping track of what scene the show was in. The projection system polled the ZigZag system regularly to find out whether a scene change was necessary via http.

Resource Management

Half Real. Photo courtesy of Chris More

Half Real

All those scenes meant a lot of media resources. Half Real’s projected content consisted of images, video files, and procedurally generated content, such as the vote options. In all, 38GB of assets, mostly video, were used during the show. A level loading approach was used to manage these assets.

Each scene was described in an XML file, which listed the assets required for the scene, transitions and events, etc. A content manager was responsible for freeing data no longer required and loading new assets if required. Assets that were needed in consecutive scenes were reused, rather than reloaded.

void HalfReal::loadSceneAssets(const vector &assetList) {
    // set all loaded assets to unused...
    foreach (Asset &asset, mLoadedAssets) {
        asset.second->inScene = false;
    }
    // iterate over the needed assets
    foreach (string& assetName, assetList) {
        // if it's already loaded, set it to in use...
        if (mLoadedAssets.find(assetName) != mLoadedAssets.end()) {
            mLoadedAssets[assetName]->inScene = true;
        }
        else {
            mLoadedAssets[assetName] = loadAsset(assetName);
            mLoadedAssets[name]->inScene = true;
        }
    }
    // free unneeded assets
    for (AssetContainer::iterator it = mLoadedAssets.begin(); it != mLoadedAssets.end(); ++it) {
        if (!((*it).second->inScene) && !((*it).second->inUse)) {
            delete it->second;
            mLoadedAssets.erase(it);
        }
    }
}

Note the additional flag inUse. Half Real had smooth transitions between scenes. We didn’t want to delete an asset that was currently be projected, so we left it in RAM until the next transition.

Success?

Who killed Violet Vario?

Who killed Violet Vario?

Half Real successfully completed a tour of regional South Australia, before playing a three week, sold out season as part of the Melbourne Festival in 2011. That achievement is proof the technology and software developed was a success. However, as with any production there are lessons learned and room for improvement.

One of the major issues that had to be overcame was reliability and robustness. In Half Real, if the projection software crashed, the stage went dark. The system had to function correctly day after day, for extended periods of time. Decoupling subsystems was one of the most important factors in making the system robust. For example, it was important that the projection system kept running if the tracking system stopped responding.

Another issue was sequencing content to be projected in each scene. The projection system used XML files for each scene. This effectively meant there was one scene description for the projection, and another for lighting and sound.If there was time, making the projection system interoperable with existing stage management software, such as QLab, which would reduce the duplication, and make modifying the sequences of projected content much easier.

While Half Real made an important step in using SAR for interactive performance art, there are many more possibilities to be explored. For example, using projectors to simulate physical light sources, such as follow spot lights that automatically track the actor. Or, using the projectors to project directly onto the actors in order to change their appearance.

Turns out, I did some of these things working with Cirkidz!

Join the conversation!

Hit me up on Twitter or send me an email.
Posted on May 22, 2016Software ProjectsTags: c++, Half Real, interactivity, Linux, opengl, projection, sar, theatre

No writing for a bit because I’m currently travelling around South America! I’ll be visiting Peru, Bolivia and Ecuador. We are writing about our trip over at ummthatway.com if you are interested in what we are up to.

Umm… That Way!

Regular programming returns June.

Cheers!

Join the conversation!

Hit me up on Twitter or send me an email.
Posted on March 21, 2016ThoughtsTags: bolivia, ecuador, holiday, Peru, south america

This post explains how I built a projection mapped, interactive graffiti wall for If There Was A Colour Darker Than Black I’d Wear It. This was built in collaboration with Lachlan Tetlow-Stewart. First, here’s a video of the end result:

Let’s break down the project.

  • Multi projector, projection mapped display onto buildings
  • Allow audience members to send SMS messages to tag the building
  • Animate tags onto wall
  • Video elements

That’s the basics. Oh, did I mention we’re projecting out of a van, powered by generator, in rural South Australia?

3D Projection Mapping

My research involves Augmented Reality using projectors; quite convenient. Here we create a simple 3D model of the geometry to be projected onto, and create our content mapped to the 3D environment.

This differs from most projection mapping techniques, where content is produced for a specific projector viewpoint and mapped in 2D. The advantage of 3D projection mapping is content can be created independently of the projectors. Projectors can be added as necessary, or viewpoints drastically changed, without having to re-author content. In our case, it meant we could project onto the buildings every night, without having to get the van or projectors into exactly the right location each time.

Finding known points on the 3D model in the real world

Finding known points on the 3D model in the real world

The 3D models are actually incredibly simple, just enough detail to allow perspectively correct projections. It does, however, require actually measuring the buildings to make sure the 3D model matches the real world. The calibration process involves finding landmarks in the 3D model (corners) in the projector image using a crosshair. From there, maths takes over and we end up with a correctly aligned projection.

This whole project was just an elaborate way to get the phone numbers of audience members. We would be using them later in the show. We played a video telling the audience to text in their tags, which would then appear on the wall.

The computer system received texts using a GSM Modem, as described in my post about SMS and Linux. The handler put the message into a MySQL database, then an incredibly simple web API allowed the projection system to get new messages as they were received.

Alpha mask for animating in new messages

We used animated alpha masks and OpenGL blending to make the texts appear in a pleasing manner. All lengths of time (time spent visible, etc) were randomised to make everything feel a bit more natural.

The result was a compelling experience for the audience. They got something fun to occupy themselves before the show proper started, and we got their phone numbers for use later in the show!

Technical Details

The projection system was a standard desktop computer with 2x Nvidia GTX560 graphics cards, running Ubuntu 12.04. The software was OpenGL and C++, built on top of a Spatial Augmented Reality framework developed in the Wearable Computer Lab during my PhD.

Text rendering was accomplished using FTGL texture fonts. The software generated pool of fonts at different sizes, so the best fitting font could be chosen for messages of different lengths. The generation on startup is important – generating the texture map of an FTGL TextureFont is an expensive process. Changing the size of a font at runtime will give you serious performance problems.

The correct font size had to be calculated for each message. Here’s my incredibly elegant algorithm for this:

void GraffitiSpot::setText(std::string text) {
   mText = text;
   int index = 0;
   while (!willItFit()) {
      index++;
      if (index == fontPool.size()) {
         break;
      }
      else {
         mFont = fontPool[index]; 
      }
   }
}

So basically it looks through the fonts and chooses the biggest font that fits the message in the space available. What does willItFit do?

bool GraffitiSpot::willItFit() {
   string word;
   float currentLineLength = 0;
   float maxWordLength = 0;
   stringstream currentLine;
   stringstream ss;
   ss << mText; unsigned numLines = 0; FTPoint point(0,0); while (ss >> word) {
      float wordLength = mFont->Advance(word.c_str());
      if (wordLength > maxWordLength) {
         maxWordLength = wordLength;
      }

      // if we are at the start of the line, and it fits.. 
      if (currentLineLength == 0) {
         currentLine << word; currentLineLength += wordLength; } // midway through a line, and it won't fit else if (wordLength + currentLineLength > mWidth) {
         numLines++;
         point.Y(point.Y() - mFont->LineHeight() * LINE_SCALE);
         currentLine.str("");
         currentLineLength = wordLength;
         currentLine << word; } // midway through a line, and it does fit else { currentLineLength += mFont->Advance(" ") + wordLength;
         currentLine << " " << word; } } if (currentLineLength > 0) {
     numLines++;
   }

   float heightRequired = mFont->LineHeight() + ((numLines-1) * mFont->LineHeight() * LINE_SCALE);
   return heightRequired <= mHeight && maxWordLength < mWidth;
}

Each graffiti spot has a width and height that limits how much text will fit. This function simulates rendering the text and does some word wrap. FTGL doesn’t do any text wrapping so it’s up to us, and we’re using variable width graf style fonts. We use mFont->Advance() to calculate how many pixels wide a portion of text is. If a word fits on a line, we move to the next one. As soon as a word overflows the width, we drop down a line. We use mFont->LineHeight() to calculate the Y position. LINE_SCALE is just a line height adjustment because we found the default line height to have too much spacing for what we wanted.

If, after simulating rendering the entire message, we have gone beyond the bounds of the graffiti spot, we return false. The function above then tries again with a smaller font. If you wanted to be clever you would do a binary search to speed up finding the optimum font size, but in practice we never hit any performance problems.

Join the conversation!

Hit me up on Twitter or send me an email.
Posted on January 15, 2016Software ProjectsTags: black, c++, graf, opengl, projection, sms, theatre

Tweaking your own Latex template for a PhD dissertation is a rite of passage/time waster for most PhD candidates. There are lots of templates around on the Internet too, of varying quality.

I spent a fair bit of time procrastinating perfecting the template used for my thesis. I’ve pulled out all the unnecessary bits and put it up on GitHub. Hopefully some other poor PhD student will find it useful. This template is itself based on styles developed by Peter Hutterer, an earlier PhD student from the Wearable Computer Lab.

This template is particularly suited to students at the University of South Australia. It meets the guidelines specified by the Graduate Research Office. That said, with some adjustments this template should be useful for anybody.

Features:

  • Nice cover page
  • Author’s publications
  • Acknowledgements
  • TOC, List of Figures, Abbreviations, etc.

How to use:

  • Fork and clone the GitHub repository
  • Edit the information in thesis.tex
  • Update images/00/author_sig.png with your own signature
  • Update images/00/uni.png with your university’s logo
  • Add your publications as citations in 00-publications.tex
  • Tweak the styles as necessary
  • Write your damn thesis!

I’m happy to accept Pull Requests for improvements on this template.

Check it out on GitHub!

Happy Writing!

Join the conversation!

Hit me up on Twitter or send me an email.
Posted on November 10, 2015ResearchTags: latex, phd, template, thesis

tl;dr; I’ve made a GitHub repo that makes FTGL work on OSX again.

FTGL is a library that makes it super convenient to render TrueType text in OpenGL applications. You can render text as textures and geometry, making it very flexible. There’s just one problem: if you’re using MacPorts or Homebrew on OSX, FTGL doesn’t work! Here’s how to work around it.

FTGL makes use of FreeType to actually render text. In newish versions of FreeType, some of their source files have been moved around and renamed. This is a problem on OSX since, by default, we are on a case-insensitive filesystem. We now have a name clash where both FTGL and FreeType seem to have a file named ftglyph.h. All of a sudden software that uses FTGL will no longer compile because the wrong files are being included!

The fix for this is fairly straight forward. Since FTGL depends on FreeType, FTGL should be modified to remove the name clash. Unfortunately, FTGL seems to have been abandoned, and has not had any updates since 2013. In the bug report linked above I have provided a patch that renames the file and updates references to it. I’ve also created a GitHub repository with the patch applied.

This problem doesn’t show up on Linux because on a case sensitive filesystem like Ext4, the FreeType file is ftglyph.h, while the FTGL file is named FTGlyph.h. No name clash.

So there, uninstall FTGL from MacPorts or Homebrew, clone my GitHub repo, and build/install from source. FTGL will work on OSX once more.

Long term you may want to look at moving away from FTGL in your own software. It is great at what it does, but hasn’t been updated in a long time. It uses OpenGL display lists internally, so will not work on modern OpenGL specs. But at least you can now use it if you need to.

Join the conversation!

Hit me up on Twitter or send me an email.
Posted on September 16, 2015ProgrammingTags: ftgl, osx, Programming