Battle Vortex Project Page
Battle Vortex is an experimental game concept in which the player uses smart glasses to find science-fiction style "time portals" at historic sites, showing a window into the past and bringing ancient battles to life once more. This project is a collaboration between Ogglebox Sensory Computing, Leicestershire County Council, and Bosworth Heritage Centre. Our aim is to establish the use of 3D smart glass technology in a cultural heritage context, and explore the potential of new technologies to bring new audiences into museums. This pioneering work is funded by Arts Council England.
Follow our progress on this blog as we develop the concepts and use 3D smart glass technology in brand new ways.
An exciting day for all concerned: this is the official project start and the first project planning meeting between Ogglebox, Bosworth Heritage Centre, and Leicestershire County Council. We had initial discussions on the opportunities for gathering 3D film of characters in medieval costume which will form part of our ‘window into the past’; and had our first full tour of the Bosworth Heritage centre site so we could start to think about areas we could augment with our virtual ‘time portals’. Most importantly we decided on times to review our game concepts with groups of ‘creative connectors’ (16-25 year-olds) and ‘silver champions’ (55+) - key groups which museums are interested in attracting.
The next step will be a meeting with key staff at the visitor centre to give their expert knowledge about the visitor experience, and a storyboarding session to create a narrative for the game.
Today we met with the historical experts to design a storyboard and accompanying script for the game concept. These have two roles within our project. Firstly, they act as a guiding narrative for the game itself. This will be useful later when we’re designing the software to run on the smart glasses. But our project plan also involves using a 3D camera to capture reenactments of medieval battles on 3D video, for use as some of the augmented content.The storyboard therefore plays the role of a conventional film-maker’s tool to help us get the correct framing of shots when filming on-site.
Here are a couple of panels from the first scene. The viewer gets an inside view of a medieval battle encampment, and is shortly joined by a solider who tells them to get a move on and make their way to the battlefield.
Design of Time Portals
We had already decided that the viewer should access scenes from the past by finding ‘time portals’ around the heritage centre site when looking through smart glasses. The idea is that a time portal should be a mysterious-looking object, like something out of science fiction. During today’s meeting we talked more about how visitors might best interact with these. It was seen as important that we should be able to re-trigger a time portal and see the relevant scene again, but without being able to do this by accident. Ogglebox suggested that a viewer would need to be looking in the direction of a time portal for some period of time before it triggers, thus introducing a sort of hysteresis into the system. Further, the portal can give a visual indication of how close the elapsed time is to the required amount for triggering. We suggested that the portal ‘shimmers’ with increasing strength up to the point of being triggered. With the structure of scenes as laid out in the storyboard, there are scene titles which would help the viewer to understand each scene in its historical context. We decided that we can make use of the portal triggering system to allow the portal to display a title prior to triggering, so that the viewer is aware of this.
Plans for Filming
Since this is an experimental project, we are not going to the time and expense of a full-scale 3D filmshoot with actors. Instead, we plan to capture 3D video of the medieval battle reenactments which regularly take place at the heritage centre. By far the largest of these is the reenactment weekend which takes place on 16th/17th August. The centre kindly offered to recruit reenactors to star in scenes involving actors. A final scene will feature one of the battlefield guides from the heritage centre. We will also take scenes from the battle sequences - necessarily taken from a distance for safety reasons!
Real World vs. Augmented World
An interesting thing about designing content for augmented reality games is that we design not only the synthetic content but also its relation and interaction with the real world. This is particularly important for projects like this one, because we intend to encourage the viewer to make connections between the battlefield site as it is today and the events which took place there centuries ago. We made a further tour of the site and identified suitable locations for placing content. Key scenes with King Richard III are to be located in the direction where the actual battle is believed to have taken place.
Another key event for the project today. For the first time we presented the concept to groups of candidate museum-goers to get their reaction. We showed them the storyboard, script, and locations for augmentation, and talked them through the game concept. We gave them a chance to try out the smart glasses for themselves.
Since the content for the actual scenes has yet to be created, we experimented with a low-tech version of augmented reality: we printed the storyboard onto transparencies. The audience could hold these up to the relevant location to view content in-situ.
Feedback was overwhelmingly positive, with some new and useful ideas being contributed too. We also had a couple of possible endings for the script, and the groups helped us decide which one to use.
There was a good discussion about practical issues associated with smart glasses. This informs some of the guidance materials which may be presented to visitors when trying this concept for the first time.
In preparation for gathering 3D content at the reenactment weekend, Ogglebox’s film crew visited the site today. All looks good, but they learnt there will be just one opportunity to get a 3D video of King Richard’s desparate last horse charge. No pressure people!!
This was the weekend for capturing the 3D video to be used as part of the time portals shown on the smart glasses.
Capturing the battle sequences was challenging in many ways. Although the reenactment has a structure, the action is largely improvised and unchoreographed, making it highly unpredictable; and the light was changing a great deal. Plus the audience would have a nasty habit of getting in the picture the whole time! Nevertheless we have gathered plenty of material we can use, and it will be down to the edit to make the most of it.
Choreographed sequences with reenactors and a battlefield guide performing scripts to camera were much easier to manage. A huge thank you to all actors involved - you were brilliant!
Work on the software and content for the game continues apace. Most recently we’ve been working on a 3D compass which will be a key element of the ‘treasure hunt’ nature of the game. We wanted to create a clean design to be easy to understand, but still in keeping with the historic theme. We were inspired by this artefact from the National Maritime Museum at Greenwhich which is from the century following the battle. It has a rotating disc of the form reportedly found on nautical compasses right back to the medieval period. We elaborated this concept with a 3D effect for the cardinal directions (North, South, East, West) and inter-cardinal directions (North-east, South-east, etc), but used minimal surface decoration to keep a clean feel and avoid confusing players of the game!
The image above shows a draft of the compass with separate images to be rendered for left and right eye. You can see that the images are subtly different which is how we create the illusion of a three-dimensional object when seen on the smart glasses.
The compass is animated by data from sensors on board the smart glasses so that it behaves like a real compass. Here we found an unexpected quirk of the sensor orientation on our chosen smart glasses. In case it’s useful to other developers we’ve put some comments about this in a separate technical note here.
A requirement of the game is that the players are directed by the compass to the next location of interest. As shown in the image above, we’re doing this graphically by highlighting the segment of the disc in the required direction; and with text naming the required direction, which floats next to the disc.
This is a game with pretend time portals. But what should a time portal look like? What do the greats of science fiction think they should look like? In The Time Machine, published in 1895, H.G. Wells wrote:
“...the little machine suddenly swung round, became indistinct, was seen as a ghost for a second perhaps, as an eddy of faintly glittering brass and ivory; and it was gone--vanished!”
The idea of fictitious time travel appearing like an eddy or vortex continued in later works on film and television in the 20th century, such as Dr. Who. This might reflect the idea that some scientists have suggested ‘wormholes’ (warping of space-time) as possible time portals consistent with the Einstein field equations. Some more recent work on their possible appearance can be found here. An internet search for images of time portal shows that the idea of time portals glittering and having filament-like appearance of plasma also seems common, perhaps in reference to the high amounts of electromagnetic energy which may be associated with a wormhole, ionising the gases around it.
So to fit with some of the best science fiction tradition our time portal should appear:
- to contain plasma-like filaments
- like a vortex
- ghostly and glittering
Images show that plasma filaments occuring on earth in nature, such as in the aurora, may appear light blue in colour wth a purple-ish halo. This is also in keeping with artificially created filaments in a plasma lamp.
The light is caused by moving electric charge, so should be seen to move along each filament. We wrote software to show a depiction of this using 5 phases of development. We make no excuse that these are intended to look pretty rather than corresponding precisely to any scientifically valid model of plasma! The phases are:
- Genesis: a localised light-emitting region becomes visible.
- Activation: the region grows at roughly constant speed in a given direction, increasing in light intensity as it does so.
- Propagation: once a critical strength is reached, the growth accelerates, as does the strength of emitted light.
- Decay: after a certain length the filament becomes subject to deceleration. Growth slows and intensity decreases until only a faint halo remains.
- Extinction: all light from the filament disappears.
To give something of a glittering effect, the growth is then subject to random deviations, which one might think of as turbulence in the medium. The images below show depictions of a linear filament as it evolves. In this example a deviation in the vertical direction appears due to simulated turbulence during the propagation phase.
We assume that in a time portal filaments appear randomly at a set average rate of creation, and use our software to simulate these. Finally, we deflect the growth of filaments into a spiral vortex prior to display. An example screenshot of a portal produced in this way is shown below.
When activated, the vortex and filament creation of a time portal will intensify, and eventually it will transform into a scene from the past, which will be the topic of future posts.
With key design elements established, the next task was to assemble them into a suitable game architecture, and one that is relatively easy to scale, maintain and adjust. The application is unusual in that the primary input from the user is their location and orientation in the world. From a computer-programming point of view we are basically using the person as a giant 3D computer mouse! (Perhaps mouse is the wrong word when we’re talking about a person?)
Location controls which portals can be seen and when they can be activated. Orientation controls what is shown on the graphical display, such that it remains fixed relative to the world.
In this first application, gameplay is linear, so once one portal is activated the next can become visible. We noted from comments in feedback sessions that it is important to allow a previously activated portal to be reactivated, in case the user missed the content or wants to show it to a person standing near them.
Our developers created the core game engine using Java, with important game parameters being specified through a configuration file. The latter is needed for rapid development because it allows quick tweaks of the game via USB connection to the smart glasses, without the need to reinstall the app.
One interactions were enabled, some additional design elements could be built as described below.
Additional element #1: portal activation
Portal activation occurs by manipulating the portal vortex to become more intense as the viewer continues to look at it. This is done by steadily increasing the rate of plasma filament creation, and also increasing the speed of the vortex. When a vortex reaches a critical intensity, a title is superposed on it. The title is simply a graphical image which has been preconfigured. The set of graphical images to use are specified in the configuration file.
Additional element #2: 3D video with approximate variable viewpoint
After a portal has been activated, the display switches to show 3D video with a scene from the past. We would like our viewer to feel as much a part of the scene as possible. To perform true variable viewpoint video would require a full 3D model of video content, filmed from multiple angles. This was beyond the budget of this initial project. Instead we took an approach in which video action as assumed to occur at a nominal depth, and so the entire video scene is modeled as a plane. As the user looks around, the relative location of this plane compared with the viewer is adjusted. Of course this simplified model will become less believable the more depth there is in the video. Nevertheless it provides a sense that the scene is external to the glasses, not simply fixed to the glasses reference frame.
The software on-board the smart glasses is being built to show portals at particular locations. Ultimately we would like to give heritage sites the ability to set such locations for themselves; and perhaps choose partiular ones to tie in with events going on. We also found that we needed a quick way to adjust portal locations when setting things up.
We decided to ‘bit the bullet’ and make a Portals web application for doing this quickly. An early implementation can be found in the technical notes section at http://www.ogglebox.com/technotes/portals/.
Until this time we had been hand-coding locations, which was getting pretty hard work! With this new method, the web application automatically outputs a text file which can be transferred to the smart glasses by USB to update the portal locations.
A screen-shot from the web application is shown above. The example map and portals are from a nearby park. Of course we’re not going to tell you where the Bosworth portals are hidden - that would spoil the fun!
The round dots indicate desired portal locations. The turqoise shaded areas near each one indicate the desired range of visibility, that is to say the area where the user must stand in order to be able to see a portal when they look towards it. There are smaller areas, marked in red, which show desired range of triggering. The idea is that when the user is in a triggering region and looks towards a portal, then it will begin to trigger, meaning that the plasma vortex of the portal will swirl with increasing vigour, until it transforms into a ‘scene from the past’.
The portal locations and regions can all be adjusted via the Portals web application. The portal in the centre of the above image is in the process of being edited, which is why it appears with extra round blobs. These are function as ‘handles’ to adjust the portal using the mouse.
Our application requires good GPS signal to locate the glasses and allow the user to discover time portals. After much testing we found that the on-board GPS system of the Moverio BT-200 was rarely able to function. This may be because of the frequent overcast skies here in the UK, though even some relatively clear days caused difficulty. We tested it alongside iPhone GPS (with WiFi turned off so that it used GPS only), and found that even when the phone was functioning well, the BT-200 failed to obtain a signal. In addition to testing with our own software, we made use of the publicly available GPS Status & Toolbox Android application which will run on the smart glasses. It reported the same difficulty in obtaining a position fix.
We decided to try an external GPS module. We purchased one of the popular Dual XGPS150 bluetooth modules. Ours is the XGPS150E, where the E stands for ‘Europe’. Each module has its own serial number reported through bluetooth, so they can individually ‘pair’ with different glasses. This is important as there can be multiple users near each other and we didn’t want the glasses confusing one GPS module with another.
The module pairs successfully with the smart glasses, and shows good satellite visibility in the GPS Status & Toolbox, even when the on-board GPS may not be reporting any visible satellites at all.
Software integration with our game was relatively straightforward. We found that we could obtain a connection socket to it and simply read GPS coordinates as lines of text. Source code can be found in the technical notes here.
The device functioned succesfully within the game to provide GPS location as required.
The system has been delivered to the site. We spent a day there with heritage centre staff, and although it was good to see it being tried out, it was ulimtately a frustrating day for all concerned.
The team were able to see some portals and give initial feedback on content. The sequence of interactions seemed to work well and we’re hopeful of an engaging experience for users.
There were several issues with the technology that need to be resolved before this can be called a functioning system. Some of these are about user experience. Others are more fundamental, and had not shown up in our off-site tests.
User experience: lessons learned
Even when the user is in the correct place to see a portal, it takes too long to be able to find it; and the guides were concerned that visitors may not feel sufficiently engaged due to this delay.
At present there is no way to tell that the device is switched on and functioning (once it has started up). When users did not immediately find a portal, this caused them to question whether the system was working properly (even when it was) - and in at least one case whether their own eyes were working properly!
It appeared that for some there was a perceptual issue - when presented with bright surroundings the relatively low contrast of the portals made them difficult to see.
To resolve points 1 and 2 we think a new system of on-screen arrows could help. The arrows could point in the direction of the next portal, and the colour/appearance of the arrows could be different depending on whether the portal is in visible range. Also at the moment portals ‘fade in’ which means that when looking in the correct direction they still take time to appear. This was by design and can be changed.
Point 3 can be resolved by using the darker version of the sunshades which are included - but whoever administers the system to visitors would need to be aware of this.
Technology: lessons learned
We know that the system relies on consistent orientation information being generated using on-board gyroscope and magnetometer. Unfortunately, this system appeared unstable when tested on-site. Whilst we had thought early location/orientation problems were purely due to GPS, it seems this is not always the case. This had not been picked up in our off-site tests using the development unit.
More work is needed to track down and remedy the cause of this issue.
After our experiences with users of the system, we introduced an arrow system to help guide users to portals. Our aim is to avoid doubt in the user’s mind as to whether they are “doing it right” without destroying the enjoyment of a treasure hunt style game. So the arrows point the user in the correct direction without giving away the exact location. There are different colours of arrows depending on whether the user is close enough for activation. The figure below shows the ones which appear when close.
Despite a major rewrite of sensor fusion code to add many diagnostics, there were still issues during what was supposed to be our final installation trip to the site. As a result we added a simple magnetometer diagnostic check which appears on the visual display at start-up.
This allowed us to trace the problems to a magnetometer fault in one of the test units, and this was replaced. It was decided to make this diagnostic a permanent feature of the startup procedure to avoid issues like this happening again.
The day had finally arrived - time for a final test of the system on-site.
Walking the trail
We have built, to our knowledge, the first real-world demonstration of the use of wearable 3D augmented reality display in a cultural heritage context. The overwhelming response from those who tried the final version was one of increased engagement with the history of their surroundings, which was our main aim. We learnt a lot from the development process and there is a lot more to learn still.
Initially there were technology issues which were overcome by additional hardware (GPS receivers) and software (compass diagnostics).
We have shown an initial palette of user interface elements for interacting in the mixed reality world. It is clear to us that this project has only scratched the surface of what is possible there. In the latter stages we were able to get more detailed feedback about user experience which guided us towards introduction of additional user interface features such as arrows guiding users to the time portals.
The practicalities of providing glasses to museum or heritage site visitors should not be underestimated. During testing/feedback there were a total of four site visits to present incremental developments to staff and gain feedback which was used in an iterative development process. In a future project we would aim for even more frequent and regular contact time.
We found that giving glasses to a group of people makes for some interesting mixed-reality social interactions, with some responding to what they see from the display and others responding to elements of the real-world. There is great potential in future work to make this duality an integral part of the gameplay.
By showing that these kinds of application are within reach of modern technology, we believe there is scope for conducting projects such as this one on a much bigger scale, and with even greater user involvement in the interface design and gameplay elements.