Stories in code
Come, Film Makers, Come. The Game Engine Is Afoot!
Making films with game engines? Yes, it’s true. Just ask the folk at Weta Digital.
When asked about making automobiles Henry Ford famously never said: “If I had asked people what they wanted, they would have said faster horses.” This quote is trotted out (pun intended) frequently in discussions about innovation, particularly disruptive innovation. The argument often becomes one about whether businesses should listen to their customers. But this misses the point — innovation is about something a customer probably has no knowledge about because it hasn’t been invented yet!
An innovator should have understanding of one’s customers and their problems via empirical, observational, anecdotal methods or even intuition. They should also feel free to ignore customers’ inputs. (Vlaskovits, 2011)
Ask a film maker what they want and (apart from funding), they might answer, “a better camera”. A game engine is probably not on the list. Increasingly, as will be discussed in this piece, perhaps it should be. As the cinema camera emerged at the same time as the automobile, game engines are being developed in parallel to online platforms. The cinema camera catalysed a whole new approach to story creation and storytelling and the growth of a whole new industry. The game engine may well do the same.
A game engine is a software framework for the development of video games. Core functionality typically includes: a rendering engine for 2D and 3D graphics; a physics engine for collision detection, physics-based animations and behaviours; keyframe animation; audio; networking and streaming; localization support; basic video creation and editing for cinematics; artificial intelligence; and, an integrated programming / scripting environment. A game engine is also one of the few truly multiplatform frameworks, allowing outputs to desktop / laptops, VR/AR, mobile devices (phones / tablets), gaming consoles (PlayStation, Xbox, Nintendo) etc. As the utility of this functionality has come to be recognised outside gaming, the list of capabilities is being expanded to accommodate architectural and industrial design, education and training, and film making, just to name a few.
To say that a game engine is just an integrated development environment for building games does not do the technology justice. It merely indicates its point of origin. A game engine is a development environment for imaginations. Here’s why…
It was early 2012. I was working at the BBC on a small project called the London Olympics. We were building a new OTT platform for delivering what was touted as the first “digital Olympics”. I was pair programming with a colleague when he popped a browser window up on his other monitor and said, “have you seen this?” I looked over to see the website for a company called Unity3d. The page had a big colourful image and a button to download an application called a game engine. As we read through the list of features describing what this thing claimed it could do the Earth tilted slightly on its axis.
As a programmer and a film maker I saw my passions converge and an astonishing future present itself. I could make films using the same iterative methodologies I use to make software. I could take the powerful but static storytelling medium of film and combine it with the dynamic power of algorithms and experiment with story form. And do it in real-time!
At the time, I had an idea for something I called in-camera VFX, where. instead of having an empty green screen, you could create the story world programmatically in the game engine, wire the game engine up to a camera and have the camera movement track and control a 3d projected world. This would ensure that things such as parallax, perspective and lighting were correct. You could then place the actors in that virtual world and film them as if they were on location.
This approach would reduce the risk inherent in the hit and miss production pipelines where things frequently didn’t quite match and had to be fixed in post-production. It would give actors their missing eyelines back and directors would be able to actually see what they were shooting. Clearly, I wasn’t the only one thinking this way.
When I got home that evening, I downloaded Unity3d and started playing…
Cue the scratchy sound of the needle being dragged across a vinyl record.
Don’t get me wrong, all the potential I thought could be there was. But that’s all it was, potential. The graphics were awful; and the audio…erk… None of it was near the fidelity required for a film. Coupled with that the knowledge required about shaders, modelling, physics, animation etc. and the resulting learning curve was daunting. And they were still working out which scripting languages to support…
There were other game engines around at the time, including Unreal Engine 3 and Crytek’s CryEngine (which was later used by Amazon Web Services (AWS) as the base for their game engine Lumberyard). Each had their strengths and weaknesses in relation to rendering graphics, animation, audio, and coding. But we were a long way from making anything that would pass muster as a film.
Significantly, in 2016 an event occurred that probably slipped past most people unnoticed. Unity3d released a short film made entirely in their game engine. The film is called Adam (Unity Technologies, 2016) and it was a clear indicator of how far and fast the technical capabilities of both computing hardware and game engines were progressing.
Roll forward to the present. The analogue record player is gone, replaced by a cloud streaming service and an increasing number of films and TV series are being made using game engines somewhere in their pipeline. The innovations that have had to occur in the intervening years to enable this are too numerous to list, but developments in computer graphics through deep learning / neural networks and developments in GPU technology are central. This has enabled the game changer — real-time creation and making.
The Star Wars spinoff The Mandalorian created by Jon Favreau for the streaming service Disney+ in 2019 is one of the most documented examples of a virtual production pipeline using game engines. He talks about this in the video below.
Are game engines in film just a fad or a fringe experiment that will pass? We’ve had a lot of hype about technological innovation such as 3d films in the past. Why is this any different? Here is a question to answer a question:
What do BMW, Tencent, Pokémon Go creator Niantic, movie director Jon Favreau and construction giant Skanska have in common? (Peckham, 2019)
The answer is that they all use a game engine in their design and development pipelines. Previous fads in film such as 3d were about audience experience, not production methodologies. They came and went because audiences voted with their feet.
A game engine is a making methodology, not an audience experience. As Adam, the Mandalorian and the myriad other films being made demonstrate, the audience can watch them as they watch any film.
Also, industry is putting its money where its mouth is and investing heavily in these technologies. This brings us to the reasoning behind the timing of this story. Two large players in the film and game space made significant announcements this month.
Firstly, Netflix formally entered the gaming realm with the announcement of the release of their new gaming platform.
Just like our series, films and specials, we want to design games for any level of play and every kind of player, whether you’re a beginner or a lifelong gamer. (Verdu, 2021)
A week later, Unity3d announced that it had just bought Weta Digital, the New Zealand based company responsible for the Lord of the Rings trilogy, King Kong, Ghost in the Shell and many others.
By combining the industry leading VFX tools and technical talent from the incredible team at Weta, plus the deep development and real-time knowledge within Unity, we aim to deliver tools to unlock the full potential of the metaverse. (Whitten, 2021)
Note that reference by Unity to the Metaverse. With these two announcements sitting atop what Epic Games, Facebook, Microsoft, Nvidia et.al. are working towards, the idea that this is a fad can be put to one side. As Venture Capitalist Matthew Ball states, the
Metaverse has become the newest macro-goal for many of the world’s tech giants…The tens of billions that will be spent on cloud gaming over the next decade, too, is based on the belief that such technologies will underpin our online-offline virtual future. (Ball, 2020)
Making films with a game engine is not going to appeal to all film makers and there are many who I have spoken to who are openly hostile to the idea. The game engine is not a replacement for conventional film making, but as a multiplatform framework, it does allow cross-over.
As the cinema camera showed us the moving image 120 years ago, the game engine shows us a real-time interaction in the moving image. In doing so, it provides the first set of tools for a multiplatform, real-time world where the possibilities for innovation with story form in the moving image are just starting to reveal themselves.
The video above is a short walkthrough I created for a talk in late 2019 to show what real-time making in game engines looks like. It includes setting up virtual cameras and post-production effects — lighting and colour grading. It was done in Unreal Engine 4. It is unedited to be really real-time and to show the things that are usually edited out of sizzle reels. I hope it captures something of the experience for those who have never seen the like!
— Please note that it has no audio as I was originally talking as it played.
Stories in code is made for sharing. If you enjoyed this article, please show your support by giving it a clap, following me and sharing with others who you think might be interested.
References
Ball, M. (2020). The Metaverse: What It Is, Where to Find it, and Who Will Build It [online]. Available from: https://www.matthewball.vc/all/themetaverse [Accessed 13 November 2021].
Peckham, E. (2019). How Unity built the world’s most popular game engine. TechCrunch [online]. Available from: https://social.techcrunch.com/2019/10/17/how-unity-built-the-worlds-most-popular-game-engine/ [Accessed 12 November 2021].
Unity Technologies. (2016). Introducing Adam — A Unity Short Film Rendered in Real Time [online]. Available from: https://unity.com/demos/adam [Accessed 13 November 2021].
Verdu, M. (2021). About Netflix — Let the Games Begin: A New Way to Experience Entertainment on Mobile [online]. Available from: https://about.netflix.com/, https://about.netflix.com/en/news/let-the-games-begin-a-new-way-to-experience-entertainment-on-mobile [Accessed 13 November 2021].
Vlaskovits, P. (2011). Henry Ford, Innovation, and That “Faster Horse” Quote. Harvard Business Review [online]. Available from: https://hbr.org/2011/08/henry-ford-never-said-the-fast [Accessed 13 November 2021].
Whitten, M. (2021). Welcome Weta Digital! [online]. Available from: https://blog.unity.com/news/welcome-weta-digital [Accessed 13 November 2021].
