The following article was published in COSMOS Magazine Issue #93.
This article was written by Amanda Yeo, she is a Sydney-based writer and co-creator of tech podcast Queens of the Drone Age.
On the night of 15 April 2012, American rapper Tupac Shakur appeared to grace the stage at the Coachella Valley Music and Arts Festival in California – 16 years after the artist was killed in a drive-by shooting. Performed in front of 90,000 people, Tupac’s holographic set was the work of American visual effects production company Digital Domain.
“It was a tribute to him,” said Digital Domain’s then-chief creative officer Ed Ulbrich. “No one was making money off him. It was done with the blessing of Tupac’s mother.”
Virtual Tupac quickly went viral around the world, garnering cautiously awed reactions. While some expressed concerns about the ethical implications surrounding the animation of a dead celebrity, others excitedly speculated about what such holograms might mean for the future of entertainment. Yet almost a decade later, holographic performances remain relatively novel – and how they are created is still a mystery to many
What we’re talking about when we talk about holograms
From Princess Leia’s plea in Star Wars: A New Hope (“Help me, Obi-Wan Kenobi…”) to AI assistant Cortana in the Halo video game series, holograms have been a staple in science fiction for decades. But our understanding of exactly what constitutes a hologram hasn’t always been the same. While it was an impressive use of technology – and no doubt influential – Digital Domain’s Tupac hologram wasn’t even technically a hologram in the strictest sense of the word.
“The term ‘holography’ was originally coined to describe a particular technique to record images that would also provide different perspectives from multiple viewing angles,” explains CSIRO’s Craig James. A senior researcher at CSIRO’s Sustainable Mining Technologies group, James helps the organisation’s research teams develop extended reality experiences such as “holograms” for educational purposes.
“Today, the use of the term ‘hologram’ is more widely used to describe technologies where a viewer can see an image of a subject that isn’t physically there, without visible electronic or projected screens,” James says.
“While the marketing term is called ‘holographic’, at its core, the projection of a fabricated performance is still two-dimensional in nature,” says Aruna Inversin, a creative director at Digital Domain.
“The performer looks holographic in part because of its juxtaposition with a real performer on stage.”
These holograms can’t quite match the fictional hologram technology dreamt up in sci-fi classics, which project images onto thin air. However, the invisible screens used to create our real-world “holograms” make them the closest we’ve managed to achieve.
Putting on a show
Advancements have facilitated considerable innovation in hologram generation, however the most commonly referenced method remains the use of Pepper’s ghost.
Named after the British scientist who popularised it in the 1860s, Pepper’s ghost is an optical illusion created by projecting an image onto a transparent, reflective screen such as a pane of glass. When this screen is erected between a stage and its audience, both the reflection and the live scene behind it are visible to viewers (see box, bellow).
“The original illusion used half-silvered mirrors that allowed a ghostly image to appear beside a live performer, “says James.
“Current performances typically use tightly stretched, transparent films and high-resolution, laser-powered projectors that provide very crisp, true-to-life images while still allowing audiences to see the background stage details.” This more modern method is currently used by both Digital Domain and BASE Hologram, the production company behind performances such as The Whitney Houston Hologram Tour in 2020 and Jack Horner’s Dinosaurs: A Holographic Adventure in 2019.
“While Pepper’s ghost and variations on the technique do not provide a true hologram that can be seen from all sides, the audience is typically far enough away from the stage that a single perspective is sufficient to preserve the illusion that a holographic performer is present on the stage,” says James. “The screen being used is also typically arranged so that other ‘live’ performers on the stage are able to interact by walking in front of or behind the ‘hologram’.”
“The general idea of Pepper’s ghost has remained the same, but the technology has drastically improved,” says Inversin. “For some of the most well-known holographic presentations, the display is relatively easy. The real trick is creating the performer, which requires skilled artists.”
Crafted for the stage
Not all videos can be turned into convincing holograms. Though archival footage provides reference material for creating digital performers, the video typically isn’t suitable for turning into a hologram itself, and not just due to resolution issues. For a successful holographic display, the person or object needs to be filmed while isolated on a blank backdrop, ensuring background elements aren’t also projected into the scene. Further, the lighting in the recorded footage needs to be matched to that of the live stage in order to maintain the illusion.
“It wouldn’t make sense to show a video of someone in full sunlight on a stage at night, for example,” says James.
“The projected video for holographic concerts has to be created from the ground up specifically for this purpose. Actual actors form the basis of these productions, in combination with motion capture, CGI and other production techniques,” says BASE Hologram CEO Brian Becker.
Footage of a hologram’s subject is created by filming a body double in front of a greenscreen, as well as motion-capturing their performance. The video of the actor is then augmented with a digital avatar of the subject, puppeted by their motion-tracking data. The combination of these elements creates a realistic video which can then be projected onto a transparent surface to turn it into a “hologram”.
“For the Tupac Coachella performance, we started with a live stand-in, who matched Tupac’s physical build and could replicate (to a degree) his musical prowess and stage performance,” Inversin says. “We then recorded him against a greenscreen and meticulously tracked his head.
“For this type of performance, our team pairs our facial capture technology with a head mounted camera to capture the stand-in talent’s facial performance. The actor can then drive the digital head.
Our artists then created a digital version of Tupac’s head using reference material and matched the digital version with the performer. From there, additional facial animation can be applied to serve the creative needs of a performance.”
Deeper and faker
Deepfake technology appears to be the perfect match for hologram performances. Using machine-learning AI, this technology can create deceptively realistic computer-generated images that are indistinguishable from reality to the average observer. Even so, deepfakes haven’t yet become a widespread tool in hologram production, which still largely relies on artists’ “analogue” efforts.
Still, machine learning is beginning to creep into the hologram industry, with Digital Domain recently introducing their tool Charlatan.
“Charlatan is a face-swapping tool that uses machine learning to help create realistic digital versions of characters,” says Inversin, noting that it was deployed to digitally recreate NFL coach Vince Lombardi for Super Bowl LV.
Digital Domain fed Charlatan videos of Lombardi, which the machine-learning tool used to create a digital model of the coach. Lombardi’s avatar was then animated with motion-captured facial movements and overlaid on live action footage of an actor.
“Charlatan allows our team to use 2D neural rendering layers to create facial performances, without using an existing digital model,” says Inversin. “In the case of Lombardi, we were able to use 2D reference images as a starting point. Live action performers are then augmented with the new face.”
Honouring the dead, celebrating the living
Though hologram performances are most famously used to recreate an artist who has died, this is not always the case. ABBA’s upcoming Voyage concert plans to utilise holograms to make the Swedish pop group appear as they did in the 1970s, at the height of their popularity – as well as save them from having to physically perform on multiple nights.
Fortunately, creating a hologram of a living person is much more straightforward than rendering one who has died. This is because rights issues in such cases are less complicated, and any reference material can simply be created.
“In the ABBA show, the performers were available to provide motion capture data to guide their on-stage ‘holograms’,” says James.
“While the holograms were digitally crafted by a team of artists, the motions and voices represented on stage would be drawn directly from the singers’ pre-recorded performances.”
Despite this, hologram concerts are best known for channelling deceased entertainers. Becker stresses that he doesn’t consider BASE Hologram’s work to be “bringing back” dead artists, but rather paying tribute to and celebrating them. Both he and Inversin note that their performances are created with the collaboration and approval of those who own the rights to the recreated person’s image, such as their families or estates
“Every attempt is made to be authentic and true to the artist, with some modifications to how the artist may perform in today’s day and age,” says Becker. “The main interest in these types of shows are to preserve the legacy of these one-of-a-kind musicians and give fans the chance to see them perform – many for the first time.”
Inversin says it typically takes Digital Domain six to nine months to put together a show. The size of the creative crew can stretch into the hundreds, with modellers, animators, riggers and deep-learning engineers just a few of the teams needed to create the digital performer.
“Creating a virtual human requires a lot of people, regardless of whether it is focused on just a head replacement, as with the Tupac hologram, or if it needs a full digital performance,” says Inversin.
An approaching mirage
Almost 10 years after Tupac’s Coachella performance, holographic concerts are still an emerging medium, but one with significant potential.
“When international movements are restricted, or if there are personal reasons that limit performers’ availability, this form of event could provide an option that would otherwise prevent staging a local performance,” says James. “We already have instances where audiences watch remote stage performances through large screens, so ultimately it would be up to the fans of the performers to determine if holographic events give a convincing concert experience.”
A hologram isn’t the same as the real thing. Still, it might get close enough.
This article was written for COMSMOS Magazine Issue #93 by Amanda Yeo, she is a Sydney-based writer and co-creator of tech podcast Queens of the Drone Age. Her most recent story for COSMOS Magazine, on drones, appeared in Issue 91.
COSMOS magazine is Australia’s only dedicated print science publication. Subscribe here to get your quarterly fill of the best Science of Everything, from the chemistry of fireworks to cutting-edge Australian innovation.
RiAus Education have created a STEM learning resource collection for this article. Learn more HERE.
