On-set virtual production

Australian film director Clayton Jacobson first had the idea of improving the green screen that was then in use, when filming a TV ad for detergent in 2003. Watching his son playing videogames and seeing the 3D technology used in them gave him the idea. Eventually, in 2016, Jacobson and his son made one of the first prototypes for a virtual production stage in their shed, using a set of LED screens. However, he could not get anyone to take an interest in developing the technology further, so gave up on it. Other filmmakers had also caught on to the idea though, and in 2018 an Australian cinematographer, Greig Fraser, used the technology to film the Star Wars franchise spin-off series, The Mandalorian (released 2019). Instead of using the green screen during the filming stage, the team combined post-production with the production stage of the series. They installed huge LED walls linked to powerful computers which ran Unreal Engine gaming software (used for Fortnite, among others). They called this soundstage "the volume", a term already used to refer to a stage where visual effects techniques take place.[1]

Since its inventive use in The Mandalorian, which used ILM's StageCraft, the technology has become increasingly popular. Miles Perkins, industry manager of film and TV for Epic Games and maker of the Unreal Engine, estimated that there were around 300 stages by October 2022, increased from only three in 2019.[2] Most of these were built during or after the COVID-19 pandemic, when lockdowns meant that production studios had to find ways to produce films without travelling to other locations.[1]

In March 2023, the world's largest virtual production stage was opened at the Docklands Studios Melbourne, in the city of Melbourne, Australia.[1]

On-set virtual production (OSVP) is also known as virtual production (VP), In-Camera Visual Effects (ICVFX),[3] or The Volume.

With careful adjustment and calibration, an OSVP set can be made to closely approximate the appearance of a real set or outdoor location.[4] OSVP can be viewed as an application of extended reality. OSVP contrasts with virtual studio technology, in which a green screen backdrop surrounds the set, and the virtual surroundings are composited into the green screen plate downstream from the camera, in that in OSVP the virtual world surrounding the set is visible to the camera, actors, and crew, and objects on set are illuminated by light from the LED screen, creating realistic interactive lighting effects, and that the virtual background and foreground are captured directly in camera, complete with natural subtle cues like lens distortion, depth of field effects, bokeh and lens flare. This makes it a far more natural experience that more closely approximates location shooting, making the film-making process faster and more intuitive than can be achieved on a virtual set.

To render parallax depth cues correctly from the viewpoint of a moving camera, the system requires the use of match moving of the background imagery based on data from low-latency real-time motion capture technology to track the camera.

Industry organizations including SMPTE, the Academy of Motion Picture Arts and Sciences and the American Society of Cinematographers have started initiatives to support the development of OSVP.[4][5][6][7]

Stages that use OSVP include the various StageCraft stages, Pixomondo's Toronto-based LED stage, which has a longterm lease from CBS or Lux Machina various stages.[2] In Japan, the LED wall and virtual production were used by Toei Company for its Super Sentai shows Avataro Sentai Donbrothers[8] and Ohsama Sentai King-Ohger, with the latter also being produced in collaboration with Sony PCL Inc.[9]

As of March 2023 the largest OSVP is at Docklands Studios Melbourne in Australia.[1]