Indiana Jones and the Autonomous Vehicle

How did they use an autonomous vehicle in the new Indiana Jones movie to create the re-aging effect?

Well, now that I have your attention, I will share the rabbit hole I have gone down after seeing Indiana Jones and the Dial of Destiny.

Maybe it is my unique experience having worked in the film industry and in automotive, but as I investigated the anti-aging film production techniques, I started to see some parallels to autonomous vehicle systems.

Starting With the Hardware.

The typical sensing rig for an autonomous vehicle includes:

· Traditional Cameras for imagery and video

· Ultrasonic sensors

· Infrared cameras and sensors

· LiDAR

· Radar

Now let's look at the hardware rig used for Face Re-aging for film creation and compare the two technologies. You must use a little imagination here, so be open minded and look for the correlations.

The new re-aging camera rig includes:

• Infrared cameras

• LiDAR

• Traditional cameras

This is all speculation on my part, but the automotive world has been applying these techniques for over a decade now. There is a lot of proof and data captured to demonstrate technology and mathematical viability. In this instance the automotive industry is leading in the application of these technologies and creating maturity in knowledge which is enabling broader adoption in other industries.

Why Do They Need Infrared and LiDAR for De-aging?

Imagine how much data needs to be captured for dynamic movement through a world of light. You must have spatial understanding with rich detail on the machine's or human's form and how this form reacts to light within it's spatial atmosphere.

Now, when making films they begin this process by scanning the film set with LiDAR. Then, using high dynamic range images (HDRI) the set is "filmed" with high dynamic range to capture the maximum and minimum light intensities in the environment of the shot. In addition to this, infrared cameras have been added to the new rig to capture the facial performance detail of each actor needed for Re-Aging process. This is not done with one rig, by the way. There are multiple rigs and racks of hardware to support this new process. The goal was to not burden the actor with wearing any extra gear so they can perform naturally.

Here is the typical process flow for de-aging for a two-actor conversation:

• HDRI and LiDAR are combined and the location and movement of the cameras are established as well as the intensity and color temperature of all the lighting on set.

• The infrared illumination information is processed by the Facial Lux software and compared to the HDRI and LiDAR profile as shot for alignment.

• The digital doubles of the actors are then added, and they are deformed on a per-frame basis to match the Flux solution of the actors’ actual faces.

The pictures show the infrared camera output on the left. This is then used to create a 3D

model of what was shown on the set. Facial LUX can see how the face is shaded, the subtle nuances in how the light is interacting with the actor, and then translate that into detailed information that captures the little twitches, imperfections, and little wrinkles to provide a lot of information directly to the plate. A plate is a term for the final shot btw.

Much like the sensor suite in an autonomous vehicle. Infrared cameras, radar, LiDAR, and ultrasonic sensors are working together through sensor fusion to create the details of the world dynamically changing as the vehicle through the spatial world.

Now think about the dynamic world of vehicles in the context of a machines view of the "world of light". They create "cognitive awareness" of the world around around them through their sensor and algorithmic suite. They make decisions based on this perception and understanding of themselves interacting with light (safety, guidance, awareness). These sensor and camera arrays enable much the same light performance and spatial awareness now being adopted in film production. The depth of understanding of how light performs in our world is the challenge that a machine must translate. This is highly difficult on the face of an actor, but imagine how difficult it is for a vehicle travelling at speed with many other vehicles and other objects around it. So, when you think about it the only natural process for film making was to adopt more sensors to understand the world of light in much deeper detail, much like automotive did a decade ago.

AI for Autonomous Vehicles

Convolutional neural networks (CNN) are used to model spatial information, such as images. CNNs are very good at extracting features from images, and they’re often seen as universal non-linear function approximators.

They are used extensively in Autonomous vehicles as well as in the De-Aging process used in film production.

AI for De-Aging

The flow for De-Aging is: 1) Image-to-image translation using a fully convolutional neural network (CNN) architecture with skip links. 2) Train the CNN using a supervisorial method with several face image pairs with the synthetic photorealistic person. 3) Label the corresponding source and target ages. Look closely at the image above, it shows a representation of the application of CNN.

These parallels, in my opinion, are what is fueling the AI boom. Many industries can take advantage of the core AI novelties and "game changers" while contributing to the core acceleration from orthogonal product paths. These then fuel the rapid adoption expansion, growth, and quality of the core.

This is just the tip of the iceberg. I stopped chasing the rabbit after a while. I do have a day job. ;0) There are many other cool things going on in film that remind me deeply of autonomous vehicles "Continuous Landmark Detection with 3D Queries" published June 4th, 2023, for example by Disney Research. That's a rabbit hole to go down and explore. Maybe Disney should be hiring automotive people? hmmm?

Think about the challenge for autonomous vehicles, then imagine the challenge for facial re-aging film (you may need to close your eyes or go for a long peaceful walk to imagine it :-). Autonomous vehicles bring together many components to create a working system that enables a machine to perceive the world in the same way we, as humans, perceive the situations while driving. This is what Disney Research and ILM faced when trying to create a new method of production ready re-aging for the film industry. The result has to be perceived by us as real, but also capable of doing this in a cost-effective, timely manner.

Our time today has come to an end. I purposefully stopped writing so you may go do some research on your own or reach out to me and connect to chat more. I just found this parallel intriguing and explored it a little. Its one of my super powers, keeps me fresh and interested in new tech, and forces me to learn. I hop you enjoyed it. See what I did with that typo. ;0)