What is Unreal PaGAN

Unreal PaGAN is Pinscreen’s state-of-the-art AI-based performance-driven facial animation software developed in UE4. Our solution can generate photorealistic renderings of a virtual 3D avatar in real-time using Pinscreen’s proprietary neural rendering engine called PaGAN (photoreal avatar GAN) and produce results that are nearly indistinguishable for real ones.

Technology Stack

User
Input

High-Performance Facial Capture

Facial Tracking

Facial Retargeting

Unreal Engine

Persona Rig

Physically-Based Rendering

PaGAN Neural Face Rendering

Deep Model

GAN Synthesis

Photoreal
Avatar

Project: Unreal PaGAN

Background

For the past decade, digital humans that are indistinguishable from real ones have been featured prominently as high-end visual effects in blockbuster movies, but the production costs are in the millions and months of work and computation are needed for a couple of seconds of footage. Real-time graphics technologies have also evolved significantly over the past few years and compelling performance-driven virtual humans are impacting a wide range of areas such as gaming, virtual production, and AR/VR applications. Some notable recent highlights include the Digital Mike and the Siren demo from Epic Games, 3Lateral, and Cubic motions, which demonstrates that extremely convincing virtual humans can be rendered in real-time in the UE4 game engine. While the results are impressive and close to crossing the so-called “Uncanny Valley”, computer generated faces are still not 100% on par with real ones, and it is still possible to tell they are synthetic.

About Pinscreen

Pinscreen is an artificial intelligence startup founded in 2015 by former USC professor and MIT 35 Under 35 winner, Dr. Hao Li (www.hao-li.com), that builds photorealistic AI-driven virtual humans as the next generation of human-machine interface. Applications include virtual assistants, companions, and influencers. Pinscreen is known for developing cutting-edge deep learning technologies that can synthesize photorealistic 3D avatars in real-time. Compared to conventional graphics pipelines where assets are tweaked by digital artists, Pinscreen introduced a machine learning paradigm where deep models are trained using massive amounts of images and videos.

Pinscreen Neural Rendering Technology

We have recently made two important technological breakthroughs in neural rendering: (1) PaGAN HD: a photorealistic enhancement of computer generated faces using a generative adversarial network (GAN)-based synthesis approach using our PaGAN (photoreal avatar GAN) engine; and (2) PaGAN RT: real-time capabilities using a highly optimized deep neural network architecture running on a state-of-the-art GPU. Our PaGAN system shares some similarities with deepfake technologies, but generates far superior fidelity and can run in real-time. We have published our algorithm at SIGGRAPH and demonstrated different variants of our capabilities at multiple SIGGRAPH Real-Time Live! events (2020, 2019, 2018), as well as the World Economic Forum 2020 in Davos.

PaGAN Unreal Integration and Performance-Driven Animation

Pinscreen is introducing Unreal PaGAN, a neural rendering-based facial animation software allowing users to produce photoreal faces using our proprietary AI synthesis technology. The goal of this project is to integrate our research prototype for neural face rendering (currently in Linux) into the Unreal Engine in the form of a standalone software and develop tools that allow content creators to produce photorealistic digital humans and performances at unprecedented fidelity and in real-time. We will demonstrate an end-to-end real-time solution for performance-driven facial animation using a webcam and Pinscreen’s advanced facial synthesis technology. We will incorporate our latest high-performance video-based facial tracking solution with Epic Games’ latest Persona Rig and collaborate with Sydney’s Motus Lab, headed by Dr. Mike Seymour (www.motuslab.org) for asset creation and to assess the performance of our technology using a comprehensive perceptual and user study. This project will demonstrate the first integrated solution for AI-driven facial synthesis using a high-end game engine and validate the use of machine learning to cross the “Uncanny Valley” for virtual humans in real-time applications.

Impact and Applications

Pinscreen’s objective is to develop an end-to-end software solution for photoreal AI-driven virtual assistants and companions, as well as build a highly-efficient production pipeline for the creation of true-to-real virtual influencers. We believe that AI-synthesized faces using our PaGAN technology will facilitate the democratization of producing true-to-real digital humans in virtual production environments. Our approach has the potential to impact how virtual humans will be produced and rendered for a wide range of real-time applications, including next gen AR/VR-based telepresence systems, video games, virtual fitting solutions for e-commerce, immersive simulation and training platforms, as well as various avatar-based health applications.

Links