Volumetric capture with multiple azure Kinects in Unreal Engine

Abstract

Volumetric capture (Volumetric video) is a technique that allows to create “holographic” recordings of actors, sets and props. The technique can be used to create immersive stories that sometimes reflect aspects of reality better than realistic 3D models. For example, volumetric captures of actors do not seem to cause uncanny valley effect. Volumetric video is a core underlying technology for emerging Mixed Reality systems. What was previously available for a glimpse only in science fiction movies and futuristic predictions, now with the ongoing research on volumetric video capturing, coding and presentation, realistic mixed reality experiences are close to become a reality. At the same time, computer generated holography and other digital 3D projection techniques start to became more common and affordable. The emergence of solutions for capturing volumetric video and devices which can display volumetric video mixed with the real world are paving the way to a new media, where a real object and its volumetric virtual image are indistinguishable. Virtual simulation of human faces and facial movements has challenged media artists and computer scientists since the first realistic 3D renderings of a human face by Fred Parke in 1972. Today, a range of software and techniques are available for modelling virtual characters and their facial behavior in immersive environments, such as computer games or storyworlds. However, applying these techniques often requires large teams with multidisciplinary expertise, extensive amount of manual labour, as well as financial conditions that are not typically available for individual media artists. In this thesis first created a metahuman from photographs with the help of KeenTools FaceBuilder and the MetaHuman plugin. Also thesis aims to use three Azure Kinects DK devices(depth sensors) to capture in real-time the ''hologram'' of the user. All of the above is built on Epic Games' Unreal Engine game platform in C++ programming language in visual studio 2022.