#SIGGRAPHAsia | #SIGGRAPHAsia2022
SIGN IN TO VIEW THIS PRESENTATION Sign In
Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
SessionMaps and Fields
DescriptionThis paper aims to tackle the challenge of efficiently producing interactive free-viewpoint videos.
Some recent works equip neural radiance fields with image encoders, enabling them to generalize across scenes. When processing dynamic scenes, they can simply treat each video frame as an individual scene and perform novel view synthesis to generate free-viewpoint videos. However, their rendering process is slow and cannot support interactive applications.
A major factor is that they sample lots of points in empty space when inferring radiance fields.
We propose a novel scene representation, called ENeRF, for the fast creation of interactive free-viewpoint videos. Specifically, given multi-view images at one frame, we first build the cascade cost volume to predict the coarse geometry of the scene. The coarse geometry allows us to sample few points near the scene surface, thereby significantly improving the rendering speed. This process is fully differentiable, enabling us to jointly learn the depth prediction and radiance field networks from RGB images. Experiments show that our approach exhibits competitive performance on the DTU, NeRF Synthetic, Real Forward-facing, ZJU-MoCap, and DynamicCap datasets while being at least 60 times faster than previous generalizable radiance field methods. We demonstrate the capability of our method to synthesize novel views of human performers in real-time. The code is available at https://zju3dv.github.io/enerf/.
Some recent works equip neural radiance fields with image encoders, enabling them to generalize across scenes. When processing dynamic scenes, they can simply treat each video frame as an individual scene and perform novel view synthesis to generate free-viewpoint videos. However, their rendering process is slow and cannot support interactive applications.
A major factor is that they sample lots of points in empty space when inferring radiance fields.
We propose a novel scene representation, called ENeRF, for the fast creation of interactive free-viewpoint videos. Specifically, given multi-view images at one frame, we first build the cascade cost volume to predict the coarse geometry of the scene. The coarse geometry allows us to sample few points near the scene surface, thereby significantly improving the rendering speed. This process is fully differentiable, enabling us to jointly learn the depth prediction and radiance field networks from RGB images. Experiments show that our approach exhibits competitive performance on the DTU, NeRF Synthetic, Real Forward-facing, ZJU-MoCap, and DynamicCap datasets while being at least 60 times faster than previous generalizable radiance field methods. We demonstrate the capability of our method to synthesize novel views of human performers in real-time. The code is available at https://zju3dv.github.io/enerf/.
Event Type
Technical Communications
Technical Papers
TimeFriday, 9 December 20229:00am - 10:30am KST
LocationRoom 324, Level 3, West Wing
