Flow Supervision for Deformable NeRF

CVPR 2023 Highlight

1Carnegie Mellon University, 2University of Adelaide

FSDNeRF enhance deformable NeRF by applying flow supervision to the derformation field.

Abstract

Deformable neural radiance fields (e.g. Nerfies , HyperNeRF, D-NeRF, NR-NeRF ) has been a notable technique to represent dynamic scenes and shows plausible space-time view synthesis results. However, the current implementations only warrant success on teleporting-like videos whose camera motions are significantly more rapid than object motions. Quality of their results significantly decrease on videos with more rapid object motions [4]. For example, in the "broom" example above, both Nefies and HyperNeRF show significant artifacts when trained only using a single camera (i.e. the left camera from the stereo camera rig), as oppose to teleporting between the left and right cameras in the original paper of Nerfies.

In this work, we conjecture the deficiency of these deformable NeRF-based methods is mainly due to lack of temporal regularization. The community has explored optical flow as an additional cue to help supervise the temporal transitions of other motion representations, such as scene flow fields and blend skinning fields. However, enforcing flow constraints with respect to a generic backward deformation field as in Nerfies is non-trivial. We propose an elegant solution to enforce flow constraints for deformable NeRFs, without inverting the backward deformation function. Our method produces plausible view synthesis result, as well as accurate depth map and motions.

FSDNeRF synthesize flow directly using the deformation field.

Flow supervision is critical to reconstruct plausible motion and structure.

Video

Related Projects

For reconstructing articulated objects such as animals, please see our RAC project, which is an extension of previous work BANMo.

Both RAC and this work are built using an efficient SE(3) libarary dqtorch which provides faster CUDA extensions for batched (dual) queternion-based operations.

There have been works (e.g. CaDeX, NDR) using bijective neural networks to represent bijective deformation field. The derivation of this work is more general and can be applied to any deformation field representation.

BibTeX

@InProceedings{Wang_2023_CVPR,
      author    = {Wang, Chaoyang and MacDonald, Lachlan Ewen and Jeni, L\'aszl\'o A. and Lucey, Simon},
      title     = {Flow Supervision for Deformable NeRF},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2023},
      pages     = {21128-21137}
}