StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields
CVPR 2023

Kunhao Liu
Nanyang Technological University, Singapore
Fangneng Zhan
Max Planck Institute for Informatics, Germany
Yiwen Chen
Nanyang Technological University, Singapore
Jiahui Zhang
Nanyang Technological University, Singapore
Yingchen Yu
Nanyang Technological University, Singapore
Abdulmotaleb El Saddik
University of Ottawa, Canada
MBZUAI, United Arab Emirates
Shijian Lu
Nanyang Technological University, Singapore
Eric Xing
Carnegie Mellon University, USA
MBZUAI, United Arab Emirates
overview

Abstract

3D style transfer aims to render stylized novel views of a 3D scene with multi-view consistency. However, most existing work suffers from a three-way dilemma over accurate geometry reconstruction, high-quality stylization, and being generalizable to arbitrary new styles. We propose StyleRF (Style Radiance Fields), an innovative 3D style transfer technique that resolves the three-way dilemma by performing style transformation within the feature space of a radiance field. StyleRF employs an explicit grid of high-level features to represent 3D scenes, with which high-fidelity geometry can be reliably restored via volume rendering. In addition, it transforms the grid features according to the reference style which directly leads to high-quality zero-shot style transfer. StyleRF consists of two innovative designs. The first is sampling-invariant content transformation that makes the transformation invariant to the holistic statistics of the sampled 3D points and accordingly ensures multi-view consistency. The second is deferred style transformation of 2D feature maps which is equivalent to the transformation of 3D points but greatly reduces memory footprint without degrading multi-view consistency. Extensive experiments show that StyleRF achieves superior 3D stylization quality with precise geometry reconstruction and it can generalize to various new styles in a zero-shot manner.

TL;DR:

StyleRF is a new 3D style transfer technique that resolves the dilemma of accurate geometry reconstruction, high-quality stylization, and being generalizable to arbitrary new styles. It performs style transformation within the feature space of a radiance field, and employs an explicit grid of high-level features to represent 3D scenes.


StyleRF

overview

StyleRF achieves zero-shot 3D style transfer by first reconstructing a 3D grid that contains high-level VGG features. During the rendering process, a batch of sampled points is taken along a ray, and their corresponding features are extracted. Each of these features is then transformed independently using Sampling-Invariant Content Transformation, without considering the holistic statistics of the point batch. The features are then converted to a feature map using Volume Rendering. Next, the Deferred Style Transformation algorithm adaptively transforms the feature map by incorporating the sum weight of the sampled points along the ray and the style information, while strictly maintaining multi-view consistency. Finally, a stylized novel view is generated using a CNN decoder.


3D Stylization Comparisons

StyleRF achieves superior 3D stylization quality with precise geometry reconstruction and it can generalize to various new styles in a zero-shot manner.

Open fullscreen for details

Compositional Style Transfer

Thanks to its precise geometry reconstruction, StyleRF can be seamlessly integrated with NeRF-based object segmentation for compositional 3D style transfer. Due to its zero-shot nature, StyleRF can create infinite combinations of styles without additional training, producing numerous artistic creations and inspirations.

Open fullscreen for details

Style Interpolation

StyleRF can smoothly interpolate different styles thanks to its high-level feature representation of a 3D scene. StyleRF can not only transfer arbitrary styles in a zero-shot manner but also generate non-existent stylization via multi-style interpolation.

Open fullscreen for details

Citation

Consider citing us if you find this project is helpful.
@article{liu2023stylerf,
  author = {Kunhao Liu and Fangneng Zhan and Yiwen Chen and Jiahui Zhang and Yingchen Yu and Abdulmotaleb El Saddik and Shijian Lu and Eric Xing},
  title = {StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields},
  booktitle = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023},
}

Acknowledgements

This webpage integrates components from many websites, including RefNeRF, RegNeRF, DreamFusion, and Richard Zhang's template. We sincerely thank the authors for their great work and websites.