StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields
CVPR 2023

Abstract

3D style transfer aims to render stylized novel views of a 3D scene with multi-view consistency. However, most existing work suffers from a three-way dilemma over accurate geometry reconstruction, high-quality stylization, and being generalizable to arbitrary new styles. We propose StyleRF (Style Radiance Fields), an innovative 3D style transfer technique that resolves the three-way dilemma by performing style transformation within the feature space of a radiance field. StyleRF employs an explicit grid of high-level features to represent 3D scenes, with which high-fidelity geometry can be reliably restored via volume rendering. In addition, it transforms the grid features according to the reference style which directly leads to high-quality zero-shot style transfer. StyleRF consists of two innovative designs. The first is sampling-invariant content transformation that makes the transformation invariant to the holistic statistics of the sampled 3D points and accordingly ensures multi-view consistency. The second is deferred style transformation of 2D feature maps which is equivalent to the transformation of 3D points but greatly reduces memory footprint without degrading multi-view consistency. Extensive experiments show that StyleRF achieves superior 3D stylization quality with precise geometry reconstruction and it can generalize to various new styles in a zero-shot manner.

TL;DR:

StyleRF is a new 3D style transfer technique that resolves the dilemma of accurate geometry reconstruction, high-quality stylization, and being generalizable to arbitrary new styles. It performs style transformation within the feature space of a radiance field, and employs an explicit grid of high-level features to represent 3D scenes.

StyleRF

StyleRF achieves zero-shot 3D style transfer by first reconstructing a 3D grid that contains high-level VGG features. During the rendering process, a batch of sampled points is taken along a ray, and their corresponding features are extracted. Each of these features is then transformed independently using Sampling-Invariant Content Transformation, without considering the holistic statistics of the point batch. The features are then converted to a feature map using Volume Rendering. Next, the Deferred Style Transformation algorithm adaptively transforms the feature map by incorporating the sum weight of the sampled points along the ray and the style information, while strictly maintaining multi-view consistency. Finally, a stylized novel view is generated using a CNN decoder.

Citation

Consider citing us if you find this project is helpful.
@article{liu2023stylerf,
  author = {Kunhao Liu and Fangneng Zhan and Yiwen Chen and Jiahui Zhang and Yingchen Yu and Abdulmotaleb El Saddik and Shijian Lu and Eric Xing},
  title = {StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields},
  booktitle = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023},
}

Acknowledgements

This webpage integrates components from many websites, including RefNeRF, RegNeRF, DreamFusion, and Richard Zhang's template. We sincerely thank the authors for their great work and websites.