RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting