University of Toronto

Visual abstraction and neural representations

Diffusion Abstraction Spectrum (DAS) on Toronto Scenes: the same photographs morphed from line drawings (λ=0) to photos (λ=1), scored with frozen ViT-B retrieval.

Aryan Pournajafian · aryan.pournajafian@mail.utoronto.ca · Homepage

Overview

We take photographs from the Toronto Scenes dataset and morph them toward sketches. Retrieval at the sketch pole is not the same across ViT-B checkpoints: DINOv3 stays high on category R@1; MAE does not.

At λ=0: DINOv3 category R@1 is 97.9%; MAE is 25.5%. Same ViT-B/16 architecture, five pretrained checkpoints.

Diffusion Abstraction Spectrum

Source images are from Toronto Scenes. We morph each scene with SDXL + ControlNet along λ ∈ [0, 1] (line drawing → photograph). The probe set used here is 475 scenes × 10 levels (4,750 images). Models are scored with zero-shot category and instance retrieval along the spectrum.

One office scene, λ = 0.00 to 1.00.

Several scenes at multiple λ values.

Retrieval across λ

Model	Pretraining	Category R@1 (λ=0)	Instance R@1 (λ=0)
DINOv3	Dense SSL	97.9%	59.8%
SigLIP	Vision–language	93.5%	—
BEiT3	Masked multimodal	80.2%	—
Supervised	IN-21k	60.8%	—
MAE	Generative SSL	25.5%	0.4%

Instance retrieval across λ.

Category vs instance protocols.

Instance vs category accuracy.

DINOv3 before and after spectrum fine-tuning.

Geometry note

MAE can keep relatively high attention-key self-similarity across λ while category/instance retrieval at λ=0 is still low. Layout stability on this metric does not by itself explain the retrieval numbers.

Key self-similarity across λ.