Probing vision transformers across abstraction and style

Overview

Two measurements on frozen ViT-B models: (1) how retrieval changes when Toronto Scenes photographs are driven toward line drawings, and (2) whether a linear probe on CLS can recover artistic style when raw cosine kNN cannot.

Key findings

1. Content and style can be read off the same CLS token with two linear probes. Freeze a ViT-B/16, take CLS at a given layer, and train two independent linear maps: one for object identity, one for artistic style. On Things × ArtBench, raw cosine kNN on late CLS barely retrieves style (Recall@1 under 1%). The 256-D style probe reaches about 88%. Content is already easy from the same token.
2. Sketch-level category retrieval depends on how the ViT was pretrained. Same architecture (ViT-B/16), Toronto Scenes at λ=0: DINOv3 category R@1 is 97.9%, MAE is 25.5%.
3. Stronger style probes make more in-category mistakes. As probe dimension and layer depth grow, incorrect style neighbors more often share the query’s object class. The probe is picking style while still leaking content.
4. Short prefixes of the style probe work if you L2-normalize them. Nested slices of the 256-D style vector stay close to the full probe after re-normalization. Without it, truncated prefixes drop.

1. Diffusion Abstraction Spectrum (DAS)

Source photographs come from the Toronto Scenes dataset. We morph each scene with SDXL + ControlNet along an abstraction axis λ ∈ [0,1] (line drawing → photograph). Models are evaluated with zero-shot instance / category retrieval along the spectrum. Expanded figures and curves live on the visual abstraction report.

Toronto Scenes examples across λ. At λ=0, DINOv3 category retrieval is high; MAE is not.

• DINOv3 ~97.9% category R@1 at λ=0.
• MAE ~25.5% category R@1 at λ=0.
• All models here are ViT-B/16; the gap is between these pretrained checkpoints, not a general architecture result.

2. Style benchmark (Things100 × ArtBench)

UCAST-style pairings[1] give independent content and style labels using THINGS objects[2] and ArtBench styles[3]: the same object under many artistic styles, and the same style across many objects. This enables dual zero-shot retrieval (content vs style). See the zero-shot style report for the full probing suite.

5×5 content×style matrix (Starry Night, The Scream, Great Wave, Kandinsky, Seurat).

3. Method: CLS token linear probing

Freeze a ViT-B/16 backbone. At every layer ℓ ∈ {1…12}, take the CLS token h^(ℓ) ∈ R^768 and train two independent linear maps to 256-D embeddings for content and style with a sigmoid contrastive loss. Evaluation is zero-shot kNN with cosine similarity (Recall@1 / mAP).

Dual content/style probes on frozen CLS features.

4. Results: probes recover style

Across Supervised, CLIP, BEiT3, DINOv3, and MAE, raw late-layer CLS style retrieval is under 1%. Trained probes raise style Recall@1 / mAP; higher probe dimensions help, and the unprobed CLS baseline stays low in late layers.

Style-split mAP across layers and probe dimensions (solid = probed; dashed = raw CLS).

Style-probe within-category incorrect %. As probe dimension and depth grow, more style mistakes land inside the same content class.

The poster keeps this single style heatmap for simplicity. Full content+style grids, per-split breakdowns, Recall@1 curves, and the Matrioshka re-normalization study are in Bonus analyses.

5. Full incorrect-retrieval gallery

The poster shows four representative pairs. Below are curated incorrect neighbors from the DINOv3 dim-256 content probe. Several pairs share artistic style and differ in object class (tagged below).

#01 broccoli → lobster wrong class · same style

#02 pumpkin → lemon wrong class · same style

#03 apple → lemon wrong class · same style

#04 plate → plate wrong instance · same style

#05 cat → wolf wrong class · same style

#06 ant → hedgehog wrong class · same style

#07 broccoli → broccoli wrong instance · same style

#08 elephant → deer wrong class · same style

#09 lemon → lemon wrong instance · same style

#10 orange → orange wrong instance · same style

#11 potato → potato wrong instance · cross style

#12 leopard → ant wrong class · cross style

#13 zebra → cow wrong class · cross style

#14 potato → potato wrong instance · same style

#15 duck → key wrong class · cross style

#16 duck → turtle wrong class · same style

#17 lemon → lemon wrong instance · same style

#18 penguin → leopard wrong class · cross style

#19 pumpkin → pumpkin wrong instance · same style

#20 eagle → eagle wrong instance · same style

#21 squirrel → shark wrong class · cross style

#22 plate → plate wrong instance · same style

#23 grape → lobster wrong class · cross style

#24 lemon → lemon wrong instance · same style

#25 koala → grape wrong class · cross style

#26 rabbit → grape wrong class · cross style

#27 plate → plate wrong instance · same style

#28 pig → koala wrong class · same style

#29 helicopter → helicopter wrong instance · same style

#30 lemon → lemon wrong instance · same style

Extra plots

Plots that did not fit on the poster: retrieval grids, within-category heatmaps, total-error heatmaps, and nested-dimension (Matrioshka) re-normalization.

B1. Full model comparison grids

Style and instance splits for both mAP and Recall@1. Solid curves = probed dimensions; dashed = raw CLS kNN baseline.

Style split · mAP.

Instance split · mAP.

Style split · Recall@1.

Instance split · Recall@1.

B2. Within-category incorrect heatmaps

Fraction of incorrect retrievals that still land in the same content category. Useful for separating style-looking mistakes from random neighbors.

Left: content probe · Right: style probe (the poster uses the right panel).

2×2 by probe × split (instance vs style evaluation).

B3. Total incorrect-retrieval heatmaps

Average number of incorrect retrievals (content vs style probes).

Broken down by evaluation split.

B4. Matrioshka / nested-dimension study

Nested slices of the 256-D probe test whether early dimensions already carry the task, and whether L2 re-normalization of a truncated prefix matches the full vector. For style, small K with re-norm stays close to the full 256-D probe. Without re-norm, truncated prefixes degrade.

Matrioshka study · style split.

Matrioshka study · instance split.

6. References

1. Zhang et al., A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning, arXiv 2023.
2. Hebart et al., THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images, PLOS ONE 2019.
3. Liao et al., The ArtBench Dataset: Benchmarking Generative Models with Artworks, arXiv 2022.