Zero-shot style transfer and content/style probing

Overview

We ask whether a linear map on frozen ViT CLS can improve artistic-style retrieval when raw cosine kNN on the same token does not.

Data: Things100 × ArtBench

We build a UCAST-style grid: object concepts from THINGS stylized into ArtBench artistic domains. Content and style labels are independent, so the same image can be queried for content retrieval or style retrieval.

5×5 content×style matrix (Starry Night, The Scream, Great Wave, Kandinsky, Seurat).

Method: dual CLS probes

Freeze a ViT-B/16. At each layer, map CLS h^(ℓ) ∈ ℝ^768 with two independent linear heads to 256-D content and style embeddings (sigmoid contrastive loss). Evaluate with zero-shot cosine kNN (Recall@1 / mAP).

Dual content/style probes on frozen CLS features.

Results

Probes raise style Recall@1 / mAP relative to raw CLS. Higher probe dimensions help. Content retrieval is generally easier than style on this set.

Style-split mAP (solid = probed, dashed = raw CLS).

Content / instance-split mAP across layers and dimensions.

CLS vs patch / Gram

CLS vs patch pooling for style.

Gram-style features vs CLS probing.

Nested (Matrioshka) probe dimensions, style split.

Error structure

For incorrect style-probe neighbors, the fraction that stay in the query’s object category increases with probe dimension and depth.

Content and style within-category incorrect grids.

Style-probe panel: high means errors stay in-category; low means they leave it.

References

1. Zhang et al., A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning, arXiv 2023.
2. Hebart et al., THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images, PLOS ONE 2019.
3. Liao et al., The ArtBench Dataset: Benchmarking Generative Models with Artworks, arXiv 2022.