Zero-shot style transfer and content/style probing

Overview

We ask whether a linear map on frozen ViT CLS can improve artistic-style retrieval when raw cosine kNN on the same token does not.

Data: Things100 × ArtBench

We build a UCAST-style grid: object concepts from THINGS stylized into ArtBench artistic domains. Content and style labels are independent, so the same image can be queried for content retrieval or style retrieval.

5×5 content×style matrix (Starry Night, The Scream, Great Wave, Kandinsky, Seurat).

UCAST pipeline

The benchmark is a Cartesian product: 1,000 content photographs × 1,000 style paintings, stylized one pair at a time by style_transfer.py into ucast_things100_artbench1000/. That is 1,000,000 JPEGs.

Content and style pools

• Content: THINGS, 100 categories × 10 images = 1,000. setup_things_content.py takes the first 10 files per category in sorted zip order and writes {category}_{1..10}.jpeg. The category list is frozen in manifests/content_categories.txt.
• Style: ArtBench-10 train parquet, 10 classes × 100 images = 1,000, pulled by download_artbench_1000.py and numbered 1 to 1000. Per-image provenance is in manifests/artbench_1000_sources.csv.

The 5×5 grid above is a display subset built from canonical paintings (Starry Night, The Scream, and so on) so the figure is readable. The probes never see it. Training and evaluation use the 1,000 ArtBench styles.

Inference

Framework is UCAST[1], but the forward pass is written out inside style_transfer.py rather than called through CAST_pytorch/test.py, since a million pairs makes per-image process launch the dominant cost. VGG-19 truncated to the first 31 layers (through relu4-1) encodes content and style. AdaIN then matches the content feature channel mean and standard deviation to the style feature mean and standard deviation at that layer, and the decoder (latest_net_Dec_B.pth) upsamples back to RGB. Weights are models/vgg_normalised.pth, checkpoints/UCAST_model/latest_net_AE.pth, and latest_net_Dec_B.pth.

• Both inputs at 224×224, normalization range_5 (mean 0.5, std 0.5, so tensors in [-1, 1]), float16.
• Written as JPEG quality 90, named {contentRel}_stylized_by_{styleRel}.jpeg with nested folder separators flattened to _.
• Existing outputs are skipped, so a run is resumable after a preemption. Each file is written to .tmp and then os.replaced, so a killed job never leaves a half-written JPEG that the skip logic would later accept.

Splits

dataset.py, seed 42. The instance split holds out 20% of the content image names per category, 2 of the 10. The style split holds out 20% of the style categories. This is what turns a million stylizations into a retrieval set: queries and gallery come from held-out content names or held-out styles, not from pairs the probe was fit on.

Method: dual CLS probes

Freeze a ViT-B/16. At each layer, map CLS h^(ℓ) ∈ ℝ^768 with two independent linear heads to L2-normalized 256-D content and style embeddings. Both heads train under a pairwise sigmoid focal loss: build the 512 × 512 cosine matrix for a batch, treat each off-diagonal entry as an independent same-class/different-class decision, and discount the pairs the probe already scores correctly.

Batches are square by construction, C = 32 classes × M = 16 images, so the positive count per batch is fixed rather than left to chance. Focal γ = 1.0 with α = 0.75 on the positive class, which is the minority here at roughly 33:1. Temperature and logit bias are learnable and used only during training; evaluation is zero-shot cosine kNN (Recall@1 / mAP). The α choice, the per-group count normalization, and why not InfoNCE are written up on the poster page: pairwise sigmoid focal loss.

Dual content/style probes on frozen CLS features.

Results

Probes raise style Recall@1 / mAP relative to raw CLS. Higher probe dimensions help. Content retrieval is generally easier than style on this set.

Style-split mAP (solid = probed, dashed = raw CLS).

Content / instance-split mAP across layers and dimensions.

CLS vs patch / Gram

CLS vs patch pooling for style.

Gram-style features vs CLS probing.

Nested (Matrioshka) probe dimensions, style split.

Error structure

For incorrect style-probe neighbors, the fraction that stay in the query’s object category increases with probe dimension and depth.

Content and style within-category incorrect grids.

Style-probe panel: high means errors stay in-category; low means they leave it.

References

1. Zhang et al., A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning, arXiv 2023.
2. Hebart et al., THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images, PLOS ONE 2019.
3. Liao et al., The ArtBench Dataset: Benchmarking Generative Models with Artworks, arXiv 2022.