S2COPE: Self-Supervised Concept Discovery
via Preference Learning

Can unlabeled data teach foundation models interpretable concepts?

Shilong Xiang, Zirui Zhang, Chengzhi Mao
Rutgers University
arXiv Code
Self-Supervised Visual Concept Discovery
Figure 1 Self-Supervised Visual Concept Discovery. (Left) Standard self-supervised learning represents images as feature vectors, which are not directly understandable. (Right) S²COPE learns to propose human-readable visual concepts from unlabeled images. A pretrained vision-language model samples candidate concepts, and a contrastive objective evaluates how well they describe the source image and distinguish it from other images. These scores define preferences for training the concept generator, without class labels or concept annotations. The learned concepts support downstream prediction and provide an explicit vocabulary for inspecting model behavior.

Abstract

Self-supervised learning can learn useful representations from unlabeled images. Yet these representations remain largely opaque, making it difficult to understand what visual concepts they encode. Can self-supervision instead produce useful visual concepts that are directly understandable to people?

We introduce S²COPE, a framework that represents images as sets of concepts expressed in natural language. A vision-language model proposes candidate concepts, and a self-supervised objective evaluates how well each concept describes the source image while distinguishing it from other images. This feedback is used to construct preferences and train the model to propose better concepts.

Across eight diverse visual datasets, our self-supervised concept representations transfer without adaptation and improve downstream recognition accuracy over prior interpretable approaches. In medical imaging, our discovered concepts reveal spurious visual cues and enable a more robust classifier. These results show that self-supervision can discover useful, human-readable visual representations from unlabeled images.

iNaturalist CUB Galaxy10 Gravity Spy HAM10000 OrganCMNIST OrganMNIST3D FairFace ISIC
8/8
Datasets Beat All Baselines
+7.37
Avg Gain over Best Baseline
96.4%
Human Preference
−15.0
Shortcut Error on ISIC

Method

Overview of the S²COPE Discovery Loop
Figure 2 Overview of the S²COPE Discovery Loop. At iteration k, the generator πk samples several descriptions for each unlabeled image. A frozen CLIP model scores each description by how well it matches the source image relative to other images. We pair higher- and lower-scoring descriptions with the largest score gaps and train the generator on these preferences using DPO. The updated generator πk+1 then proposes candidates for the next iteration.

Results

Cross-Domain Recognition

MethodLabelsNaturePhysicsMedicalFaces
iNaturalistCUBGalaxy10Gravity SpyHAM10000OrganCMNISTOrganMNIST3DFairFace
TextSpan✓20.0081.0824.074.2517.107.4012.4250.95
DCLIP✓23.4682.1411.752.1733.1610.4114.2946.16
LF-CBM✓73.8564.5553.4483.0870.9878.8565.2253.47
LaBo✓42.6977.6353.5539.5063.7354.1022.3639.99
VLLM (Unoptimized)✗71.1574.7754.0960.8365.2868.0658.3950.63
VLLM + S²COPE (Ours)✗85.00
(+11.15)		85.11
(+2.97)		66.52
(+12.97)		83.55
(+0.47)		79.27
(+8.29)		91.43
(+12.58)		70.19
(+4.97)		59.01
(+5.54)

Table 1 Top-1 Concept-Based Classification Accuracy (%) Across Eight Datasets. S²COPE fine-tunes a VLLM on unlabeled iNaturalist-mini images, which is then transferred to other datasets without adaptation. “Labels” indicates class-label or class-name use in concept construction. Blue values denote percentage-point gains over the best of the four concept-based baselines per dataset. S²COPE outperforms these baselines on all eight datasets, with an average gain of 7.37 percentage points.

Concepts Before and After Learning

Concepts Before and After Learning

Figure 3 Concepts Before and After Learning. For each image, we compare concepts from the pretrained generator (top list) and S²COPE (bottom list). Red marks concepts that are incorrect or uninformative for the category.

Pairwise Preference Evaluation

EvaluatorGPT-5.6-SolHuman
OpponentUnoptimized VLLMTextSpanDCLIPLF-CBMLaBoUnoptimized VLLM
S²COPE preference85.998.796.289.798.796.41

Table 2 Pairwise Preference Win Rate for S²COPE Concepts (%). For each opponent, we report the percentage of the 39 images for which the evaluator prefers S²COPE's anonymized concept list over the opponent's. GPT-5.6-Sol compares S²COPE against five baselines, while human evaluators compare S²COPE against the unoptimized VLLM, with results averaged across evaluators.

Discovering and Mitigating Spurious Correlations

On ISIC dermoscopy, colored calibration patches are spuriously associated with benign lesions. Without being supplied artifact names or patch annotations, the learned generator produces concepts such as “orange circular marker,” “green circular patch,” and “round sticker-like texture,” exposing the confounder. Filtering the affected training images with these concepts and retraining the linear probe reduces the patch-induced error rate on malignant images from 74.9% to 59.9%.

Concept-Guided Shortcut Mitigation

Figure 4 Concept-Guided Shortcut Mitigation. Adding a patch causes this malignant image to be misclassified as benign. “+ S²COPE” denotes concept-guided data curation and probe retraining, which preserve the correct prediction. S²COPE enables targeted data curation that removes bias in the data.

SettingPatch detection (%)Error rate (%) ↓
PrecisionRecall
VLLM (Unoptimized)——74.9
VLLM + S²COPE95.583.459.9

Table 3 Concept-Guided Shortcut Mitigation on ISIC. S²COPE identifies affected training images; removing them and retraining the linear probe reduces the patch-induced error rate from 74.9% to 59.9%.

Ablations

Ablation Studies on Reward Formulation
Figure 5 Ablation Studies on Reward Formulation. (a) Reward Components: Impact of isolating the positive and negative signals of the contrastive reward. Eliminating the positive signal causes a performance collapse, while removing the negative signal yields a suboptimal accuracy plateau. (b) Reward Modality: Comparison of cross-modal image-text grounding versus unimodal text-text consensus. Cross-modal grounding against physical image features achieves better performance than relying on unimodal textual consensus.