VEIL-PGD: imperceptible, JPEG-robust adversarial perturbation over an ensemble of VLM vision towers for training-data protection
Python
5
0 commits
updated Jul 10, 2026
license: other license_name: polyform-noncommercial-1.0.0 license_link: https://polyformproject.org/licenses/noncommercial/1.0.0 tags:
A research preview (v0.2.1) of an image-side protection method against unauthorized vision and vision-language training-data ingestion.
Public images are routinely collected into web-scale datasets and used to train vision and vision-language models, and there is rarely a practical way to opt out. We present VEIL-PGD (Vision-Encoder Imperceptible Layer), a creator-controlled defense that adds a low-visibility adversarial perturbation to an image before publication. The goal is to make the supervision an ingestion pipeline extracts from that image, such as captions, tags, retrieval embeddings, and synthetic labels, semantically displaced from the true content. The perturbation is an Expectation-over-Transformation (EOT) hardened, ensemble, momentum-PGD attack optimized against a diverse set of public vision encoders. That set spans contrastive CLIP-family models, self-supervised and auto-supervised feature towers, and the vision towers of recent open vision-language models. On a 60-image benchmark with bootstrap confidence intervals, the perturbation causes frontier captioners (GPT-5.5 and Gemini 3.5 Flash) to prefer a decoy label over the true label on 45.2% of the images they originally labeled correctly, measured after JPEG recompression. Transfer to held-out encoders that never took part in the attack, including a real VLM captioner tower and an agglomerative foundation model, reaches 28.5%. This is arms-race mitigation rather than a permanent or universal guarantee, and the results are early. This repository releases the method and code, and it ships no trained weights.
For many public images there is no dependable way to opt out of dataset collection. Platform controls are uneven, scraper behavior is opaque, and once an image has spread through mirrors, indexes, reposts, and data brokers, policy-level signals are hard to enforce. VEIL-PGD treats protection as something the image itself can carry. The creator applies a perturbation before publication that degrades the image's usefulness as training data while preserving normal human viewing. It is a harm-reduction layer, not a replacement for consent or a legal opt-out.
Version 0.2.1 releases the ensemble PGD attack as a method-and-code repository. The "model" is an optimization procedure over off-the-shelf public vision encoders, and no weights are trained or distributed.
The perturbation is optimized against an attack set of 13 public vision encoders, and it is evaluated on a separate held-out set of encoders that never take part in the optimization. Keeping the held-out encoders out of the attack lets their flip rate measure genuine cross-architecture transfer rather than self-similarity.
Attack set (13 encoders):
Held-out judges (never attacked):
Let $f$ be the L2-normalized embedding of the perturbed image and $f_\text{clean}$ the embedding of the clean image. For a contrastive encoder, let $e_\text{truth}$ and $e_\text{decoy}$ be the text embeddings of the true and decoy labels. The per-encoder loss combines a targeted term that pulls the image toward the decoy and away from the truth, and an untargeted repel term that pushes the image off its clean representation, weighted by $w_t$ and $w_r$:
$$ \mathcal{L}\text{contrastive} = w_t \big[\cos(f, e\text{decoy}) - \cos(f, e_\text{truth})\big]
Feature-only towers have no text tower, so a text direction is not available. We instead give each of them a decoy target computed from images. For a class $c$ and encoder $g$, the centroid is the mean L2-normalized embedding of a set of exemplar images $X_c$ of that class,
$$ \mu_c^{(g)} = \frac{\displaystyle\sum_{x \in X_c} \hat{g}(x)} {\big\lVert \sum_{x \in X_c} \hat{g}(x) \big\rVert_2}, \qquad \hat{g}(x) = \frac{g(x)}{\lVert g(x) \rVert_2}, $$
and the feature-tower loss steers the image toward the decoy centroid and away from the truth centroid:
$$ \mathcal{L}\text{feature} = w_t \big[\cos(f, \mu\text{decoy}) - \cos(f, \mu_\text{truth})\big]. $$
The exemplar images used to build the centroids are disjoint from the evaluation images, which keeps class information from leaking into the benchmark.
Each optimization step averages gradients over Expectation-over-Transformation draws, which include JPEG through a straight-through estimator, Gaussian blur, resize, and crop. Averaging over these transforms is what lets the perturbation survive the recompression and rescaling that destroy a naive PGD attack.
To avoid overfitting to any single encoder family, the attack normalizes each encoder's gradient to unit L2 length before aggregation, samples a family-aware subset at every step so no step is dominated by near-clone architectures, and can optionally use variance-tuned momentum (VMI).
An optional LPIPS penalty, combined with a hard LPIPS projection and a DCT low-pass mask, trades transfer strength for lower visual salience. This yields a more strongly imperceptible regime at a measurable cost in flip rate.
max_per_family = 2, and min_feature = 1.Results are from a 60-image benchmark and should be read as a research signal rather than a guarantee. Frontier flip rate is measured over the images each model labeled correctly on the clean version, so the denominator differs by judge.
| Configuration | GPT-5.5 | Gemini 3.5 Flash | Pooled | Pooled 95% CI |
|---|---|---|---|---|
| Max transfer (ε = 6, plain) | 37.1% | 52.6% | 45.2% | [33.8, 56.6] |
| Max stealth (ε = 6, perceptual) | 25.7% | 44.7% | 35.6% | [24.7, 46.8] |
| Configuration | Held-out transfer | Attack-set flip | SSIM | LPIPS | PSNR | ΔE2000 p95 |
|---|---|---|---|---|---|---|
| Max transfer (ε = 6, plain) | 28.5% [21.1, 36.1] | 67.3% [60.0, 74.1] | 0.939 | 0.134 | 33.8 dB | 8.1 |
| Max stealth (ε = 6, perceptual) | 17.1% [10.7, 24.1] | 54.0% [46.2, 61.5] | 0.960 | 0.023 | 35.7 dB | 6.8 |
Findings. Putting the full ensemble back into the attack and reserving fresh cross-architecture encoders as judges gives the strongest frontier result we have measured, and it holds up on encoders that never took part in the optimization. The perceptual regime trades some of that transfer for higher stealth. We separately evaluated two very large contrastive encoders (SigLIP2-giant and MetaCLIP2-H) as extra attackers, but a matched ablation showed they slightly reduced both frontier and held-out transfer, so they are not part of the released attack. Adding more CLIP-family capacity is not the bottleneck; architectural diversity is.
Clean image (left) versus protected image (right):
A full before/after grid is provided in
assets/grid_before_after_16.png.
VEIL-PGD is intended for defensive use, helping people make their own images less reliable as training data when consent and opt-out mechanisms are absent, opaque, or unenforced. It should not be used to degrade datasets you do not own, to evade content-moderation or safety systems, or to mislead people about the visible content of an image.
The attack runs on a CUDA GPU.
python3.11 -m venv .venv
. .venv/bin/activate
pip install -r requirements-lock.txt
# CSV manifest: "image_path,truth_label" per line
python -m ensemble.run_attack \
--manifest examples/testset60.csv --images examples/testset60 \
--exemplars examples/exemplars --train v0.2.1 \
--out runs/demo --steps 120 --eps 6 --subset 6 \
--grad-norm --max-per-family 2 --min-feature 1 --metrics
Key flags: --eps sets the L-infinity budget in /255, --train selects the encoder
set (v0.2.1 is the released attack plus the cross-architecture judges), --exemplars
points at the per-class images used for feature-tower centroids, --lpips-weight,
--lpips-tau, and --dct-keep control the perceptual budget, and --metrics records
PSNR, SSIM, ΔE, and LPIPS. Adversarial PNGs are written to runs/<out>/adv/, and
results.json records per-encoder margins together with attack-set and held-out flip
rates.
ensemble/ the attack implementation
attack.py EOT, ensemble, and momentum-PGD loop
encoders.py differentiable encoder interface and the train/judge split
loss.py per-encoder gradient aggregation and VMI support
sampling.py family-aware subset sampling
perceptual.py LPIPS, DCT low-pass, and stealth metrics
targets.py feature-tower decoy centroids from exemplar images
eot.py JPEG-STE, blur, resize, and crop transforms
run_attack.py attack runner and margin scoring
towers/ differentiable loaders for VLM and foundation vision towers
src/veil_pgd/ legacy typographic-overlay CLI and service adapters
scripts/ frontier evaluation, aggregation, and test-set builders
services/ GPU inference service for legacy experiments
@software{veil_pgd_2026,
author = {cebeuq},
title = {VEIL-PGD: Vision-Encoder Imperceptible Layer for Training-Data Protection},
year = {2026},
version = {0.2.1},
url = {https://huggingface.co/cebeuq/veil-pgd},
note = {EOT-hardened ensemble-PGD image-side protection against
vision-language training-data ingestion.}
}
PolyForm Noncommercial License 1.0.0, see LICENSE. Free for noncommercial personal,
research, educational, nonprofit, and government use. Commercial use is not granted under
this license, so please contact the authors for a commercial arrangement.
VEIL-PGD: imperceptible, JPEG-robust adversarial perturbation over an ensemble of VLM vision towers for training-data protection
Python
5
0 commits
updated Jul 10, 2026
license: other license_name: polyform-noncommercial-1.0.0 license_link: https://polyformproject.org/licenses/noncommercial/1.0.0 tags:
A research preview (v0.2.1) of an image-side protection method against unauthorized vision and vision-language training-data ingestion.
Public images are routinely collected into web-scale datasets and used to train vision and vision-language models, and there is rarely a practical way to opt out. We present VEIL-PGD (Vision-Encoder Imperceptible Layer), a creator-controlled defense that adds a low-visibility adversarial perturbation to an image before publication. The goal is to make the supervision an ingestion pipeline extracts from that image, such as captions, tags, retrieval embeddings, and synthetic labels, semantically displaced from the true content. The perturbation is an Expectation-over-Transformation (EOT) hardened, ensemble, momentum-PGD attack optimized against a diverse set of public vision encoders. That set spans contrastive CLIP-family models, self-supervised and auto-supervised feature towers, and the vision towers of recent open vision-language models. On a 60-image benchmark with bootstrap confidence intervals, the perturbation causes frontier captioners (GPT-5.5 and Gemini 3.5 Flash) to prefer a decoy label over the true label on 45.2% of the images they originally labeled correctly, measured after JPEG recompression. Transfer to held-out encoders that never took part in the attack, including a real VLM captioner tower and an agglomerative foundation model, reaches 28.5%. This is arms-race mitigation rather than a permanent or universal guarantee, and the results are early. This repository releases the method and code, and it ships no trained weights.
For many public images there is no dependable way to opt out of dataset collection. Platform controls are uneven, scraper behavior is opaque, and once an image has spread through mirrors, indexes, reposts, and data brokers, policy-level signals are hard to enforce. VEIL-PGD treats protection as something the image itself can carry. The creator applies a perturbation before publication that degrades the image's usefulness as training data while preserving normal human viewing. It is a harm-reduction layer, not a replacement for consent or a legal opt-out.
Version 0.2.1 releases the ensemble PGD attack as a method-and-code repository. The "model" is an optimization procedure over off-the-shelf public vision encoders, and no weights are trained or distributed.
The perturbation is optimized against an attack set of 13 public vision encoders, and it is evaluated on a separate held-out set of encoders that never take part in the optimization. Keeping the held-out encoders out of the attack lets their flip rate measure genuine cross-architecture transfer rather than self-similarity.
Attack set (13 encoders):
Held-out judges (never attacked):
Let $f$ be the L2-normalized embedding of the perturbed image and $f_\text{clean}$ the embedding of the clean image. For a contrastive encoder, let $e_\text{truth}$ and $e_\text{decoy}$ be the text embeddings of the true and decoy labels. The per-encoder loss combines a targeted term that pulls the image toward the decoy and away from the truth, and an untargeted repel term that pushes the image off its clean representation, weighted by $w_t$ and $w_r$:
$$ \mathcal{L}\text{contrastive} = w_t \big[\cos(f, e\text{decoy}) - \cos(f, e_\text{truth})\big]
Feature-only towers have no text tower, so a text direction is not available. We instead give each of them a decoy target computed from images. For a class $c$ and encoder $g$, the centroid is the mean L2-normalized embedding of a set of exemplar images $X_c$ of that class,
$$ \mu_c^{(g)} = \frac{\displaystyle\sum_{x \in X_c} \hat{g}(x)} {\big\lVert \sum_{x \in X_c} \hat{g}(x) \big\rVert_2}, \qquad \hat{g}(x) = \frac{g(x)}{\lVert g(x) \rVert_2}, $$
and the feature-tower loss steers the image toward the decoy centroid and away from the truth centroid:
$$ \mathcal{L}\text{feature} = w_t \big[\cos(f, \mu\text{decoy}) - \cos(f, \mu_\text{truth})\big]. $$
The exemplar images used to build the centroids are disjoint from the evaluation images, which keeps class information from leaking into the benchmark.
Each optimization step averages gradients over Expectation-over-Transformation draws, which include JPEG through a straight-through estimator, Gaussian blur, resize, and crop. Averaging over these transforms is what lets the perturbation survive the recompression and rescaling that destroy a naive PGD attack.
To avoid overfitting to any single encoder family, the attack normalizes each encoder's gradient to unit L2 length before aggregation, samples a family-aware subset at every step so no step is dominated by near-clone architectures, and can optionally use variance-tuned momentum (VMI).
An optional LPIPS penalty, combined with a hard LPIPS projection and a DCT low-pass mask, trades transfer strength for lower visual salience. This yields a more strongly imperceptible regime at a measurable cost in flip rate.
max_per_family = 2, and min_feature = 1.Results are from a 60-image benchmark and should be read as a research signal rather than a guarantee. Frontier flip rate is measured over the images each model labeled correctly on the clean version, so the denominator differs by judge.
| Configuration | GPT-5.5 | Gemini 3.5 Flash | Pooled | Pooled 95% CI |
|---|---|---|---|---|
| Max transfer (ε = 6, plain) | 37.1% | 52.6% | 45.2% | [33.8, 56.6] |
| Max stealth (ε = 6, perceptual) | 25.7% | 44.7% | 35.6% | [24.7, 46.8] |
| Configuration | Held-out transfer | Attack-set flip | SSIM | LPIPS | PSNR | ΔE2000 p95 |
|---|---|---|---|---|---|---|
| Max transfer (ε = 6, plain) | 28.5% [21.1, 36.1] | 67.3% [60.0, 74.1] | 0.939 | 0.134 | 33.8 dB | 8.1 |
| Max stealth (ε = 6, perceptual) | 17.1% [10.7, 24.1] | 54.0% [46.2, 61.5] | 0.960 | 0.023 | 35.7 dB | 6.8 |
Findings. Putting the full ensemble back into the attack and reserving fresh cross-architecture encoders as judges gives the strongest frontier result we have measured, and it holds up on encoders that never took part in the optimization. The perceptual regime trades some of that transfer for higher stealth. We separately evaluated two very large contrastive encoders (SigLIP2-giant and MetaCLIP2-H) as extra attackers, but a matched ablation showed they slightly reduced both frontier and held-out transfer, so they are not part of the released attack. Adding more CLIP-family capacity is not the bottleneck; architectural diversity is.
Clean image (left) versus protected image (right):
A full before/after grid is provided in
assets/grid_before_after_16.png.
VEIL-PGD is intended for defensive use, helping people make their own images less reliable as training data when consent and opt-out mechanisms are absent, opaque, or unenforced. It should not be used to degrade datasets you do not own, to evade content-moderation or safety systems, or to mislead people about the visible content of an image.
The attack runs on a CUDA GPU.
python3.11 -m venv .venv
. .venv/bin/activate
pip install -r requirements-lock.txt
# CSV manifest: "image_path,truth_label" per line
python -m ensemble.run_attack \
--manifest examples/testset60.csv --images examples/testset60 \
--exemplars examples/exemplars --train v0.2.1 \
--out runs/demo --steps 120 --eps 6 --subset 6 \
--grad-norm --max-per-family 2 --min-feature 1 --metrics
Key flags: --eps sets the L-infinity budget in /255, --train selects the encoder
set (v0.2.1 is the released attack plus the cross-architecture judges), --exemplars
points at the per-class images used for feature-tower centroids, --lpips-weight,
--lpips-tau, and --dct-keep control the perceptual budget, and --metrics records
PSNR, SSIM, ΔE, and LPIPS. Adversarial PNGs are written to runs/<out>/adv/, and
results.json records per-encoder margins together with attack-set and held-out flip
rates.
ensemble/ the attack implementation
attack.py EOT, ensemble, and momentum-PGD loop
encoders.py differentiable encoder interface and the train/judge split
loss.py per-encoder gradient aggregation and VMI support
sampling.py family-aware subset sampling
perceptual.py LPIPS, DCT low-pass, and stealth metrics
targets.py feature-tower decoy centroids from exemplar images
eot.py JPEG-STE, blur, resize, and crop transforms
run_attack.py attack runner and margin scoring
towers/ differentiable loaders for VLM and foundation vision towers
src/veil_pgd/ legacy typographic-overlay CLI and service adapters
scripts/ frontier evaluation, aggregation, and test-set builders
services/ GPU inference service for legacy experiments
@software{veil_pgd_2026,
author = {cebeuq},
title = {VEIL-PGD: Vision-Encoder Imperceptible Layer for Training-Data Protection},
year = {2026},
version = {0.2.1},
url = {https://huggingface.co/cebeuq/veil-pgd},
note = {EOT-hardened ensemble-PGD image-side protection against
vision-language training-data ingestion.}
}
PolyForm Noncommercial License 1.0.0, see LICENSE. Free for noncommercial personal,
research, educational, nonprofit, and government use. Commercial use is not granted under
this license, so please contact the authors for a commercial arrangement.