Awesome-GenAI-Watermarking

This repo include papers about the watermarking methods for generative AI models. Watermarking is a method for embedding an imperceptible, but recoverable signal (payload) into a digital asset (cover). With generative models, there are approaches which train the model to produce the watermark in every output and this behaviour should be hard to disable. We refer to this as "Fingerprint Rooting" or just "Rooting".

• 1. Introduction
  • 1.1 Watermarking Goals
  • 1.2 Differences Between Watermarking Schemes
  • 1.3 What Information is Transported by the Watermark?
  • 1.4 Attacks on Watermarking
• 2. Image Domain
  • 2.1. Papers on Watermarking Diffusion Models Image
  • 2.2 Watermarks to Guide Other Objectives
  • 2.3. Misc Papers (to Be Categorized)
• 3. Audio Domain
  • 3.1. Papers on Watermarking Audio
  • 3.2. Audio Synthesis Datasets
  • 3.3. News on Audio
  • 3.4. Further Links on Audio Synthesis and Detection
• 4. Text Domain
• 5. Related News
• 6. Generative Model stealing Papers
• 7. Survey Papers
• 7.1 A Systematic Review on Model Watermarking for Neural Networks
• 8 Further Links

1. Introduction

---

1.1 Watermarking Goals

• Deep fake detection (Is a digit asset AI-generated?)
  • Counter misinformation
  • Prevent data crawlers from picking up generated content and cause "Self-Consuming Generative Models Go MAD"
• Deep fake attribution (By whom (which user of A model API) has it been generated?)
• Enhanced Model Fingerprinting (By which model has it been generated?)
• IP protection
  • Protect valuable models
  • Protect valuable training data (e.g. style)
• Tamper Localization (Where has an asset been doctored?)
  • see "EditGuard: Versatile Image Watermarking for Tamper Localization and Copyright Protection"

1.2 Differences Between Watermarking Schemes

• Fingerprint Rooting vs. Post-Hoc Watermarking
  • Fingerprint Rooting
    • The model is manipulated in a way that its usual generating process produces watermarked output
    • This DOES protect open models
  • Post-Hoc Watermarking
    • The Watermark is added after the fact, in a separate process
    • This does NOT protect open models, as the watermark embedding procedure can be simply disabled
• Static vs. Dynamic Watermarking
  • Static watermarking
    • "[...] specific pattern in its static content, such as a particular distribution of parameters" (see "Intellectual Property Protection of Diffusion Models via the Watermark Diffusion Process")
  • Dynamic Watermarking
    • "[...] specific pattern in model’s dynamic contents, such as its behavior.", e.g. trigger-prompt-watermark-backdoors
      • Requires at least API-access to the model, as the watermark will only be present upon specific trigger
      • Examples for introducing backdoors in general (not for watermarking specifically) into diffusion models:
        • TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets
        • How to Backdoor Diffusion Models?
• Tuning mechanism
  • Via Training data
    • e.g. Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data) -> "Plug-and-play", as it works on all architectures due to transferability
  • Via joint fine-tuning of a model and a decoder (taken from a encoder/decoder-pair) on few samples
    • e.g. The Stable Signature: Rooting Watermarks in Latent Diffusion Models (Requires latent generative model)
• Flexibility: How quick can a model be rooted:
  • Full training for each instance
    • e.g. Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data
  • Fine-tuning for each instance
    • e.g. The Stable Signature: Rooting Watermarks in Latent Diffusion Models
  • Fine-tuning once
    • e.g. using message matrix as in Flexible and Secure Watermarking for Latent Diffusion Model

1.3 What Information is Transported by the Watermark?

• Generated asset yes/no
• Identity of watermarking party
• Identifier of the asset in provenance database (can replace perceptual hashing, mentioned in "RoSteALS: Robust Steganography using Autoencoder Latent Space")

1.4 Attacks on Watermarking

• Watermark removal
  • Removing a watermark from a given digital asset
  • Attacker goals
    • A fake asset can be claimed to be real
      • Disable IP claims
      • Misinformation
      • Sensor spoofing (weird, but it was mentioned in Responsible Disclosure of Generative Models Using Scalable Fingerprinting)
  • Robustness property
    • Removing the watermark should decrease the asset quality. This negates the usefulness of the asset for malicious goals
• Watermark forgery (referred to as spoofing by Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks)
  • Adding a watermark to a given digital asset
  • Attacker goals
    • False IP claims
    • A real asset can be denounced as fake (Misinformation)
    • Reputation loss after linking obscene content with model (mentioned in Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks)
• Model purification
  • A watermarked model which should only produce watermarked output, even if distributed to untrusted parties (i.e. Stable Signature), is "purified" in a way that removes the watermarks in its output.
  • Attacker goals
    • Obtain a model which does not produced watermarked content
  • Robustness property
    • Removing the watermark functionality of the model should decrease the output quality. This negates the usefulness of the asset for malicious goals

Threat models

• Whitebox
  • Attacker has full access to a generative AI model
• ... TODO

Difference between Watermarking and Cryptography

2. Image Domain

---

2.1 Papers on Watermarking Diffusion Models (outputs) (Image)

2.2 Watermarks to Guide Other Objectives

2.3 Hardware-Assisted Watermarking

2.4 Misc Papers (to be categorized...)

3. Audio Domain

---

3.1 Papers on Watermarking (Audio)

3.2 Audio Synthesis Datasets

• For more (but older dataset see Awesome-DeepFake-Learning)

3.3 News on Audio Watermarking

• Microsoft: Introducing the watermark algorithm for synthetic voice identification

3.4 Further Links on Audio Synthesis and Detection

• Github topics: Audio Synthesis
• Github topic: Audio Deepfake Detection
• Awesome Deepfakes Detection
• Awesome-DeepFake-Learning
• Stable Audio: Fast Timing-Conditioned Latent Audio Diffusion (is supposedly also very fast, like HIFI-GAN)
• Meta AI: VoiceBox
  • SotA Speech model with many functionalities (noise removal (barking), style transfer, ...)
• TTS libraries (including speech synthesis)
  • Coqui-AI library
  • Mimic3
  • Amphion-AI
  • Tortoise TTS
    • pet project by some student
    • includes tools for detection

4. Text Domain

5. Related News

---

• Coalition for Content Provenance and Authenticity (C2PA)
  • Is based on a trust model with signing authorities which certify signer of some digital asset through a chain of trust, similar to internet PKI.
  • C2PA Specifications
    • Explainer
      • "Provenance generally refers to the facts about the history of a piece of digital content assets (image, video, audio recording, document). C2PA enables the authors of provenance data to securely bind statements of provenance data to instances of content using their unique credentials. These provenance statements are called assertions by the C2PA. They may include assertions about who created the content and how, when, and where it was created. They may also include assertions about when and how it was edited throughout its life. The content author, and publisher (if authoring provenance data) always has control over whether to include provenance data as well as what assertions are included, such as whether to include identifying information (in order to allow for anonymous or pseudonymous assets). Included assertions can be removed in later edits without invalidating or removing all of the included provenance data in a process called redaction."
      • "In the C2PA Specifications, trust decisions are made by the consumer of the asset based on the identity of the actor(s) who signed the provenance data along with the information in the assertions contained in the provenance. This signing takes place at each significant moment in an asset’s life (e.g., creation, editing, etc.) through the use of the actor’s unique credentials and ensures that the provenance data remains cryptographically bound to the newly created or updated asset."
      • "Soft bindings are described using soft binding assertions such as via a perceptual hash computed from the digital content or a watermark embedded within the digital content. These soft bindings enable digital content to be matched even if the underlying bits differ, for example due to an asset rendition in a different resolution or encoding format. Additionally, should a C2PA manifest be removed from an asset, but a copy of that manifest remains in a provenance store elsewhere, the manifest and asset may be matched using available soft bindings."
        • This includes ISCC - Content Codes
      • -> allows authors and subsequent actors to sign assets and make this act publicly known to establish the asset's history.
• Open-source tools for content authenticity and provenance
  • Uses manifests defined in C2PA Specifications
  • Enables camera manufacturers to insert authenticity meta-data on-device at time of capture
• Deepmind SynthID
  • Images (Vertex AI imagen)
  • Audio (Google DeepMind's Lyria)
    • Transfer artist's styles to audio prompts (style transfer)
    • Most likely, the watermark will include the prompt used to generate audio (e.g. Artist name). This allows for copyright claims.
  • In Beta (as of 26 Mar 2024)
  • Closed off as can be (as of 26 Mar 2024)
• Google hosted a workshop in June 2023 (Identifying and Mitigating the Security Risks of Generative AI): "Watermarking was mentioned as a promising mitigation. They are robust when attacker has no access to detection algorithm"
• Watermarking is identified as tool for establishing trust in a post GenAI environment by big tech (OpenAI moving AI governance forward statement, Google "Our commitment to advancing bold and responsible AI, together") and government (Biden-⁠Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI )
• Watermarks can already be easily inserted into Stable Diffusion models (This method invisible-watermark repo is referenced by the official Stable Diffusion repo). This is based on Digital Watermarking and Steganography" (DwtDct and DwtDctSvd). Also see watermark option in the Stable Diffusion repo https://github.com/CompVis/stable-diffusion/blob/main/scripts/txt2img.py#L69.
• Stable Diffusion XL recommends using invisible watermarks pip package. The supported algorithms are Dwt, Dct, and RivaGAN .
• China bans GenAI without Watermarks

6. Generative Model stealing Papers

---

7. Survey Papers

---

7.1 A Systematic Review on Model Watermarking for Neural Networks

Summarization of the systematization given in this review.

Taxonomy

• Embedding method
  • Watermark in model parameters
  • Trigger-Watermark-Backdoor
• Verification access
  • Whitebox (access model parameters)
  • Blackbox (access via API)
• Capacity
  • Zero-bit (is watermark exists)
  • Multi-Bit (watermark contains arbitrary info)
• Authentication
  • Model is watermarked
  • By whom model is watermarked
• Uniqueness
  • All model instances carry same watermark
  • Different model instances carry different watermarks

Requirements & Security Goals

Threat Model

• Attacker Knowledge:
  1. existence of the watermark
  2. model and its parameters
  3. watermarking scheme used
  4. (parts of) the training data
  5. (parts of) the watermark itself or the trigger dataset
• Attacker Capabilities (irrelevant)
  • passive (eavesdropping)
  • active (interaction)
• Attacker Objectives
  • For what is model being used by the attacker? (rather unspecific)

Attacks against Watermarking

• Watermark Detection (weakest)
• Watermark Suppression, i.e. avoid watermark verification
  • e.g. dissimulating any presence of a watermark in the model parameters and behavior
  • e.g. suppressing the reactions of the model to the original watermark trigger
• Watermark Forging
  1. Recovering the legitimate owner’s watermark and claiming ownership (if there is no binding between the watermark and the owner)
  2. Adding a new watermark that creates ambiguity concerning ownership
  3. Identifying a fake watermark within the model that coincidentally acts like a real watermark but actually is not
• Watermark Overwriting
  1. Adding Watermark to model with deactivating old one (strong)
  2. Adding Watermark to model without deactivating old one (weak)
• Watermark Removal
  1. depends on the presence of a watermark
  2. depends on the underlying watermarking scheme
  3. depends on availability of additional data, e.g. for fine-tuning or retraining
     • Methods
       • Fine-Tuning
       • Pruning
       • Quantization
       • Distillation
       • Transfer-Learning
       • Backdoor Removal

Categorizing Watermarking Methods

• Embedding Watermarks into Model Parameters
  • Adding patterns into model which can be verified locally
• Using Pre-Defined Inputs as Triggers
  • Adding behaviour triggered by special input
• Using Model Fingerprints to Identify Potentially Stolen Instances
  • No additional action needed, just recognizing a model based on some criteria

8 Further Links

Practical tools for text watermark checks

• Mydentify AI Watermark and Claude Checker — browser tool for checking AI-watermark signals and Claude-related text artifacts.

• Awesome github with resources on neural IP protection https://github.com/ZJZAC/awesome-deep-model-IP-protection