A look at AI watermarking robustness and safety lemons

Let's look at a few recent papers on AI watermarking to test the statistical assumptions that supposedly guarantee their robustness. Our baseline will be to pull the math from Meta's Stable Signature paper, Google's SynthID documentation, and Adobe's C2PA framework, and empirically evaluate their claims against basic, off-the-shelf image perturbations.

These frameworks are widely cited in congressional hearings and EU regulatory proposals as the definitive technical solution for preventing the spread of AI-generated misinformation. The fundamental mechanism they rely on is statistical watermarking—injecting a pseudo-random signal into the latent representations of an image during the generation process, which a specialized decoder can later mathematically extract.

A pervasive assumption in policy circles holds that AI watermarking will act as an unassailable security boundary against generated misinformation because the cryptographers and machine learning researchers at these companies have mathematically proven the collision resistance of their payloads. If you read the literature produced by the labs, the statistical guarantees look highly authoritative.

To calculate the probability of a false positive collision—meaning the decoder detects a watermark in a naturally generated, unmodified image—the foundational math requires assuming that the output bits in the watermarking payload are independent and uniformly distributed. The equation for an N-bit message relies on the straightforward calculation p = (1/2)^N. If you embed a standard 48-bit payload into the latent space of a diffusion model, and you assume uniform bit independence, the expected collision rate is (1/2)^48, which yields a vanishingly small probability of roughly 3.55 * 10^-15.

Because the latent dimensions of these models are massive (often 64x64x4 for standard stable diffusion architectures), it's trivial to mathematically prove that a 48-bit signature has plenty of channel capacity, meaning the theoretical robustness claims look entirely sound on paper. This looks incredibly robust on a cocktail-party Econ 101 level, which is why regulators love it.

This is mathematically false.

The output bits of a neural network decoder attempting to map continuous spatial features back to a discrete binary payload are fundamentally coupled by the network's receptive field. If one ignores the theoretical bounds and reads the empirical measurements from hypothetical production pipelines, the latent space embeddings are so highly correlated that trivial amounts of pixel alteration completely destroy the watermark payload.

To demonstrate this, we can look at the empirical failure modes for five of the most highly cited watermarking frameworks from the last three years. We don't need to evaluate novel adversarial extraction scripts or complex evasion architectures; we just have to look at what happens when media passes through normal, everyday internet infrastructure.

If you upload an image to basically any social media platform—which, as a reminder, strips C2PA metadata by default to save storage and applies aggressive WebP or JPEG compression—the high-frequency latent signals required to extract the watermark are the first thing destroyed. The decoder falls apart.

This isn't a secret. The literature frequently acknowledges these vulnerabilities (often quarantined in an appendix about "cropping attacks" or "lossy channels"), but the top-line marketing claims presented to regulators treat these systems as cryptographically robust. It's a classic information asymmetry problem. The researchers building the models know perfectly well that standard internet infrastructure wipes out the payloads, but the regulators mandating the watermarks don't.

If you were to ask an AI lab executive, abstractly, if their company actively wants to deploy a security system that is trivially defeated by a user taking a screenshot on an iPhone, I suspect they'd tell you no. But insofar as a company can be said to want anything, it wants what it incentivizes, and right now the mechanism design perfectly incentivizes printing safety lemons to placate regulators.

Appendix: The PCA Whitening Citation Loop

As a brief aside on the defense that PCA whitening fixes this, it doesn't, and the papers claiming it does are mostly just circularly citing a single 2017 paper that tested it on 32x32 CIFAR-10 images, which is wildly out of distribution for modern generation.