Building the invisible watermark
hidden in LLM text,
with my own hands

In three lines

The watermark lives in the interchangeable choices an LLM makes when picking the next token — places where either word would be natural. Each single choice reveals nothing, but summed over hundreds of tokens the key-driven bias produces a statistical signature (a z-score) that human text cannot exhibit. Token-level watermarks reliably die after one paraphrase; the semantic-space watermark degrades but can survive — and its survival hinges on how paraphrase-stable the sentence embedder is.

Results: watermarked and unwatermarked, side by side

Example 1 — Semantic watermark (SemStamp-style): degrades, but does not collapse

To make the pair directly comparable, the prompt pins the structure: both essays must open with the exact sentence "Why do people enjoy walking in the rain?" and cover, in order, the sound of rain, the smell and the air, and the calming effect on the mind. Generated in English by Qwen2.5-0.5B-instruct with all-MiniLM-L6-v2 sentence embeddings (fixed-region variant; see the fidelity section for the chained variant). Within that shared skeleton, the wording differs because sampling differs on every run — not because of the watermark. The watermark leaves no visible trace; it exists only in the statistic a key-holding detector can compute. A 0.5B model writes rough English; the texts are shown unedited.

I then had an unwatermarked model (the same LLM with watermarking switched off) rewrite the watermarked essay sentence by sentence, keeping each sentence's meaning — simulating an attacker laundering the text through a clean LLM, which adds no watermark of its own. The wording changed substantially, and —

Example 2 — Token-space watermark (KGW): detected strongly, erased reliably

The same English prompt watermarked with KGW (a logit bonus on a key-derived "green list" of tokens, δ = 2). The detector calls it with overwhelming confidence — and the same paraphrase attack erases it completely, in every trial. Note the prose itself is noticeably degraded ("cuisinarily encode", "the glands of your glands"): on a small model, the logit bonus visibly distorts generation, which is exactly the quality cost the distortion-free schemes were invented to avoid.

Example 3 — Where does the watermark live? Coloring every token

If two texts look indistinguishable, where is the watermark? At temperature 0 (deterministic decoding, no sampling noise) both versions even begin with the same eight tokens — "Walking in the rain can be a delightful" — before the logit bonus flips one close call and the texts fork. Running the KGW detector's own computation and coloring each token by whether the key favors it ("green") makes the watermark visible: the unwatermarked text is 20.0% green, near the chance level, while the watermarked one is 55.0% (Qwen2.5-0.5B-instruct, same prompt, temperature 0).

No individual green token is special — "rain" and "walking" can be green or red depending on the key and the preceding four tokens at that position. The difference exists only in the aggregate over hundreds of tokens.

Example 4 — Quantitative experiments (both token-space schemes)

Detection z-scores with Qwen2.5-0.5B-instruct, averaged over 4 prompts × 200 tokens. Only watermarked text is detected; unwatermarked output, human text (the opening of Austen's Pride and Prejudice), and detection with a wrong key all pass.

Attack experiments: random token substitution degrades the signal gradually, and a single paraphrase pass with an unwatermarked LLM destroys both schemes.

On quality: KGW worsened mean NLL under the same model from 2.25 to 3.02 nats/token (a distorted distribution), while Gumbel-Max stayed at 2.77 — no degradation, exactly as the distortion-free construction promises.

How it works

An LLM samples its next token from a probability distribution, and a passage offers many moments where "either this or that reads naturally". A watermark nudges these interchangeable choices with a rule derived from a secret key. One choice reveals nothing; hundreds of them, aggregated, reveal a statistically impossible habit. Detection needs no model access — only the key and the text.

Every detector is a hypothesis test

All three schemes reduce detection to one question: how many σ does this text's statistic deviate from the no-watermark null? KGW and SemStamp use the normal approximation of a binomial test; Gumbel uses a sum of exponentials (Gamma null).

# KGW / SemStamp: green (or valid-region) rate test
z = (hits − γ·n) / √(n·γ·(1−γ))

# Gumbel-Max: per-token score is Exp(1) under the null
S = Σ −log(1 − r[chosen])
z = (S − n) / √n

z > 4 means one-sided p < 3×10⁻⁵. The test works because generator and detector share a deterministic derivation from (key, context) to the exact same pseudo-randomness — green lists, random vectors, LSH hyperplanes.

Fidelity to the paper, and robustness in practice

This implementation carries the full skeleton of the SemStamp paper — a key-derived LSH partition of sentence-embedding space, valid regions chained from the previous sentence's signature, sentence-level rejection sampling with a boundary margin, and a z-test on the hit rate. One component was deliberately skipped: the paraphrase-robust encoder.

The paper contrastively fine-tunes the sentence embedder on paraphrase pairs so that rewording never flips an LSH signature. I used off-the-shelf embedders instead (ruri-v3-30m for Japanese, all-MiniLM-L6-v2 for English). The gap is directly measurable as the signature-bit flip rate under paraphrase:

That flip rate decides everything. All of today's measurements (1–2 runs per condition, 10–13-sentence essays):

The picture is consistent: KGW sinks to z ≈ 0 after every paraphrase. The semantic watermark degrades gracefully instead of collapsing, and in Japanese it sometimes stays above the detection threshold — because ruri's flip rate is only 11%. But as long as flips are non-zero, survival is probabilistic, and 10-sentence texts leave little statistical headroom. The reason the paper bothers to train a robust encoder is precisely to drive that flip rate toward zero — turning survival from "sometimes" into "guaranteed".

Two implementation lessons surfaced along the way. (1) Off-the-shelf embeddings are strongly anisotropic, so hyperplanes through the origin partition them unevenly — fixed by centering on the mean embedding of a fixed anchor-sentence set. (2) Rejection sampling accepts whatever lands in a valid region regardless of quality, which let garbled or code-switched candidates through — fixed by a candidate quality filter applied identically to the watermarked and unwatermarked sides.

Limitations

• Short texts cannot be detected. Evidence accumulates with token (or sentence) count. Empirically, a 57-token constrained list scored z = 3.90 — below threshold — while the 113-token version of the same format reached z = 5.38. The sentence-level semantic scheme needs even longer texts.
• Low-entropy passages resist watermarking. Code or factual statements with essentially one correct continuation offer no interchangeable choices to bias. Forcing KGW onto them surfaces as quality damage (measured: the δ = 2 logit bonus broke formatting-instruction compliance and caused code-switching).
• Semantic robustness depends on the embedder. With off-the-shelf embedders, paraphrase survival is probabilistic (see the table above). Translation, summarization, or heavy rewriting that reorganizes sentence-level meaning defeats the scheme even with a robust encoder.
• Detection is not proof of AI authorship. Having an LLM proofread a human's text also imprints the watermark. Detection shows only that the model (key) processed the text — Anthropic states the same caveat about Claude's watermark.
• This is a minimal educational implementation. Robust-encoder training, key rotation, multiple keys, and production-scale false-positive evaluation (SynthID-Text was evaluated on ~20M live requests) are out of scope.

Future work

• Training the paraphrase-robust encoder — the one missing component. For Japanese, contrastively fine-tuning ruri on paraphrase pairs (JSNLI, PAWS-X) should push the flip rate to a few percent, turning survival from occasional into reliable.
• Multi-bit payloads. Embedding and decoding user IDs or timestamps rather than a single "AI or not" bit (Qu et al., USENIX Security 2025) — the leading direction for content source tracing.
• Post-hoc watermarking. Watermarking through a black-box API with no logit access (SAEMark, NeurIPS 2025), selecting among candidate outputs by Sparse Autoencoder features.
• Unifying distortion-freeness with paraphrase robustness (PASA, ICML 2026) — the two properties this project verified separately, achieved in one principled framework.
• Checking against Anthropic's detector. The technical details of Claude's watermark are unpublished; once a detection tool ships, I want to compare its behavior with this implementation.

References

1. hellorusk, "Claude started watermarking its text, so I studied how LLM watermarking works" (Japanese), Zenn, 2026. (The article that motivated this project)
2. Kirchenbauer et al., "A Watermark for Large Language Models" (ICML 2023). — KGW
3. Kuditipudi et al., "Robust Distortion-free Watermarks for Language Models" (2023). — Gumbel-Max formalization
4. Dathathri et al., "Scalable watermarking for identifying large language model outputs" (Nature, 2024). — SynthID-Text
5. Krishna et al., "Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense" (NeurIPS 2023). — the paraphrase attack
6. Hou et al., "SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation" (NAACL 2024). — the basis of this implementation, including the contrastively trained robust encoder
7. Qu et al., "Provably Robust Multi-bit Watermarking for AI-generated Text" (USENIX Security 2025).
8. Yu et al., "SAEMark: Multi-bit LLM Watermarking with Inference-Time Scaling" (NeurIPS 2025).
9. Ai & He, "PASA: A Principled Embedding-Space Watermarking Approach for LLM-Generated Text under Semantic-Invariant Attacks" (ICML 2026).
10. Anthropic, "How Claude marks AI-generated content" (2026).

Models used: sarashina2.2-3b-instruct-v0.1 (SB Intuitions), ruri-v3-30m (Nagoya University), Qwen2.5-instruct 0.5B/1.5B (Alibaba), and all-MiniLM-L6-v2 / all-mpnet-base-v2 (Sentence Transformers) — all running locally on Apple Silicon via MPS. Every number on this page was measured on 2026-08-13.