Ask HN: Does AI watermarking present a new attack vector?

4 points by nathanfig 30 days ago | 6 comments

One thing I have not been able to determine from Claude's documentation on watermarking[0] is what kind of metadata they store in association with a given watermark. Ostensibly they could make the fingerprints as unique as they want, possibly down to the exact time, user and session.

If so, this seems like hidden risk that AI users are probably not considering. Any code you write now carries information that you might not want revealed. If a bad actor gets the keys then they may be able to de-anonymous open source contributors who want to stay hidden. Or perhaps enough fingerprints across a sample could reveal internal organization details a company would rather not disclose.

Someone with more imagination can probably come up with better examples, it just seems like an attack vector that I haven't seen much consideration for.

[0] https://support.claude.com/en/articles/16266773-how-claude-marks-ai-generated-content

armchairhacker 30 days ago |

If you really care, a) use an open model or b) obscure your identity to Anthropic.

Otherwise, you're already vulnerable: Anthropic can silently record every change Claude makes to every codebase, maybe they already do.

nathanfig 30 days ago | |

I hope open weights continue to gain traction for this reason too.

But injecting information into outputs is a little different than Claude keeping tabs internally. The point is that we do not know what information is being shipped with the fingerprint itself.

Wpnx330 29 days ago |

Hmmmm. If every model vendor starts training their models to watermark their content. It could become a real issue for privacy.

local cleaned models would become even more important. No longer just to cut the token bill, or allow enterprising individuals to host adult content sites as a business.

jjgreen 30 days ago |

Come now, this isn't about attack vectors, it's about you getting caught using LLMs where you shouldn't.

gsch1 30 days ago | |

That's a philosophical question in itself. Where shouldn't you use it?

jjgreen 30 days ago | | |

Exams, writing to your grandmother, surgery, ...