I really don't think Anthropic has the leverage to pull this kind of thing off in the face of competition. OpenAI doesn't do it (yet), Grok doesn't do it, neither does Deepseek. I'm sure the latter two have plans to intentionally never do it.

Who is even asking for this? Sounds like something some obsessive internal employees would push on the world.

> The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

And this attitude is incompatible with any models produced by frontier labs. Your needs will always be subordinate to and in service of the needs of the corporation that produced the model. And we haven't even gotten to ads yet.

Heck. The entire process itself of LLM text generation is a perversion of writing.-

Further heck: It can be said it ain't even writing.-

Best writer I’ve ever seen not understand writing. Proof positive that you needn’t understand something to produce lots of it!

There will be services online that copy/paste (or you provide) the text from ai, they OCR it to get rid of watermarking, then feed it back to you for a fee.

The premium upgrade will be "obfusicators" that further modify the text sporadically to make it look more like a human wrote it.

Students for one, will gladly pay.

> the only acceptable answer for why an LLM should choose bananas instead of pineapple (or coconut, or guava, or papaya...) is that it has determined that it’s the best fit for the intended meaning, tone, and sentiment of the text.

It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.

Get fucked? Really? As if the quality of machine-generated prose were somehow sacred?

Is Gruber only now waking up to the idea that LLM companies do not give a shit about the quality of the writing they generate? That they're disdainful of the entire concept of writing as a profession? What rock has he been under?

If you care one iota for the quality and craft of your writing, you would already recognize that any degree of AI "processing" obliterates stylistic choices. You're left with readable text, but it's fluffy, meandering, and has a cadence to the writing that screams to the reader "every second you spent on this was wasted."

> I want any LLM I use to choose the very best, most precise words at every single decision point.

No, you don't. If you wanted that, you would set the temperature parameter to 0. But that would lead to less desirable results, not better. LLMs do not set the temperature to 0; they typically set it 0.4-0.7.

Humans have been inserting benign noise into their writing without affecting the signal for thousands of years. They called it "style".

Apparently superfluous descriptions, unnecessary words, and paragraph footnotes are all fine, but gumbel softmax (or whatever's going on here) isn't.

"possibly build tools to take a chunk of writing and try to remove your fingerprint from it, i.e, protect anonymity,"

The standard procedure to do this, is to chain translations to other languages and back. The message remains, but the wording will pick up some noise. --Dec 30, 2008.

I predict we will soon see more AI providers implement watermarking, because it allows them to filter out AI generated data from their training sets. That is probably the real reason why they implement it. Gruber just doesn't get it.

As a university professor, back in the early days of this when students were just copy / pasting output directly I was really hoping the model builders would implement steganographic fuzzy hashes.

Not that it's the ideal solution. But it would be really neat.

"The more words in the text, the more accurate the analysis will be that the text was generated by a specific AI model or not."

And this is why the latest Claude models blather so much more. All for the sake of this "watermark."

LLM inference already isn't deterministic, the watermarking technique only limits the space of possible random seeds. There is no reason to believe that this subset of seeds somehow produces lower quality output.

Ai is a perversion of writing. A useful one, but still.

I don’t see how this makes it worse.

> It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of embedding hidden clues within the text to suggest its provenance.

Yeah, that about sums it up!

The watermark is based on stylometry - and we all know Claude has a style. But it doesn’t have to - everyone knows caveman - so not sure there is enough room in all cases for it to work.

Fundamental misunderstanding of what AI is and what it does if this watermark is preversion of what AI is already doing.

Not sure about the strength of that particular argument, but can't say I have any faith in this being used to benefit end consumers.

Somehow they'll find a way to use this for regulatory capture

What a strange take. LLMs themselves are a perversion of writing. I couldn't care less about the implementation details of the PRNG they use for next-token sampling (well, as long as they're not stuffing a user ID in there).

gruber is really out of his element with ai commentary, I fully support the general skepticism but he's seemingly arguing against something he doesn't quite grasp

What's gonna be a real trip, is when you can tell which LLM produced it by the subtle pattern recognition you'll have developed to catch the watermarks

Is this 'best word' with us in the room right now?

> because the nature of the watermarking algorithm requires it to sometimes increase the probability of selecting a worse word choice and decrease the probability of selecting the model’s best choice.

... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s

> My initial speculation was that maybe they’d hide invisible non-printing Unicode characters in the text.

Or just em dashes? /s

tl;dr:

> It’s unacceptable for a tool to sacrifice an iota of clarity, coherence, meaning, quality, etc. for the purpose of (watermarking)

And an example of the impact of watermarking on word choice [0]:

> The results of the study were quite [important || significant || substantial || notable]

The meaning of the sentence to changes slightly even in just this tiny example. Imagine the degradation when applied across an entire response!

[0] https://declaude.org/watermarking/

What is even the point of watermarking AI slop supposed to be? All it does is provide people with the false implication that anything which is not watermarked must not be AI-generated.

I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.

I recently found there was no website covering a historical subject, with sites begging to be visited. The information was scattered and not presented in the form people would find most useful (a map).

A couple of days later it was up with the most comprehensive review of the available info summarised and referenced ready for a human researcher to explore.

This was only possible due to AI. It would have taken me weeks to chase it down and summarise it, so it would never have happened. Let's get off our high horse about AI writing.

This (Anthropic's "watermark" stance, I mean) is so fundamentally ludicrous that I have assumed it is a (wholly insincere, but arguably pragmatic, at least from their perspective) attempt to deal with the EU and their latest misguided, ham-fisted attempt to solve a real-world problem by drenching the entire world with more regulatory slop[1].

The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule.

[1]: https://digital-strategy.ec.europa.eu/en/policies/code-pract...

(FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)

Respectfully, you are all missing the point.

Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.

As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.

It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.

... If watermarking is a perversion of writing, what does that make 'writing' with an LLM?

The author makes a lot of great points. I find it surprising that somebody who has such a nuanced appreciation for the subtleties of language would be in the crowd complaining about the watermarking policy. I expected this kind of complaints from mostly students interested in academic dishonesty, who generally don't have enough command over written language to notice the slight decrease in output quality.

Whole watermark thing is just bullshit. What if I add a watermark text and another AI as well. How many watermarks and who is the real creator. It just doesn't make sense. I ll eat my shoes if this concept is still a thing in 6 months

A) without water marking, llm will poison the open internet and poison their own source of fresh new material. B) without watermarking, it potentially exposes an infinite deluge of garbage getting to people , fostering hate, eventually leading to dis-engagement, ergo destroying their training pipeline. C) ...Sadly, them water marking the claude chats with absolute dementia levels of output text is beyond me, i seriously can't tolerate this bullshit "randomly replace words with "similar" words", bs anymore, the output that claude makes right now is outright *corrosive* to my eyes, eventually leading to their training pipeline dying if devs dis-engaging ( of the 3 i think this one is the worst )

I think they need to get their shit-together and realize this is a death warrant for the tech ( in my opinion ).

The same absolute morons who gave us cookie consent strike again.

I swear, one of those days I will get into politics just to fight those two things, and the cottage industry of batshit crazy lawyers that gave birth to those things.

Such a stupid regulation. Don’t make the rest of the world suffer because of one dumb law passed for a fractional share of users.

This also, just another precedent of anti-user, pro Authoritarian, from LLM companies.

It's because the individuals who write those laws are literally trying to shove neo-Nazi policies into EU practice. I wonder how many people would make policies like this if they and their families were publicly identified and criticized as neo-Nazi elements in society.

You think someone will write Nazi-promoting AI policy like this when society is encouraged to look at their families as examples of neo-Nazi corruption? When their wives' and kids' friends spurn them while their families engage in obvious criminal activity to harm world productivity?

Critics need to be more precise.

They just want to prevent their own slop being fed back into their AIs.

I think this would be a good guard on websites to use it as another content protection layer.

This is the obvious solution to watermarking text in a way that isn't easily removed by a dumb tool.

The output quality will likely suffer as stated in the article although this can be mitigated to an extent by only enabling it on more irrelevant filler text while leaving the more functional sections untouched.

The solution using Unicode tricks amounts to malicious compliance as only the most unsophisticated users are going to fail to remove the AI watermarks when trying to pass off AI slop as their own prose.

The real solution here is not having stupid EU–tier laws in the first place.

Great, now Claude will sound even more generic. It’s not only annoying - it’s also dumb. We are all correct to push back on this load-bearing issue

Awwww, little blogger can’t write his slop anymore without rest of the world knowing that he’s not the author?

"My writing is my work, and Anthropic's current strategy is aggressively writer-hostile."

Tl;dr the author is upset that the slop generator will generate slightly different slop, and calls this perversion of the text, ironically missing the fact that the entire technology takes existing human work and perverts it in order to give him the output that he seemingly enjoys.

Perhaps it's not actually irony, perhaps it is hypocrisy.

This isn't to say there aren't correct statements in the article, but its framed very strangely.

So Anthropic goes on the shit list with Sony Blu-ray and its cinavia watermarking.

No, machines being used to replace human writing is a perversion of writing. Some might call it worse than cannibalism[0]. If you're only noticing now because Anthropic is changing things behind your back, well... I've got some bad news for you, but the entire cloud-hosted subset of the AI space, especially Anthropic, is premised on the fact that doing things behind your back to their models is socially preferable, or worse, should be outright mandated.

While I generally hate legal mandates to stab your customers in the back, in my opinion there is no harmless way to use AI and mandatory text watermarking is a good bare minimum. The EU probably made the right call. The entire AI space - open models included - is predicated upon worker exploitation, replacement, and deskilling; we should at least be able to know how much of our media diet has Anthropic's fingerprints on it.

A lot of hay is made over the pretraining process in which copious amounts of stolen data are trained on; but parallel to this is a huge data labeling and human feedback operation staffed almost entirely by people in third-world countries with robust English as a Second Language (ESL) programs. The thing is, AI models already watermark their text, they just happen to do so with the textual watermarks of the Indians and Nigerians that the AI companies hired to do RLHF because they were cheap. That's why certain AI models love the word "delve" so damned much. It's neocolonialism, designed specifically to do the kind of replacement the anti-immigrant idiots keep screaming their heads off about[1].

Furthermore, as we've seen with Hank Green, even non-cannibalism-adjacent AI usage is a recipe for worker deskilling and AI psychosis. The other half of the RLHF pipeline is to turn a pile of compressed text into a chatbot that feeds you a steady drip of unsourced information while praising you every time you spot one of its lies and never saying no[2]. This is a recipe for addicting your customers.

Also, this might just be because this is on daringfireball.net, but I can't help but think the author has an axe to grind against the EU because the EU mandated Apple sign third-party app stores. The fact that he's balking at Anthropic not going along with gating the watermarks to just the EU seems downstream of this - "why aren't you maximally attempting to resist the EU?"

[0] https://www.youtube.com/watch?v=YCPAIg7RUq8

[1] To be clear, upwards of none of the far-right have actually clued into the fact that AI is trained by underpaid immigrants, mainly because it doesn't fit the narratives the people running the far-right want to push. There are some AI robotics companies that are even very explicit that their robots are there primarily to launder foreign labor into rich companies and make permanent labor arbitrage.

[2] Continuing on from [1], the far-right actually really loves AI specifically because it rarely says no to even stupid ideas, even if it's also a manifestation of everything they claim to hate.

Man this watermark nonsense is all over the place

Now for the human generated watermark. Timez to addd the speelling mistakes, decreaze the usegage of big words and proper gramicaly usuage. Wish I was joking.

Using AI is the only perversion here And what a self-reporting from all the AI bros!

The objection isn’t that normal sampling is somehow pure. It’s that watermarking uses token choice to carry an additional, covert signal. The quality cost may be small on average, but it is still an optimization constraint unrelated to meaning or style—and it turns ordinary prose into provenance metadata. The more serious questions are detection reliability, false positives, and what happens after human editing. https://prachub.com/

What company compliance teams do is once again, the responsibility of companies, not the EU.

And the move away from cookies was addressed. That's why we have the GDPR now.

We can always write new regulation, but its important to handle things as they come up and not use technological change as an excuse to do nothing.

Just copy Claude’s output and shove it into Gemini and ask to rephrase.

It’s super ironic to see this many bots on this thread. Sorry dang.

Not telling someone you used AI is a perversion of writing. Also agree that an AI proofreader should not claim authorship, but in most other cases, the AI is not reading your mind, it's only watermarking its own usage, and we kind of need more of that.

you can remove AI company invisible text watermarks at www.demarkify.com - and also perform other tonal text changes like removing AI lexical tells

Reminder: your favorite distilled model does not treat you, the customer, as an adversary and mess with your output.... May the free market win.

I know (and understand why) a lot of people cheer the EU’s increasingly vast regulatory environment as being “pro consumer” but I’m really tired of said regulations being inflicted on the rest of the world. If this is what Europeans want for themselves that’s fine. But I no more want their regulations to be the de facto world’s any more than I want China’s.

The first thing that came to my mind was "security theater." Making people think that AI is detectable could have the same effect as actually making it detectable.

Mind that even in their first example, "The results of the study were quite (important | significant | substantial | notable)", the meaning is by no means interchangeable.

"Important" refers to impact, "significant" to the statistical qualities of the underlying hypothesis, "substantial" to the work involved, and "notable" is a referential judgement by the speaker. The implied normalization of words and their respective meaning also marks one of the mechanisms how "slop" is typically creeping into the productions of "broad verbose interchange replicas" (it's all interchangeable, and a choice isn't really that, a choice, isn't it?).

The objection seems to be that Claude will always write worse prose than a human writer, even if the writing Claude generates is understandable.

Yeah, John. We're all OK with that.

Wait are we supposed to be mad because clankers are displacing human creative workers, or mad because clankers don't do their level best when producing creative works because they are forced to watermark text? Or is it that they use all the water (I know they don't but are we supposed to be mad about it still)?

I can't keep up with the current Chinese psycop. There should be some kind of status page like whywewantamericatofailataitoday.ai so we can keep up with it.

A lot of that rant is nonsense. Is Gruber also outraged that a RNG is already involved in everything LLM produce with a non-zero temperature? Surely it already leads to widespread use of “non-optimal words”. Simply the idea that there is a single optimal choice, down to every single word, to convey a meaning and anything straying from that is adulteration is laughable. It’s very difficult to take seriously an argument based on this kind of foundation.

He had a very similar rant a couple of days ago when he somehow thought that they would use invisible characters. It’s just as useless.

Posting for visibility a reminder that this type of watermarking is comically easy to beat. See Scott Aaronson's 2023 research in which he coined the term "pineapple attack": one must simply prompt the model to add the word "pineapple" after every other word and then do a search + replace to remove the word, to obliterate the watermark altogether.

https://www.youtube.com/live/2Kx9jbSMZqA?si=0QgCPBX2_KPZ0QTU...

I think it's pretty dishonest of Anthropic to frame their watermark as EU regulation compliance. The EU regulation, from my understanding, requires AI content to be labeled for human viewers. In the meanwhile the Anthropic new release on the watermark says this.

> The difference between watermarked and un-watermarked text will not be distinguishable to readers

https://www.anthropic.com/news/claude-text-watermark

Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.

If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.

Watermarks are context poisoning.

If you don’t like it don’t use it

I'll be honest, who fucking cares? Why would you use AI to write for you and then complain that people know AI wrote the code?? If you know people wouldn't like it, why even try!?

How does this work for code?

you can remove AI company watermarks at www.demarkify.com

is that why everything Claude writes lately sounds like a riddle?

I needed another reason to cancel my Claude sub. Thanks Anthropic!

This is akin to adding a giant watermark on things one would made with a free product "Made with XXX". Except you're paying $200/month for it, and there's no way to disable that watermark.

I don't disagree with EU regulations, but I strongly believe the onus should be on the content publisher, not the toolmaker. If the toolmaker watermarks whatever his tool produces, it opens a giant can of worms that cannot be closed. That means anything and everything you make with this tool is no longer fully yours, it contaminates everything and makes your work traceable. Who wants that?

I was already annoyed by the fact that Claude marked everything it did on my repos under its own account (I didn't ask for any of this), but now everything is invisibly marked, even the code. Not that I care that my writing would be watermarked, since I'd rather write my stuff myself, but code? No thanks.

Meanwhile I'm running a DeepSeek V4 Flash or Pro, or a Qwen3.8, and it writes my code without a peep. Resulting repos are clean, just the way I want them. No 'Claude' account, no watermarking, nothing. I won't be looking back after having tried these new models. Whoever makes good models that don't broadcast their maker will get my business.

This watermarking will simply push people more towards Chinese models. Keep pushing in the wrong direction Anthropic. Doing this right before an IPO is a great idea.

Gruber is a smart, thoughtful man. This is a bizarre take from him.

It exhibits an undeveloped understanding of LLMs, and a righteous view that generated prose should assimilate... which should be offensive to organic intelligence.

Issues with the proprietary nature of Anthropic's watermarking aside, we will look back on this as a 'thank god' moment in the history of LLMs.

Imagine using a random word generator and getting upset that the random words it generates are not truly artisanal random words.

Wow, never has a single article revealed the incompetency of a tech writer.

This is the yet another embarrassing idea from the EU they came up with, if they could focus less on dumb things and more on providing real support to the development of technology maybe we could have more healthy competition that will eventually lead to better overall technology in the hands of everyone instead of acting like dumb chickens and not just being passively useless but also stopping the real labs from doing the real work into doing this bs

"Anthropic's . . . Claude is a Perversion of Writing."

FITFY.

I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.

This scheme has a fatal flaw, the same secret key is used for watermarking and detection. This means you need to trust that Anthropic:

* Shows the real result from their detector instead of manipulating them, you have no way to verify * Has good enough security to prevent a key leak * Rotates keys to reduce the impact of a leak (once a key is leaked anyone can rewrite text to look more/less claude generated and it becomes useless) * Will not secretly give watermark-less access to governments or high profile corporate users * Will not use multiple secret keys to track individual users. This one might be less realistic because embedding ~32 bits of signal would probably affect quality a lot more than 1 bit.

And don't forget that the detection API will work as an oracle. If it detects your content you can send it to a different model and try again until it comes back clean.

>in areas where there is an arbitrary choice between particular words or terms within the code, the watermark can be used, such as comments within code.

I wonder if this is why Opus 5 keeps writing excessively long comments, even though I keep instructing it not to (both in chat, CLAUDE.md, and in its memories)

A question not addressed here is whether there exists any solution to the EU requirement that doesn't suck.

If the same red/green algorithm described in the article is applied to generated code, I cannot imagine how that does not degrade code quality (probabilistically, not at every point).

I agree with the substance of this article, and disagree with the author's reaction to it.

The part I agree with:

It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the output, and using those bits of entropy for a purpose that does not improve the quality. If you maximize your tradeoff of bits for quality, those extra bits lower the ceiling of what's possible. It's a very simple information theoretic argument, and the only plausible argument against it (that we're using those bits so inefficiently now that the new PRNG is no worse than the old) only holds in the short term.

I also agree that having TOS that forbid removing the watermarking is deeply, deeply problematic. Hell, the whole essay is well-written and persuasive, and gives good reasons why this is a poor approach.

What I disagree with, and the reason for this comment, is the entitlement.

> The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

This attitude is what is patently offensive for me. This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral. It's another form of certain people being above the law, shareholder profit maximization über alles, might makes right, we have to do it or someone else will, "we just help people connect", {code,a gun,roofie} is just a tool.

So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.

It's no more a perversion of writing than the act of using an LLM to write in the first place.

I've been seeing a lot of the same questions about watermarking over the last week, so I made this playground that lets you try out 3 different watermarking schemes (including a version of SynthID-Text) on your own text: https://watermark.keito.me/ to understand it experientially myself. I found it illustrative to try different examples like code, text rewriting, etc. to see how it affected the tokens. Playing around with the detector is interesting too to see how much of the watermark can remain in edited text. I hope this is helpful for others too.

I'm doing a fiction project using AI to write the framework, then fixing it. AI writes terrible fiction. Every single sentence in the project is approved and/or changed by me. And I like em dashes for this project. I've been using em dashes since TeX came out.

The question is: does this make my project "AI" or does it make it mine?

People use professional editors all the time. But they agree to give up any copyright in their edits. If I approve a change made by an editor, it becomes mine. But there is this idea that approving an AI sentence still make it AI.

I think writing is the killer use case for local LLMs.

We've had so many advancements in LLM samplers for improved text generation (off the top of my head: min-P, adaptive-P, XTC, DRY, p-less, Top-H, Top-n-Sigma, and so many more) but hosted LLM APIs only provide three basic knobs: temperature, top-k and top-p which are old as the mountains in LLM years at this point.

One thing that doesn't help the local LLM case is that all the popular VC backed local LLM wrappers also only support the same three ancient knobs because I suppose they're more preoccupied with their next fundraise than with keeping up with the advances in tech.

This post makes no sense. You can either care about writing and write things yourself, or you can not care and farm it out to a model. There is no third option where you care about writing and farm it out to a model.

I guess I understand the complaint, but LLMs are already crap at writing, IMNSHO. And, yes, maybe this will make them marginally more crap, but in my mind we're talking the difference between a 30% grade and a 29%.

Doc Searls, the long time Editor-in-Chief of the Linux Journal, brings an editors viewpoint to this, and mixes in a number of other voices.[1]

Many writers use AI to edit their own words, and this watermarking poisons the well for that use.

I think the backlash from this could be the seed that undoes their attempted 2 Trillion IPO this fall.

[1] https://doc.searls.com/2026/08/17/you-can-hear-the-squeak-of...

As the algorithm tries to keep the stenography in place this will mean if you specifically ask for a different phrasing of one paragraph, other parts of the document will need to change to keep the supposedly impossible to detect AI watermark in place? Won't people also just quickly do analysis on this to figure it out and remove the watermark - it's hard to do this for individual messages but when you can create an infinite number of messaging to train on, I'd think analysis of how the signal works will be quite trivial.

He's complaining that a coin-flipping synthetic text extruder is using an unfair coin. The text generated by an LLM is already adulterated. His objections are so much titling at windmills.

This was nice but without hard evidence. I'll bet if nobody were to have said anything, nobody would nave noticed. That doesn't make it right, but it might be less intelligible and intrusive than the claims here.

Also this:

"> I want any LLM I use to choose the very best, most precise words at every single decision point."

I don't think the author quite appreciates the level of randomness here.

This is more subtle then Claude changing prose.

Again - I suggest that the author would have to run a test on themselves to determine if they can actually find a difference.

Using LLMs for prose that matters is already a perversion of writing.

It's good to be the King.

And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.

> Someone with the secret key can determine which list a word will be on at each token generation point (which is how the watermarking is detected); those without the secret key cannot.

How is this supposed to work in an actual lawsuit? Will Anthropic offer some sort of tool / (paid?) webservice to check for watermarks using that "secret key", and a judge is supposed to just believe whatever that tool's verdict is? And then it takes the EU another 20 years to understand what a silly idea this was?

I have no doubt they can benchmax their way into believing the differences caused by watermarking are imperceptible. But I'm skeptical there won't be a drop in quality that puts them at a disadvantage relative to competitors that don't do this. This adds a new constraint changing predicting the next token to predicting the next token that carries the watermark. It's hard to see how this can avoid making the watermarked output <= the desired output in quality.

I just don't like the idea of an AI company saying that they've added a secret signature that isn't verifiable by any third party to all responses. And we're supposed to both take them at their word, and feed them all the content we want to check so they can continue gobbling up a bunch of fresh works. What is to stop them from saying "Oh yeah, that is ours. We signed it. Trust us."

The obvious conflicts here are wild.

His argument for needing it to write every word at its best is weak. The black box within can change quality at every moment based on many factors they he hasn’t known about, say system prompt, or other harness adjustments. Models get better every 2 months, and they write better, but he is comparing to what?

The real argument should be watermarking itself. I don’t want my shit water marked if I ask you to just rephrase a certain part.

While I like that LLMs won't be able to produce the entire internet anymore, I am worried that the real reason for this move is for Anthropic to claim everything is theirs even though they STOLE humanity's collective knowledge including billions in private property worth of knowledge (or maybe even trillions), and all they do is regurgitate it, but now with a watermark on top as if it was theirs.

Are we now going to live in a world in which people bitch and moan about large private corporations' LLM text generation service and whether it's good or bad etc.? As though they're supposed to be benevolent and serve the public interest? They're not and they don't. Also, write your own damn text.

> "My error was believing Anthropic"

The error is rearranging part of one's life around Anthropic.

They chose the wrong word "watermarking." Perhaps they find the statistics _after_ a certain amount of text is generated at scale, so that it doesn't affect the fidelity or integrity of the LLMs output.

There is evidence this exists already, and it's why "I have to be honest..." and "This is the right lens, ..." keep popping up.

Probably would be cheaper too. Maybe I am missing something?

I was using Claude yesterday and the "advice" it was giving me quickly became confused and irrelevant to the prompt, even though there was not much text in the context window. Speculation: since Claude re-reads the entire chat at every turn, the "minimal" text revisions required by watermarking quickly compound such that even Claude can't follow the discussion.

I just don't agree, and that's ok.

I love your point about social media. IMO, initially they didn't know what they had with social media - it took the right kind of uninhibited sociopath to monetize/weaponize it. That's why the MySpace guy is traveling and taking photos while Zuck is building island lairs.

This is where a functional government would step in. There's no magic in social media technology or LLMs. I'd think of them as a car. You can buy a Nissan Leaf of a Porsche 911. One is an ok basic car. The other is an (over) engineered experience in the form of a car. You cannot legally drive a Porsche to it's potential on the public highways... because we have laws that regulate driving and hold the operator accountable.

We're thinking about Claude Code or Gemini or whatever. The people running these companies are like "I want to exceed the power of John D. Rockefeller or Stalin." If we or the EU are going to regulate AI Labs, you need to grab them by the throat and they should be screaming about it. They seem to be very pleased with themselves. Barring real regulatory teeth, we would be opening the aperature to the Chinese companies to rationalize the valuations.

And it mandates that Anthropic put it in their TOS rather than make it law.

We were talking about text.

Claude text in Claude is already clearly labelled as AI.

There are many reasons to hate this watermarking but affecting the output quality isn’t one of them. The central argument he’s making is wrong. Switching out one RNG for another doesn’t make the results worse.

> The provider must mandate in their terms-of-service that users not remove the watermarking.

So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?

Security theater, and vague enough to be used as a weapon against who the government wishes.

I hope it's left off for non-EU customers.

Claude's writing was already easy to recognize. The fact that Anthropic complied without complaint makes me wonder if they already watermark their outputs and used the opportunity to create goodwill. Presumably they want to avoid training their new model on text generated by the previous model, so they have reasons to be able to recognize AI-generated text.

> I want any LLM I use to choose the very best, most precise words at every single decision point.

Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.

I commented it last time the post about Claude watermarking went viral and I'm going to say the same thing again:

"I've been working in the media and model IP space for quite many years. What this article misses a bit is the threat model for watermarking in general. Watermarking and fingerprinting have inherently weak security guarantees -- they rely a lot on security through obscurity, weak assumed adversaries to deliver. There are clear trade offs between true positives, false positives and maintaining the quality of the media. It's true for audio-visual media, models and their outputs alike. As much as I like to take shots at poor technical choices by corps and govs, this one is unjustified. Sure, inserting glyphs is bad but biased sampling is as good as it gets in 2026."

Since the announcement, there have been many people who don't seem to fully understand what a security guarantee is, what trade offs it might involve, or how popular the type of technological solution is in general (media watermarking is ubiquitous).

And naturally, there are challenges with how you will make sense of the score in your org, e.g. you wrote an email, and it's flagged as LLM-generated because you copied two generated/edited paragraphs.

Yeah, the article might disagree with watermarking as a matter of principle, comparing it to censorship. But the methodological arguments that I have read so far have been thin in these articles.

I often agree with John Gruber, but I think he’s lost the plot with this one.

The thing I don’t understand is why he seems to care so damned much about this subject--enough to write over 4,500 words on it!

John writes for a living. That’s his profession. He’s been writing for over 25 years now. When you’re that good at writing, and you care this much about your writing, you don’t allow an LLM to take over your job. I just can’t imagine that he’s in the market for LLMs and that literary excellence is his number one selection criterion.

So why is he so livid about it? It’s like being angry that wine is going to start coming in smaller bottles even though you don’t drink wine.

Even if he’s angry on behalf of other people, I don’t get it either. In my view, having LLMs write publishable content on your behalf is not a socially-acceptable use case, nor a professionally-acceptable one in most professions, even though people are abusing it for this purpose anyway. And besides, the models aren’t even all that good at it today. If you agree with that, then you certainly should not care if it’s using different phrasing than you otherwise might prefer if the meaning is similar enough.

I can't help but wonder if perhaps his hatred of EU technology regulation (which, admittedly, is mostly pretty dumb and is mainly just making life worse for users) is getting the better of him.

There is no coherent position in which the watermarking is a perversion of writing but AI writing as a whole is not a worse one.

> My error was believing Anthropic that their system wouldn’t adulterate and corrupt the semantics of the text their models generate.

Has that ever been the case? Are they not actively tweaking their models, their fine tuning, the system prompts, the tool definitions and implementations, the guard rails, tool calls, instant responses. There are hundreds of knobs that they can change daily, or between each prompt, or even half way through a generation.

Author read the posts but didn't understand them.

> At each decision point, they’re a little more likely to pick a word from the green list than the red list.

Wrong. There is no global red and green list. It's dependent on context and balances out on average. It won't change the result when one token is predicted overwhelmingly likely.

The fact that Gruber points to a "must-read" article about how the watermarking works, which, in turn, was very obviously entirely LLM-generated, says a lot about his lack of experience reading LLM output.

If Gruber can't tell a fully AI-generated article from a human-written one, perhaps he shouldn't care so much.

NB: I was told yesterday it's apparently a meme to call out Claude-generated output, but here I am, as I believe it's quite relevant to the topic at hand.

Watermarks are garbage because they may embed account id, IP address and deanonimize you. That's why we should be using open-weights LLM whenever possible.

Watermarking will be one more nail in the coffin of proprietary models if the world is so fortunate.

Reminds me of printer tracking dots.

https://en.wikipedia.org/wiki/Printer_tracking_dots

The idea that language models can write anything remotely useful yet is - mostly - a fallacy.

It's a good idea for many human endeavors to be able to identify AI writing. Communication, after all, is our main way of building the social fabric.

However - and crucially - good writing is still beyond the frontier of any model I've seen so far.

Watermarks for the things that truly matter may not be important at all.

Finally, as X commentators have shown, simply removing punctuation or changing a word here or adding an adverb there manually will screw up the whole process enormously.

The best will be the clever folks who retroactively apply the model distribution to fraud or other crimes to try to implicate the companies via watermark.

Gotta feel for their product, policy and legal team.

"The difference between the almost right word and the right word is really a large matter. ’tis the difference between the lightning bug and the lightning." - Mark Twain

"Just flip a coin to pick a random synonym. Who cares?" - AI Labs

Sorry, this article's argument just doesn't hold water. Yes, we want Claude to write "the best text" and having its word choice even very slightly varied could arguably be construed as "not the best." But "best" is highly subjective, always has been. Claude has never, and will never, write what _you_ consider the best version of a piece of text. It has many choices, influenced by all kinds of random, context-dependent weights and environmental settings. Slightly tweaking weights to prefer certain phrasings might even tilt it toward your idea of "best."

Lots of faux outrage, rambling and hyperbole here from Gruber.

“Absurdly and insultingly”? Come on…

Are there some narrow cases (like in outputting the input almost verbatim) where the probability for every best token is always much higher than second best, and in those cases would there be no watermark?

The issue here isn't (just) adulteration, it's that watermarking in general is unworkable.

If all the providers use watermarking systems with different shifting logit weightings, and the keys are secret, you have to check every provider to see if it produced a given text.

Which is clearly ridiculous.

And if all providers collaborate and use the same weightings, or if the weightings are constant and not rotated cryptographically, a generic watermark remover becomes trivial.

That's not even getting into the legal complexities of businesses running open source models without watermarking locally.

I dont see how there would be remotely enough entropy in most model outputs for this to be close to feasible with any kind of accuracy.

Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.

Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.

This is moronic. This is like being mad that the slot machine you think is lucky is occupied.

I think we will see a hidden motivation behind this as ultimately so the output can have an author, the author can be attributed and finally the output will be copyrighted and so the LLM has more value.

Everyone who invests in AI companies wants to see the value of their investment increase.

I'd give it about 3-5 years until an AI company claims copyright over code their LLM produces. This is a crucial step in that path.

Watermarking seems feasible for creative tasks (that admit many valid results).

But if I craft a prompt that doesn't leave space for creativity, how can they include a watermark? e.g. "Rewrite the following text, replacing 'foo' with 'bar'."

I'm curious to see where they draw the line, and whether the watermarking really affects the (perceived) quality of results.

The main reason we don't see much quality degradation in LLM writing output is because they're already poor writers. This is the load bearing reason.

I was bulding a small interpreter and writing an article in ~markdown yesterday with Fable. And while it codes like a pro, it writes like a sixth grader.

Let's see how these watermarking stats hold up if/when llms start writing well.

Some here are arguing that mechanisms used by LLM providers already derail the goal of "the very best, most precise words at every single decision point", therefore the author is misguided.

The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.

I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.

Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.

I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.

If we wish to use “the very best, most precise words at every single decision point” then I will note that “writing” is not the best word for what LLMs are doing. A more precise synonym might be “generating”, in which case this essay becomes “Watermarking is a perversion of text generation” - which is true, albeit somewhat trivial.

So the watermark can be removed by rearranging words and choice of words. This seems trivial to bypass with a local model. If I understand this correctly.

Why does everyone love AI and is impressed by AI quality but when the time comes to admitting that they use the magical AI, no one wants the watermark?

Related:

How Claude's text watermarking works

https://news.ycombinator.com/item?id=49303350

Gruber was a good voice in the industry but this article misses the mark in a lot of ways.

A company the size of Anthropic would not voluntarily jeopardize their massive valuation if they didn’t feel the resulting output would maintain a similar level of quality as before. Is there a similar worry that their system prompt, which is injected at the start of every conversation also influences token generation in an artificial way?

If regulation will ruin Claude as a product, market forces will fill the void. There are also a ton of open weight models to choose from. It’s going to be okay.

I keep seeing an irritating misconception in this space, which is that the alternatives chosen by these algorithms are supposed to mean the same things as what they're displacing. That's not true, and not how LLM generation works. Complaints that two different choices don't mean the same thing miss the entire point.

Great question buried at the bottom:

> Also, what happens if another major global market makes it unlawful for AI to secretly watermark generated text?

People are very upset, especially in the arts, that Anthropic is changing the text to watermark it, but isn't that missing the point a little bit? They're not changing _your_ text whose every word you've carefully chosen for the exact effect, they're changing text that they're generating, i.e. text you've already chosen to give up control over. LLMs can't understand emotional nuance anyway.

The phrasing of the announcement implying that phrasing and diction don't change the meaning of text is insultingly dismissive of the whole field of literature, and I can see why people might take it as an afront, but the actual technology shouldn't have a negative impact as far as I can see. It seems to me that this one is more of a PR problem than something with real-world impact.

any watermarking ai researchers who can explain this?

what if the llm should

- repeat something verbatim (important in a compaction prompt)

- there is just one correct order of tokens for a somewhat long chain (a certain sequence of control signals)

- provide a diff of 2 inputs

without punctuation or whitespace wiggle room?

how does the drifting work?

does it postpone the drifting and drift stronger later?

what if max_tokens is set to a low number?

in what way does this not affect output quality?

Doesn't this require the whole message history including the system prompt for them to check for the watermark?

Claude is already a perversion of writing.

You would think that if the EU is requiring this, then why not just implement this for the EU users.

I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156

Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:

"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"

A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:

"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."

Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:

"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."

Plagiators are having a fit these days

Fix it for you:

Generative AI is a perversion of writing

I assume this means Gruber has fully embraced generating his blog, but is too embarrassed to admit it?

There is no very best token to choose at each decision point. It is context dependent and subjective.

I don’t understand how this works for anything but prose. Is that the point? In any code or structured output, there just isn’t the flexibility, and depending on how the user requests the output be constrained there is even less (“answer only True or False”). So is it just chat responses? If I ask the API to tell me a story about Alice and Bob then it watermarks it, but when I ask it some implausibly constrained thing like write a story about Alice and Bob with each word starting in rotation with the letters alicebob, does it try to do so and hope there are roughly équiprobable tokens regularly?

Text watermarking is another EU rule made without real world input. The Union is stuck on major economic crises (electricity prices for instance) because nobody can agree on anything. However, the bureaucracy forces tech into a privacy nightmare. Brussels cannot bring together its own members but it loves pretending it can govern the internet.

I will be happy to move off Anthropic given the chance. They are burning all of their good will.

Then then don’t use Claude ? What’s wrong with all these people getting vendor locked in.

I wonder if this watermarking system to lead to an increase in tortured phrases, believed to be caused by plagiarism evasion tools that change random words to applicable synonyms.

https://arxiv.org/abs/2107.06751

I wonder how much they tested on non-English languages with fewer synonyms.

What a stupid take. Generating text with an LLM is already a ‘perversion’ of writing. Tweaking the last random-choice step doesn’t meaningfully change that at all.

I almost never post a comment here, but everything about the author's position is offensive and self entitled. I am so enraged that I can't even beging to formulate a response without resorting to very bad language. It's sad, because until now I respected the author. But clearly, and sadly, he has been afflicted with AI brain rot, and is likely in some stage of withdrawal. I wish him a speedy and safe recovery.

There is a difference between precise word choice and concise word choice. You can be precisely accessible the same as you can be concisely terse.

Less unlikely, but I would still suggest that it is somewhat unlikely for any recent LLM to place any significant probably on, e.g. "disappointment" instead of "failure" as the appropriate token (or series thereof) after "kidney" or any similar example.

I don't understand Gruber's points either, I wonder if there is some fundamental technical misunderstanding. Does he think that the logits should be sampled from in a "pure" manner without introducing any other bias? Does he know that there's already a sampling temperature, and that most providers have probably moved on to sampling strategies other than top-k? Does he know that the word choices have already been altered irreversibly during RLHF which is how you get the obvious Claudism like "load bearing" and "seams"?

Perhaps it would be useful to publish examples of samples with/without watermark. I'd suspect that the variability from simply sampling repeated times would dwarf any semantic differences you'd detect with the watermark.

I think Anthropic should have put all the info into one blog post. Splitting it up is really confusing people.

It's very weird looking in from the outside. I mean, sure, the US is the dominant power and everything, so things like Iraq and Iran could be construed as just collateral damage from their imperial maneuvers. But it has just piled on, more and more, and even when they are hit right in the face with the massive bombing of NordStream, still practically nobody tries to draw any sort of line.

Reminds me a bit of that 'Yes Prime Minister' sketch about nuclear deterrant, when the skeptical conversant asks the PM: "So what is the last resort, Picadilli?"

https://www.youtube.com/watch?v=IX_d_vMKswE

As an EU citizen I've found myself in agreement with everything he's written regarding EU policy

In addition to holding the key, wouldn't you additionally need to know exactly which model to check against? So for passive detection to happen, I think each company would need to check every message against every model version? Also would need to spend resources re-invoking the each model version against each message.

Yeah kind of surprised that people didn’t have enough patience for Gruber’s writing.

It’s quite fun and engaging.

I completely agree with you. But that doesn't change that Gruber's fixation on the EU is strange, specifically because he's so ego-tied to Apple that he strikes out like a untrained chihuahua at his list of Apple adversaries. He has done this for many years and it's just weird.

It's like some weird k-pop stan sending death threats to someone that dissed their favourite singer. Just super strange stuff.

From what I’ve read, they won’t be imposing specific choices, but using a different (biased) RNG for those “toss-up” choices. With enough sampling, you could detect if the RNG was biased or not.

Sure – but I’m entitled to make a judgement on what I consider to be “good” and “bad” output from an LLM, where “good” just means “helpful to my process”.

Copyright (or any other such restriction on free use of information) creates power for owners by the simple fact that it turns information into something that can be owned.

Make it a right, then companies/universities will think twice before using said APIs. Instead of this grey area where we will never know.

We don't hand over more power to the owner class by making fewer things ownable.

I assume this oracle will be behind 20 layers of anti-bot protection, CAPTCHAs and hardware attestation challenged. It will be incredibly painful to use. It won't stop the motivated attackers, but will make it too annoying for the average person.

> it says the same thing

Reference needed? I think it remains to be proven whether those detectors can be considered deterministic.

Good lord, just write the thing.

They could (..and probably will..) store that version and then refuse the check if this attack is detected, i.e. the version is too close to a known LLM output.

Alternatively they could also just keep saying "yes" if it's close enough to a version that was close enough.. Although that would enable the attack to allow arbitrary text to be "proven" AI, by slowly morphing close-enough generated material to the desired text. But perhaps this is not a problem they are not concerned with.

To satisfy the letter of the law I expect it's enough to just provide the oracle, without any mitigations.

It's a reasonable theory for sure.

From what I read, two big problems for long-running columnists are getting tire of the work and running out of things to say. I have no knowledge of Gruber, but I can certainly see why people who are expected regularly to have something to say would turn to "AI". It doesn't get tired and is always ready to spew infinite words.

No, watermark detection is not binary, you get a real number. You decide on a threshold when looking for the watermark. This is the problem - by random chance, some human text will be detected as watermarked. You can turn the detection threshold up until it guarantees <0.001 false positive rate at the expense of higher false negatives, but seems inevitable that someone gets wrongly flagged.

Today I learned a new word, "Oulipo". Interesting.

But what about the general idea that they can watermark results to tell where they came from. The next step is tracking down which user got a result. I hate both of these things. Must everything we do be tracked? Next altering wikipedia results so they can tell who looked at the page or something?

I'd like "the best answer" from an llm and don't want to be tracked, but this isn't for me, it is for them. I understand llm results are already using a varying statistical input so they aren't always the same. But I really hate watermarking and likely tracking too.

Yeah. And the detector can’t even tell you “X% chance this is watermarked” because it doesn’t know the input distribution. It can only tell you “Y% chance that an unwatermakred text would score this high” and how many history professors understand Bayes rule well enough to understand the distinction?

Good for you! As an EU citizen I have found myself in almost total disagreement with everything he's written regarding EU policy. Horses for courses.

I think they should release metrics for "how much of this is LLM-written" (the raw test statistic, not including length) along with detection.

You've entirely missed the point.

A proofreader, like an english teacher, can return you your writing simply annotated and marked up, with suggestions and edits in red pen for example.

Then you rewrite your draft into a final using those edits and notes as suggestions.

Making an LLM act like that proof reader would likely not cause your output text to be labeled as generated, even by this system.

After "stoned" you triggered into a different state.

You be a human who's brain shifted into LLM mode (chainneling Markov?).

Or perhaps you're an LLM impersonating humanity.

I often wonder how much LLMs are just mirroring our own brain's patterns.

You're thinking of Markov chains.

To sample from the probability distribution you already need random numbers.

If you get your random numbers from a cryptographic PRNG, then to notice the difference between that and 'real' random numbers even in theory, means you need to break the cryptography. In practice, your gut feeling about how good some text is won't break modern cryptography.

Okay, then it should be acknowledged if a work was AI-edited-written, or AI-ghostwritten.

Part of me wishes we had the same regulation for ghostwriting etc. Nobody should be claiming to have written a book they didn't.

> There’s tons of ways like declaude etc to get around the check

How effective is this on the new fingerprinting mechanisms?

> Is the cheating epidemic so bad?

From what I’ve heard, yeah it’s out of control. And all the existing llm detectors that academics use have a high false positive rate, which catches a bunch of innocent students in the cross fire.

> it does feel like the assignment and ways education happens needs to change?

Why? Was there something fundamentally wrong with how universities teach and assess?

The sector is responding. For example by moving back to more in person exams and reducing the load of any take home exams. Is that good, for some reason?

> There’s something Orwellian about how the phrasing of a passage embeds hidden information

Interesting. I don’t have the same response. LLMs give me an acute sense of existential dread each time their capabilities improve. But fingerprinting doesn’t move me at all. Do some soul searching on why this bothers you. I’d love to hear why, and I bet you aren’t alone.

I don't even ask LLMs to go that far. Tell me if I've made a spelling, punctuation, or grammatical error, period. Don't rewrite a thing, because LLMs suck at that.

> You are one person. The corporation is not. Scale matters

Correct, if you violate it too often to count, you have to pay around less than ~2.5ct per violation.

So the lesson here is: Create a company to do torrenting professionally, and resell its values for higher prices. Then get sued and pay a dime on the dollar you made.

edit: Actually it's 2.5ct per violation.

What if I'm reading it for work? What am I but a tool for the corporatioN?

But if the user's prompt is in the context, you don't know exactly what the RNG chooses between. I don't know what trick they use to get past that, but it seems impossible to get by it in the general case (i.e. if the prompt can be anything) and you'll probably quickly compromise quality if you try.

> Won't we just be able to fine tune OSS models to detect these patterns across providers?

A good fingerprint should make use of cryptographic signatures. Without knowing the keys, the fingerprint should be indistinguishable from noise (or just random token selection)

> You will understand if you run some LLM models with a greedy sampler that does that. The text quality begins to deteriorate rapidly.

Right, I've done this, and this makes sense to me, but I'm not following how that falsifies the top probability word being the best choice in any particular instance.

"Picking only the best word at each decision point results in a worse final result" seems like an imminently reasonable hypothesis.

But the choices are weighted based on those probabilities. This doesn't affect the weightings, only how the final weighted pseudo-random selection is made.

I agree, but thats also learned, and it is done to encourage specific responses for a task, different than applying a mask to the distribution based on a key.

Like their fp rate is a lie? It works well in my limited testing.

what do you mean by "work"? I think it works perfectly as UGC honeypot.

How would that change anything? The proposed watermark is applied while the output tokens are being chosen, taking that text and running it through an LLM again would just repeat the process.

But what if those measures might apply to your favourite multi-trillion-dollar globo-corp and cause them to nominally change their business practices? That's a step too far, I say.

>It doesn't, though. From the outside, without access to the parameters, you can't distinguish the watermarking system from a random number generator.

You can't as a user tell by how much the quality of the output was degraded. True.

>We already use an RNG at inference precisely because it leads to higher-quality output. Changing what function is generating our random numbers changes the sequence, not the randomness from the point of view of a user.

I'm not saying it wasn't random and now it is. I know how these things work. I said that the quality of the system is in the quality of the probabilities. That quality is being degraded.

> Gruber has long been a talented, excellent writer. I hugely doubt he uses AI at all, nor does he plan to.

He notes in various other posts that he uses AI/LLMs and chatbots quite extensively. (I don't recall what for exactly, but not for writing his pieces.)

Not all values of "equally good" are equally good.

Writing has rhythm, or at least it's supposed to, and synonym swapping compromises it.

Never mind metaphors and similes, which are even more tightly constrained.

LLM writing is still a long way from good. Sometimes you get lucky with the odd line, but there's a difference in quality between influencer slop, genre fiction, and literary fiction and/or best-in-class journalism.

LLMs are still somewhere between the first two, and nowhere close to approaching the third.

Indeed.

Evidence is a massive problem here. As well as the extremely high threshold for US defamation; political candidates routinely tell the most absurd lies about each other.

>You just need to know your LLM distribution and the encryption key, then with each new token you exponentially increase the chance of knowing whether it fits your encryption key. Without actually affecting the token choice in a perceivable manner.

I struggle to understand the relevance of that comment.

The blue/green token list biasing process literally does cause different tokens to be occasionally chosen. Not only that, but because a different token was chosen at one point, this changes the probabilities of all subsequent tokens, and the resulting later token stream every time it happens. If you had access to the token stream as it would have been, and the watermarked one by the end of the text they will be very noticeably different.

This says we can only install Apple-approved apps.

You'll be happy to learn about C2PA https://c2pa.org/

Some sort of signatures to verify legitimate unmodified photographs (taken with a relatively secure device, e.g an iPhone) seems like a reasonable idea

But even then, people will be able to point that camera at a manipulated/generated image (either printed or on a screen). Maybe that one could be solved if the photo included some depth information?

a lot of cameras do have an in camera hardware based cryptographic signing (i think it runs off its own chip on them?) but any modifications to the image immediately break that. so for stuff like journalism etc where youd hopefully have a lot of settings dialed in on camera for quick turn around would keep that fingerprint intact.

Fair point, for text it is far harder to prevent signatures being applied to generated text vs images at the moment of capture and most approaches I can come up with to remedy this can either be bypassed (edit histories can be output by models similar to humans) or will be controversial. Taking a page out of the anti-cheat textbook, mainly written for gaming, there are methods which might hold in the medium term. Less a fan of kernel level myself, though it might be worth exploring as there has been massive investment by the games industry into making it somewhat robust, but the approach Valve has taken with VACnet could be an inspiration worth exploring that is less invasive into peoples systems. Keystroke analysis, etc. could be relied upon as a basis for signatures, harder to spoof for current day LLMs over generating edit histories.

I will fully admit that at a point in the future, maybe not too soon, models may be trained to bypass that too, at which point we are back where we started. As a skeptic of the extend that capabilities are emergent in LLMs vs specific to training data, I am somewhat hopeful that unless models are specifically trained for evading such human detection solutions, they'd struggle to do so, but it could still end up as a byproduct of improved, lower latency computer use focused training. Not emergent as the term is used in regard to models because that is still output performance improvements clearly traceable to very specific training data, but incidental as the goal of said training data was not to bypass.

For what it's worth, I find human authorship being verifiable to simply be the more crucial problem over watermarking model output, so if research is to focus on one, I'd rather it the former. Maybe both signing human authored content and watermarking LLM output are both only possible in the near term, I hope not but fear it that might be the case. If so, we as a society will have some major challenges ahead (beyond all the ones we'd have anyways).

Alternatively, we could also just start scanning everyones eyeballs...

It's exactly the same problem as with humans, though.

It's part of why we sign NDAs, and why their duration is measured in years (and that's not even targeting the human retention - just duration after which information ages enough that its disclosure is not likely to negatively impact anyone who cares).

> Humans reading texts historically come with little negative consequences

IDK, we do have laws against opening other people's mail. Those have been on the books for hundreds of years. Seems like someone figured out a while ago that certain unauthorized humans reading certain restricted text wouldn't be good.

> Humans reading texts historically come with little negative consequences, but machines reading and processing texts en masse is more dangerous and should be regulated.

Citation needed. This is sounding tautological.

Claude has those knobs, they are just not exposed to the user. They could make Claude nearly completely deterministic if they wanted to (of course it would be a far inferior product then. But they could).

Again, given the limits of LLMs (stochastic, rapidly changing, everything's a hallucination, widely known prose issues) I am skeptical that you really care that much about optimal prose. I could believe it's one of the things that you care about, but at a pretty low priority level.

Taking you at your word, though, I'd be interested to see what you think of the watermarking technology in a blind A/B test.

Yes, models can reason and plan, which helps them write more coherently. But when they write the final output, it’s still a single generation. It would be like letting a human make notes and write an outline, but not let them use the backspace once they start typing their response.

Presumably you could use the same reasoning trace, run multiple generations, and get different outputs (if the temperature is >0).

But now I’m interested in playing more with Cowork or Claude Code/Codex for prose writing to see if the set of tools there affects outputs at all. I guess you might need a more custom “writing” harness.

Isn't this just chain-of-thought though, doing the same thing multiple times without necessarily defining one path?

As an EU citizen and user of Apple products, I feel the same

> Where is the problem with using LLM generated text?

In the fact that you didn't write it.

> You could use your own hypothetical house elf to do it for you, or pay someone to do it.

Yes, and those would be similarly problematic (and more expensive).

Nobody says it's a problem. We'd just like to know.

Factory farming also makes meat cheaper than organic practices. I'd just like to know which one I'm getting.

If there was a ghostwriting detector, I would use that too.

Sure, but mightn’t there be several plausible long horizon plans?

Here’s an example: I had asked Claude for some music recommendations in a certain style. Part of its output was:

—

*Long journey tracks*

Clinic — “The Return of Evil Bill”

Guided by Voices — not really, wrong band

Silver Apples — “Oscillations”. Proto-everything, deeply repetitive, hypnotic.

—

So at some point there, the next token produced was “Guided” or “Guide” or whatever, and then because it can’t go back, it had to correct itself after the fact.

Reasoning/CoT have helped a lot, but I feel like small versions of this still happen all the time.

Human writing is like 90% editing.

It can but it is limited because it's only got a single pass through the network to fit the entire "longer horizon plan".

Very intelligent people need intelligent-others to bounce ideas off of, and the LLM can be that.

It says they observed no difference in people clicking thumbs up or down. There are loads of other behavior that they didn't observe; like, say, switching to a different LLM.

> People get angry about DRM schemes

Yes but thats a thing that degrades something in a catastrophic way, as in I can use the thing one day, and not the next.

A different randomisation system on something that is a text generator which is designed to be unperceptable sounds like the people who are annoyed at FLAC vs MP3[1]

Done right you won't know the difference, done badly and you will.

[1] ex audio engineer, try me.

> People get angry about DRM schemes in video games, even if the slowdown these cause is practically imperceptible.

Any potential "slowdown" doesn't even come close to making the list of top reasons people get upset about DRM.

I get what you're saying, but I think it's ridiculous for people to think of LLM services generating text as either their own domain or something that they own.

To the extent that it's anybody's, it's either Anthropic's (they run the service) or everybody's (in that we created the content it's remixing). Legally LLM prose isn't copyrightable for good reason.

Even if honor were real, LLMs do not have honor, and people using LLMs to write without disclosing that fact (or indeed at all, to some purists) do not have honor either.

> because a stranger manipulates what they regard as their own domain, without consent by or benefit to the owner.

It's LLM output! It's not your domain, it's the LLM owner's!

>No one uses a pure random function over the whole probability distribution described by the LLM's output.

So what? By definition with this system the LLM will chose tokens it otherwise would not, purely for watermarking reasons. Yes this token may have had a decent likelihood of being chosen anyway, but it wouldn't have been chosen and now it was for reasons nothing to do with output quality.

I'm not sure what your last paragraph is trying to say. The blue/green list system changes what output the LLM would otherwise produce. You can't train it to produce watermarked output with this system. If you tried to, there would be no delta between trained output and watermarked output for you to be able to detect.

> affects their own word choice.

exactly. in the same way that printed books affected word choice, so did the radio.

> LLMs collapse and start outputting garbage after a number of tokens if you do not sample and just pick the "best token" each time.

LLMs are likely to get stuck even with sampling if asked to generate tokens on their own long enough, though sampling does tend to stretch out the time before that happens (as do other techniques that don't involve sampling, like applying repetition penalties directly to token logits). But LLMs generally aren't left to infinitely extend their own output, and the length response typically needed in the use case is much shorter than the would result in collapse given the kinds of inputs expected in that use case, the existence of the theoretical eventuality may not really matter.

Citation needed

But not insult, and even for defamation, you will be prosecuted and have to prove the statement is true, even if the politician and everyone else knows it's true.

There's no difference between those two things. The distribution that matters is the distribution of tokens that are picked not the distribution of tokens the LLM model passed to the selector.

This comment disagrees with you: https://news.ycombinator.com/item?id=49324387

You’re not getting it. The probabilities do not change at all. The only change is given some probabilities there is a deterministic method for determining which symbol was sampled from that distribution. The distribution or sampling process itself is not modified.

Part of the legislation requires them to make a public AI text detector (ala GPTZero I assume).

Wouldn’t having that be enough to eventually reverse engineer the key?

Well, it cant be that he is super worried on behalf of people who publish AI slop. That’s not a credible motivation. In fact, he complained a lot about the new ChatGPT app so I can’t believe your claim that he is not using AI.

Seems like he really likes to use LLMs and is worried that quality will be degraded. But he will never demonstrate such degradation scientifically, we don’t have anecdotes even.

> He is not using AI

That's ... even worse? So we're all here in the comments trying to figure out what the author means, and what their overall point is, while clearly they don't even use the damn thing? Oof... What a waste of time for everyone involved.

1. As watermarked text is added to the training data, watermark-related tokens will be associated more with AI outputs and thus lower quality outputs which will hasten model collapse. Especially because every provider has its own secret key and they are all training on eachother's outputs anyway.

I guess they can at scale filter the watermarked documents (by necessarily allowing eachother to at scale checked for watermarks, but banning the labs not part of the watermarking-cabal). Makes me wonder how useful the human quality filter is on AI output - if a human judges a given output as genuinely good and posts it somewhere for the scrapers to find and take into the training sets, will these types of outputs also be filtered out?

2. (raw, pre-watermarked) Output token probability situations where 1 output token has the majority of the probability mass associated with it, but it is not in the watermarked set, will force the model with much higher probability to walk a non-optimal latent space. E.g., if the next OBVIOUS token for a given sentence would be a point, but the model is in this way not allowed to output it, it might put a comma and start off on a whole different tangent just to make the initial non-optimal comma grammatically make sense.

These are services, so "how would <x> do this today?" is irrelevant.

The real question is: "What can manipulation of pseudo-random number generation do?"

We know that in the cryptographic world, attacking "randomness" is a key offensive capability. It will be here as well -- if Anthropic can watermark text as generated it's LLM, will it be able to watermark outputs as generated by "Spooky23/FooCorp"? Can I pay Anthropic to steer inquiries in a way that benefits my company or governemnt?

Pseudo-random to the end user appears random. Most treat it like a random chance. It is not.

Same reason that you don't just replace your rand() implementation with "return 4; // chosen by fair dice roll". If you need randomness for whatever reason, biasing the generator is compromising quality.

( https://xkcd.com/221/ )

In case of LLMs, you can look at it from high and low level.

At low level - if you could do with less randomness, you can always lower temperature. You usually keep it (or for SOTA providers' chat UI, they keep it) at a level where it's about right level - high enough to allow for more creative leaps and interpretations, low enough that it doesn't go off into crazy land after the third paragraph.

At high level - creativity is driven by randomness. If you had an author (fiction or nonfiction) you like for their both broad and deep range of insightful thoughts, would you be happy if they suddenly developed an acute porn obsession and uncontrollably added lewd subtext to every other sentence? Still creative, still deep, but now with that one strong attractor that biases their every thought in a single direction? Would you trust/enjoy their output as much as you did before?

That, slightly exaggerating to make it more obvious, is what "loss of quality" means here.

Interesting, are you being literal about requesting 20 ways of saying the same thing? It seems pretty excessive and I'm actually impressed that asking an LLM to rewrite a statement 20 times yields output that isn't excessively redundant. Does the LLM do a pretty good job reading your mind, or do you still find yourself manually piecing together pieces from the 20 suggestions into a satisfactory sentence?

As a writer, Claude’s metaphors are trite and obvious 90% of the time. It also has a terrible penchant for an immediately recognizable emphatic voice that makes even the best outputs super cringe. But people do not notice and do not care. In our bubble and John Gruber’s bubble we really overestimate how much people must care. Truth is, we’re in this predicament because statistically speaking the people who hive a fuck are a rounding error.

Yes, but i cannot create a robot that can get married, people do not have the right to give personhood to tools

That’s my point of the post … I had the feeling that the English text editing skills of Claude went significantly down in August.

I was frustrated at first not knowing what they are doing.

After I read this post, (seeing they introduced it in August) I think it has to do with the watermarking.

Try it on a paragraph … the connections between sentences feel clunky now.

I will play with it more and see if that’s really the case (the watermarking making the text edits worse).

To that point, given a corpus of writing from person A and another from person B, I have no doubt it’s easy to train a classifier to determine who wrote what. In fact I believe law enforcement agencies already have these classifiers.

The only difference here is that Anthropic is actively trying to make the watermark undetectable.

A legislative solution to Flock seems a lot more feasible than one for literacy

I have, and I still do this occasionally. I don’t believe the plant on my desk (I don’t have a rubber duck) is intelligent. I don’t need it to be intelligent, either.

I also do it with a LLM every now and then and, while it’s feedback is more useful than a toy’s, it does not need to be intelligent either.

It’s fully possible I didn’t explain it very well in the first place, but he is making a wider point.

The point I made (quite briefly) is that watermarking is only feasible because for good writing it is necessary to use T>0, or the writing will never explore a more creative choice, and that at T=0 you don’t even need a watermark to spot LLM-generated text.

The point he is making is consistent with this, isn’t it? Either you allow temperature to drive creativity, consistently in a way that can be influenced and analysed, or you adulterate that process for the purposes of meeting a corporate/legal directive, in a way that is proprietary and obscure. These are ethically distinct approaches, and since he disagrees with the EU objective he comes down on one side I guess.

Me, I don’t care about the hypothetical enough.

Not least because I think Claude writes depressingly badly and I doubt any steganographic change will enrage me less.

> The people who cheat miss out on learning.

They aren't there to learn. They are there to jump through hoops to get a degree that will let them get a job so they can make money and prosper. The learning is entirely secondary.

The cheating will stop when there is no longer any economic incentive to be there in the first place. People with "pure" motives will refuse to cheat on their own, precisely because they want to learn and cheating prevents them from learning.

Watermarking AI output is a treatment for symptoms. The cause is the higher education meme. Somehow, getting a degree just became the default. Can't get a good job without a degree. That meme needs to die, and higher education will never recover its integrity until it does.

That's an intriguing twist, isn't it? It could lead to a tug-of-war.

Working backwards: if it is possible to confirm 100% confidence that a chunk of text is LLM output, then it is "PD until proven otherwise". How can a human reliably assert human authorship of their source text? When all watermark tests fail? Is that proof of humanity now?

If a human proves human authorship, and LLM watermarking tests positive, then is that going to be considered a "derivative work" or not? What if there is an applicable license for the source work, such as "CC-BY-ND" that prohibits derivative works?

This has not been court-tested, and I expect that it will need testing at that level before we can have any assurances.

Europe trusts its institutions way too much. It may well be that only the police can access the checker.

I can’t speak for them, but I do this occasionally. Every now and then among the propositions there are one or two that did not come to my mind and that are actually quite good. I still write the whole thing myself, I just ask for advice occasionally.

How did you understand this as not writing the piece myself?

Update: https://x.com/mrcslws/status/2089850106292162982

Pasted below:

Finally convinced myself that non-distorting watermarking is real, a la Anthropic / Google SynthID. It's not just spin / marketing. This really is a "Monty Hall" like problem. (h/t @random_walker for that analogy.)

Sharing here in case it helps anyone else.

On one hand, watermarking is "obviously" damaging to the output. The LLM does all this work to compute token probabilities... then you essentially perturb the probabilities? Of course that's bad! Every single token is using perturbed probabilities!

On the other hand, you obviously can freely inject a layer of randomness into sampling. Take any categorical distribution. For each sample, perturb the probabilities, then draw the sample. Done correctly, over many samples the results will match the original distribution.

It still kinda feels like magic, but it's not surprising that you can take that core trick and shape it into a non-distortionary watermarking scheme.

(i.e. it gives you sequences that were just as likely to be generated by the original model as any other sequence)

No, if you switched to Japanese the LLM wouldn't ignore it, it would "assume" there's a reason for that. The same if the previous iteration of the LLM switched to Japanese. Else you're expecting an LLM to ignore its own previous outputs and restart "thinking" from scratch with every token?

You've somewhat mischaracterised what I have said.

I suggested he as a tech writer has reason to be invested in understanding how they work. It's not really a sustainable position to not understand, is it?

Which ones?

This is very poignant. The entities that controlled the content broadcast over radio and TV or what was published in books and newspapers had an enormous impact on both culture and language. Radio and print didn’t just transmit the voice of a single person, the message was shaped by whole teams of editors, bureaucrats, politicians, censors and the necessities and constraints of the technology. Any message that a person tried to transmit through these channels was transformed more than it would be if you ran it through an LLM or more likely not transmitted at all.

There are two different things being talked about here, one of which is "what effect does widespread LLM use have on a culture", and one of which is "what is the effect on a specific text of running it through an LLM".

A radio device at BBC did all this and not humans?

> again, your argument is against LLMs and globalisation of culture, not finger printing.

ABSOLUTELY NOT. Do not ever put words in my mouth. My argument is that radio is a medium through which humans communicate. An LLM is not.

For a given short-context (hash function seeding) you do have detectable manipulation of the completion (how they can read the marking). But, because (assuming enough entropy in context) the hashing function is decoupled from the log-probs, the perturbations are independent from the underlying distribution, so you're still sampling from the same distribution quotient some noise.

The only way you'd notice this is they weren't independent, and the most plausible way that happens is if you're re-completing pre-fills (resampling the same hash function against the same log-probs).

This is far too cynical. I use what I learned in my computer science program constantly.

I agree that higher education shouldn’t be as required to get a good job. But reality isn't black or white. Reducing the entire sector to a corrupt degree mill throws the baby out with the bath water.

I don't think watermarking breaks this relationship. Watermarked text is still being sampled from the model's output distribution, and adjusting the temperature still has the same affect on that output distribution.

I think a good intuition here is that watermarking is sort of like picking a specific PRNG seed. It's not changing or interfering with the temperature - we're still sampling from the model's probability distribution. But we're making it so the analog of the PRNG seed is coupled to the previous context.

Your response seems to be missing the point completely. Gruber thinks "best" writing is produced by choosing the "best" word (highest scoring token) at each step. This is very clear from his writing.

Is that realistically how that is proven in court today?

They do exactly that "which is the better answer?" test -- I've seen it pop up a few times.

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits.

But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.

> The system is fragile, and small edits (or rewrites by a local model) can fool the detection.

Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens.

And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.

In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

Does that also mean owner must re-run very model ever released by them to detect anything?

What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?

>There is no “oops, I wrote ‘th’ but I should have written ‘tw’ so I guess I’m stuck writing three instead of tween”.

You're mixing up two claims here, and only one of these is kind of true. Yes LLMs do internally plan ahead in a way that is emergent rather than strictly part of their architecture, so that part of your claim is true. The way you word it by saying they are "coalescing the probabilities of a range of tokens at a time" is poetic sounding jibberish though. What's actually happening is one distribution output for the next token computed from a hidden state that implicitly encodes where the text headed.

Your claim that if an LLM does happen to pick a token "th" instead of "tw", then the LLM isn't stuck with that decision is entirely false for autoregressive LLMs which is what all of the frontier models are. Whatever an LLM picks as its output token is final, it has no ability to undo that token selection and it must continue on the basis of that choice. It can't go back on that decision and revise the output.

If you're interested in this, Anthropic has a summary of a very technical paper on this topic that mostly deals with this issue with respect to poetry:

https://www.anthropic.com/research/natural-language-autoenco...

  So we train a second copy of Claude to work backwards—reconstruct the original activation from the text explanation. We consider an explanation to be good if it leads to an accurate reconstruction. We then train Claude to produce better explanations according to this definition using standard AI training techniques.

No, they really do one at a time. You're incorrect on that.

Mathematically, a long chain of conditional probabilities is equivalent to a single probability over the whole range. But computationally, for that to work out, the computation for the first probability needs to somehow consider all the downstream probabilities depending on it, which obviously isn't how autoregressive language models work. They can pack in as much downstream computation as their neural architecture allows for, which is quite a lot.

Suppose in some context you have three equally plausible conpletions after "Be": "tween a rock and a hard place", "twixed he stood there" and "lieve he can fly". To model this probability distribution of the whole sentence, the next token "tw" needs to appear at 2/3 probability and "lie" at 1/3. After "tw" would be a 1/2 chance of "ix" and a 1/2 chance of "een"; after "lie" would be a 100% chance of "ve " and in any case the rest of the sentence after that would be 100%.

The model needs to somehow "think ahead" to know those are the possible completions. For example if "lieve he can swim like a dolphin" was another equally plausible completion, that first token would need to be 50/50 instead of 67/33. So the computation of the first token somehow needs to encode the fact that the guy thinks he can fly but not swim, even though it doesn't become relevant in the output until several tokens later.

In practice this probably happens to some degree but definitely doesn't happen perfectly. To perfectly model the first token's probability distribution, it would have to include knowledge of the entire distribution of all possible outputs, which is just not happening. So it approximates. Surprisingly, the approximation is good enough to produce language.

You can see this breaking down in the seahorse emoji incident from last year. When you ask the model if there's a seahorse emoji, it first completes "Yes," as if a few tokens later it's about to produce a seahorse emoji. But when it actually gets to the token that would produce a seahorse emoji, it can't because there isn't one. But it's already outputted "Yes, the seahorse emoji is" and can't just go back and change that to "No, there's no seahorse emoji." Some models would try a few times and then say there isn't one or a system error seems to be making them unable to produce one, other models (including then-current ChatGPT) would loop forever with ensuing hilarity.

But on some level there is uncertainty, right? Even if it’s not token-specific but at the word- or phrase-level? Otherwise what does the temperature setting do? Or has architecture changed significantly in the background?

everyone in this thread is saying something kind of imprecise and reductive and varies between models and even modifications among the models

There are diffusion-based models and transformer-based models (and many other "architectures"), so your comment does not make sense.

Essentially yes.

Flac takes a raw .wav and effectively zips it up to shave off a certain amount of space. (there are nuances, I think the compression scheme is designed for streaming.)

mp3 is perceptual, so throws away the stuff that humans can't hear. This yields a much smaller file.

However its all a sliding scale like PNG vs jpeg.

a .jpg with a quality setting of 85 will be almost identical to a .png in visual quality. However if you then edit that jpeg, the image degrades and you start to see artifacts. (hence why memes look like shite as they get older)

Its the same with mp3s if you compress the hell out of them, say 64kbit or lower adaptive, then you'll start to hear the tell tail "schlop" noise of mp3-like compression. You might notice it most with cymbals in drum kits. cymbals are wideband noise. as in there are loads of constituent frequencies so if you remove some of the "hidden" frequencies you tend to notice, so they sound more metallic, ironically.

But, all of this is solvable, 256+kbit is more than enough, bonus points for higher sampling frequencies. (however you need a decoder that can actually do that sample rate...)

> All those methods are applied with the specific goal of improving output quality and are applied to the extent that they do this

Yes, that's the goal that was used, but they are quite simplistic and crude methods, not some specifically designed function, with carefully fine tuned parameters or something. So, if a basic function like top_k can improve model utility, it's not impossible to imagine that watermarking could also happen to do so, or at least not have a significant negative effect. So whether the effect is deleterious or not is an empirical question, not something we can assume ahead of time.

> You're basically saying that if we train LLMs on watermarked text they'll be really good at producing text that looks watermarked

No, you're misunderstanding how the training works. If we train the model's output so that it minimizes the error function after the watermark is applied on it, the model will learn how to produce the best output it can given the watermark. It will produce better text that happens to be watermarked, not "more watermarked text". Same as if you train the model on minimizing `top_k_error(input) = |top_k_sampling(model_output(input)) - desired_output(input)|`, the model will learn to produce better output under top_k sampling, not learn to produce output that's "looks more top_k".

So you can just say some text is copyright to you and it is?

Here is some text that is copyright to me. As you infringed my copyright, please pay my $5000 license fee for every user who has read it:

> Copyright law was updated in a very helpful way in the last twenty years sometime so that as soon as you post something to the internet you have copyright. If you need a citation don't hesitate to ask someone else.

> I don't think it's weird to ask that people commenting on x doing y with tech z at least try tech z, no?

On what specific basis do you assume he hasn't tried it? He's definitely blogged about the desktop apps, after all.

Or are you arguing that a writer doesn't have a meaningful or valid opinion on LLM-generated writing until they have tried to pass some off as their own?

This just seems weird to me. I mean, I have an opinion on this and I am personally never going to use an LLM to do published writing. On an intellectual level I can still see that there is nuance in it for others (for once I agree with him about an EU regulation).

> Jobs requiring higher education pay roughly 2x more than those requiring high school education

This isn’t really evidence either way. Why do companies pay twice as much for people with higher education? We can’t tell from that statistic. Maybe it’s what you learn in class that makes you twice as valuable to potential employers.

> how did they get the job? Networking?

Probably. After all, that’s how most people in our industry find work. Degree or not.

>If it's functionally random PRNG, then how does it differ from any other random sampling?

For practical purposes it isn't.

It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.

They were thorough. It‘s in their 6/2000 magazine.

We are clearly talking about language models

Who is Gorg Prc?

And if you lowercase everything, you're e e cummings.

Companies are not in fact legally barred from directly issuing IQ tests, and several major companies do. This is for the most part an Internet urban legend (it has a tiny and irrelevant kernel of truth, in Griggs).

> It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it

It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.

Great points all around, lots to think about. It's totally possible that I am way off in my assessment of the future.

I think your comment about torrenting touches on an interestingly relevant case study, specifically media piracy. The seed of that technology was sown when the internet was still just clusters of machines passing files around and then exploded with the PC, Napster, and TPB. Napster tried to be a legitimate commercial enterprise with a business model of undermining the ability of copyright holders to rent-seek on the consumption of the material they 'owned'. At the core it was a novel technology (P2P) that revealed an economic arrangement to be out of date (if a distributor no longer has to manufacture a copy of the media for each individual consumer, their business model boils down to rent-seeking). Western legal systems were quick to rule on the matter (in favor of copyright holders), and I have no doubt that they were looking ahead to a future wherein media creation was totally disincentivized by said novel technology.

Now a novel technology (the cloud inference-backed LLM) is challenging another economic arrangement. This time around, the arrangement being challenged is the higher education->professional job pipeline. All advertising and media messaging aside, it really does seem like the frontier labs are only economically viable if they get massive enterprise deals across a broad spectrum of industry. There is fundamentally one chunk of capital organizations are going to spend either supporting their talent pipeline or padding it (to put it gently) with enterprise LLM deals. If the latter path is taken too far, consumer spending plummets (due to lack of middle-class incomes), assets backed by consumer debt/spending fail longterm, and we will have to deal with a deluge of socio-political issues stemming from the absence of real social mobility (we are in the early stages of this now, incidentally).

All this to say, I think parallel situations from the past can guide our thinking re: AI regulation and the forces shaping it. Reasoning based on regular political/economic incentive structures (e.g. economic liberalism not wanting to reduce economic activity) will fly out the window at lightspeed once fear becomes a factor. I personally think its great that LLMs empower individuals such as your friend to increase the control they have over real issues in their life; that is what technology should be doing for us. Cloud inference-backed LLMs are doing the opposite: drastically reducing the power that the everyman has over his socio-economic future by throwing high-paying career paths for a whirl and incentivizing powerful organizations to destabilize the labor market. Only time will tell how this plays out.