Not mentioned: there would be a single gatekeeper for the internet, Cloudflare.

The "level playing field" rhetoric reminds me so much of Apple talking about the App Store. This new internet business model is just the App Store, substituting websites for apps and Cloudflare for Apple. The system only works with some middleman between the AI companies and the content creators.

"Content creators" are part of the problem. Generating endless "content", which isn't very useful or valuable, and creates too much "content supply", which devalues it. This then creates a giant "soup" of content that viewers drown in, trying to find some content that isn't as identically useless as all the rest. But the big incumbents love it, because they use this content soup to collect money from - you guessed it - ad companies. (Those same companies they don't want to make ad-blockers against...)

So now that search is dead, they need to find a new way to drink from the ad-dollar faucet. At first it'll be "pay content creators from AI subscription money", but then AI will be offered "free with ads", and then later "paid with ads". And nobody's going to stop it, because everyone is "happy": the viewer gets their free crap (with ads), the content creators get a few pennies, the big companies rake in billions, and ad companies continue their "industrial welfare" by pouring their excess profits into this whole system as ad-dollars. The great commoditization of eyeballs continues.

This is also being attempted by RSL with their “Crawler Authentication Protocol” (https://rslstandard.org/guide/web-crawlers) for demanding proof of licensing from scrapers and RSL Collective (https://rslcollective.org) for providing the licensing itself. The missing piece there is the ability to detect scrapers with high accuracy without punishing regular browsing humans.

Realistically, what was the benefit of the ad-driven model? From my point of view, most of the highly-valuable information came from the various forums, wikis and personal sites, on all of which people would publish the information for free. Ads were largely used to cover hosting costs of the large forums and wikis, but the content creators saw not a single dime.

Over the last 15 years, the search results have shifted from this, to instead become 100% news articles and SEO spam, all with a lot of fluff and very little substance, tons of ads, pop-ups, autoplay videos and subscription walls. All of this is well funded thanks to the business model, but for what? I can't imagine anyone sitting through all that crap to get to what they were looking for. There's a reason people would tell others to add "reddit" to the search query only a few years back, and even that's becoming less worthwhile.

Is this really what we want to preserve? Big publications and their interests?

> Swiss Cheese — there's a lot of cheese, but there's also a lot of holes.

Such hubris! Human knowledge is not like Swiss (mostly known with a few gaps). Human knowledge is like a high-dimensional fractal broccoli: a stem of hard-won knowledge surrounded by a vast frontier of unknown. The hardest part of pushing on the frontier is deciding where to push. Deciding what to research requires taste.

> You could imagine an AI company suggesting back to creators that they need more created about topics they may not have enough content about. Say, for example, the carrying capacity of unladened swallows because they know their subscribers of a certain age and proclivity are always looking for answers about that topic. The very pruning algorithms the AI companies use today form a roadmap for what content is worth enough to not be pruned but paid for.

Honestly, this sounds like a Mozilla press release. The spin is absurd

Before ISPs existed, someone else paid. Universities, governments, selected corporations, etc.

After ISPs (post-UUNet), internet subscribers have always had to pay

Who finances the data collection, i.e., the transfer of data from/about subscribers to the so-called "tech" company for commercial exploitation. Hint: It is not the so-called "tech" companies

When I first accessed the internet, and later the www, the term "creators" was never used to describe people who used it

Methinks "creators" is SillyCon Valley speak for "unpaid independent contractors". Why not use the term "producers"

The gigantic websites comprised of other peoples' work, today's so-called "platforms", are not "creators", they produce nothing. Despite billions of dollars at their disposal, and vast amounts of published material, their production costs are small. This is because production costs are outsourced to unpaid independent contractors. The material the "platforms" publish does not belong to them, they are only third party intermediaries (middlemen)

In the original internet, people publishing on the internet via websites produced the website content themselves

Today, "platforms" are not creating anything except a baited trap for people using the internet, whether they are "creators" (producers) or consumers, where internet users can be spied on for advertising purposes. And the targets of this surveillance and subsequent targeted advertising will even pay for sending the surveillance data collected to the interlopers

Today's "creators" may be paid (if they are lucky, like winning a lottery) or not (the other 99%, their free contributions serve as bait). But "make no mistake", it is not the third party intermediary "platforms" that pay "creators", it is advertisers, but only after the "platforms" (middlemen) take a cut

SillyCon Valley calls this an "ecosystem". But today's internet, cf. the original one, IMHO looks more like a toxic waste dump

I don't fundamentally think that Cloudflare's view here is _wrong_ - by and large, when I google what internal temperature to cook pork to so I don't die, I'm not looking for an opinion or someone's life story - but I think I'm much more interested in how we create the weird niches that create the "knowledge" for the blender. Cloudflare alludes to it with the talk about Reddit and whatnot, but I'd love to know what the plan is to actually create and nurture those communities where people can really get weird into whatever topic they're interested in. Right now we're all sort of just ghettoized into various Facebook communities or whatever, but recreating an actual vibrant communal internet where people can find their weirdo subcultures and actually negotiate on some kind of reasonable footing with the LLM scrapers (who were the social network people, who were the search engine people, who have always been the ad people) would be a genuine improvement to the internet.

* read "hellish swamp diving affair"

This is the old "blockchains will create new niches for creators" / "rent-seeking is Good Actually" argument, barely reheated. Total imagination bankruptcy from C-suite.

Because the providers will act as a single tunnel that all content passes through before reaching the end user, the tolls they collect will be large. So, I don't doubt that there will still be opportunities for content creators to earn money as answer engines siphon off more and more of the web's traffic, but expect those opportunities to be broadly low-paying, falling decidedly in the "side hustle" category.

AI providers will want to incentivize content creation. There will still be a glut of ready providers, and little reason for providers to make anything but small, nominal payments.

The new model will be something like:

1. A content creator creates a web site and uses Cloudflare.

2. AI companies pay Cloudflare to allow them to scrape content.

3. Cloudflare gives a cut to the content creator.

4. Users pay AI companies and get their questions answered.

A few observations/predictions:

* If this works, there will be competitors to Cloudflare (AWS, Microsoft, etc.) who will offer better terms to content creators. Content creators can then (easily) switch to whichever reverse-proxy has the best terms.

* Media companies will transform into Cloudflare competitors, aggregating content and monetizing by selling to AI. Their pitch will be that the content will be more curated than Cloudflare. Their brands might survive if the AIs pass the source of the content all the way to the user. For example, the AI says something like, "According to a BBC contributor....". Otherwise, media brands will no longer be known to consumers (only AI companies will care).

* If this works, AI companies will try to cut out the middle-man by building their own ecosystem of content creators.

* As more and more people get their answers directly from AI, it will be easier to sell content directly to AI companies. I.e., instead of publishing something on the open web and relying on Robots.txt to protect your content, you will sell content straight to the AI company. NOTE: If this happens, then the only way this will scale is if the AI itself decides which content it wants to buy for the next training run.

* At the limit, the web and everything about it basically disappears. Everyone gets their content directly from an AI and never visits a web site directly. Therefore, web sites disappear and all that's left is the HTTP protocol, which is used by AI clients to talk to the AI cloud.

    ads have been the only micropayment
    system that has worked

I wonder how the web would look like if one could click "pay 1 cent to continue".

Maybe content would become better? Maybe it would make one think "Hmm... one moment, is this something I want to read or am I just doomscrolling?".

That said, the mental model that this article uses only recognizes that new content can fill in "holes" (interpolation). It also can expand the boundaries in new directions (extrapolation). That is a different and harder problem. If you distribute money to people "based on what most fills in the holes in the cheese", you really aren't expanding the boundaries of human knowledge as much as you are strengthening existing knowledge. They need to take boundary expansion into account here.

I also recognize that we know where the holes or boundaries are in many fields. They are "known unknowns". But this proposal does not take into account "unknown unknowns" --- things that we do not even realize that we don't know yet. It's going to be harder to incentive research into unknown unknowns when we don't even know what they are yet.

Many of us who use an ad blocker don't actually have a huge problem with ads. It's the tracking we object to.

I don't want Cloudflare to build an ad blocker, I want them to make a tracker blocker. Make surveillance so difficult and expensive and unreliable that companies stop doing it.

It sucks big time to live in an internet with cloudflare

(Note that everything CloudFlare talked about in this blog post also applies to adblock users, not just AI agents.)

The golden age of the Internet is not where people do it for money, or for views. That way lies clickbait and content farms. The golden age of the Internet is one where people share information because they want to.

Source: https://1gn15.com/cloudflare

The bigger problem I see coming down the line, given the current business model for the internet, is what happens when investors need a return. When the LLM only provides one “correct” answer, the only way to make money from advertisers is to influence that answer. At least just now we are presented with information which we evaluate ourselves, though even this is subject to manipulation.

https://stratechery.com/2025/an-interview-with-cloudflare-fo...

Ok the the toxic nature of the internet and social media in general and what has become of our digital age, totally agree that it's rage and click bait. I wrote something to that effect here https://github.com/micro/mu/issues/27. But I personally don't think we're going to directly interact with agents the whole time. They will exist, they will be somewhere in the middle layers, there might even be a chat interface that replaces search queries with answers but I think the whole web as a whole and social media needs a rethink. Ads as a business model has to die, even though clearly it won't and we need to shift our attention elsewhere.

Examples:

Pingora claims to be battle-tested, but I have a hard time believing that it's to the same level of quality as whatever Cloudflare runs internally. https://github.com/cloudflare/pingora/issues/601

https://blog.cloudflare.com/introducing-oxy/ was not open source.

Small parts of Oxy were open sourced as "foundations" but the repo gives off the impression of a checkbox for someone rather than a serious commitment to building CF's own services on top of it — not "open, collaborative, standardized, and shared across many organizations".

This is an attempt to rewrite history. Back in the day, we stood up servers at our own expense and filled them with content for free, for nothing other than the fun of it. In fact, the "vibrancy" of the internet appears to be inversely correlated to the number of people using it to generate a profit.

On the other hand, Youtube is pretty good. You can make some money off of youtube viewings, but what it's really great for is building an audience. Musicians know that Youtube is a good tool for building a successful music career.

As for devaluing content with oversaturation, yeah, agree with this. Ironically, media is both the bread and butter of the internet, but it has also become devalued with the shear amount of it. Still, this isn't a problem of the media companies. Content creators can either chose to engage and or not, and hope they breakthrough.

And you kind of said it, "The great commoditization of eyeballs continues." Money makes the world go round, and nothing is free. If we want a vibrant internet, companies have got to find a way to make money (although, monopolies aren't a good thing, but paying for good content and services seems like a good idea).

In the book he explains how these micropayments would be sent out to all the websites you visit, etc. And you can have a built-in mechanism to combat the "star system" that tech automatically enables, where a few people get the most money.

Lanier's proposal was that ISPs take charge of this issue, but perhaps Cloudflare can make it work. The less cynical interpretation would be that later they federate the system or even turn into a non-profit.

I have fully read Cloudflare's proposal, as I have RSL, and L402.

In its current state, RSL is not a full solution, but can be. The spec does not cover payments. Its basically just a new token type, which crawlers would register for to crawl your content, and predefined responses for more detailed states. But nothing around payments, yet.

Cloudflare's solution is basically the same as L402. This is very sad for me, because their attempt is basically one of locking down the ecosystem in order to create a monopoly.

If they were really about creating a fair internet where "pay per crawl" is possible (it already was with L402), they would openly support L402, implement it for their gateway, and try to push the industry towards it. But no, they just want the biggest piece of the pie; a pie that doesn't even exist yet.

As a content consumer, I'm also hoping to be part of the ecosystem. I already use Patreon a lot as "AdBlock absolution", but it doesn't fix the market dynamics. Major content platforms tend to stagnate or worsen over time, because they prefer to sell impressions to advertisers than a good product to consumers.

It always bothers me a little when people describe themselves as a content publisher or content creator. Why content? Why not say you are a short film maker or writer or musician or videographer or journalist or painter or photographer or ...?

In the past, your work has been merely content to the people who are looking for bits -- any bits -- to place between and around the stuff they really care about: ads. These days I suppose your work may be seen by some as grist for the AI mill.

By calling your work content you are acceding to the idea that your work is only valuable in other contexts. It isn't! I bet it's something you made with care and thought.

Or maybe I misread and you really are a content publisher. You're the person running the CMS who is looking for other people's work to monetize. If so, nevermind.

Imagine trying to create and distribute a movie without the backing of your local distributor/broadcasting cartel. Only instead of gating the movie theatre, Cloudflare is the single access to all things web. Also, based in the US, so good luck producing content their government doesn’t like.

It should be front and center, communicated clearly, and easy to understand. If I was an investor, the lack of clarity after reading a few paragraphs would concern me.

So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource?

You could always share things privately or behind a paywall if you don't want them available publicly. But people seem to want to have their cake and eat it too.

I get why a hosting provider would want to limit crawlers to save bandwidth. The "creator" angle is just greed.

I think of these things like they only exist in the past but I'm sure these places are still around only I don't go to them as much now. Probably because all the useful stuff these days exist in mainly reddit and the stackexchanges.

We can dream. ISP says NO!

If no, you can still do that today.

If yes, then imagine the other 1,000,000 people who think likewise. That was how we got where we are today.

https://www.cloudflare.com/en-gb/paypercrawl-signup/

Cloudflare blocks AI scrapers unless they pay the toll.

Edit: Re-read it and just noticed this:

> assuming the AI companies are willing to step up, support the ecosystem, and pay for the content that is the most valuable to them

Lolllllll. Y'all are gonna love my pitch for an airline startup. It begins, "assuming we can turn off gravity..."

Nobody at the head of these large tech firms has a clue about where this technology is heading.

The problem is the arguments they make for why this should happen are quite compelling, especially to those running sites (you'll see plenty of complaints on this forum about it), but theres also a large group of people who think information / code / data should be "free" (see open source code/maps/anything you can think off). So really its just a moral debate that will be lost in the interest of profit (which is ya know good n bad, if AI companies did more caching we probably wouldn't need this, but here we are).

[0] https://blog.cloudflare.com/introducing-ai-crawl-control/