When Apple strikes an exclusive deal with suppliers for parts, it is sound business practice.

When Google strikes an exclusive deal with Reddit, it is ..

Some of you have no idea how businesses work, and it shows.

that's what I said: it is legal.

As soon as someone shows me a search engine that restores quality of searxh, im getting a subscription for work.

It really cany be hard to whitelist sources and index appropiately.

Get goimg nerds , google has fallen.

I remember seeing an unhelpful hyperlink for the first time. It was a random word in the body of a random tech site that redirected to a list of articles from that site tagged with that term.

I remember being stunned, my expectation was that the link would lead me to another website, one that would be an authoritative source on that term and freely accessible.

20 years later we get a paywalled article about fragmented web – and we’re not slowing down.

https://cc.bingj.com/cache.aspx?d=5070227914243&w=ljIRk8yx42...

For example, when I search for product reviews, I always specify reddit. Otherwise the search results are inundated with SEO spam.

Reddit's justification for this is profoundly wrong. Their "public content policy" is absurd doublespeak, and counter to everything the open internet is and hopes to be. You cannot simultaneously call yourself "open" and "public" while refusing access to automated clients. Every client is automated. They even go so far as to say that "crawling" (also known as "downloading") is an "abuse" and violates user privacy.

This is absurd, and not justified. I would love to see legislation that restricted server operators' ability to prohibit automated access in this way, but I suppose it will never happen. Some people in this thread have attempted to justify the policy by saying "they have to protect their income streams". No they don't. You don't have a right to an income stream, and you certainly don't have a right to lie in order to get all the benefits of an open internet with none of the downsides. Noting of course that the "downsides" are in this case actually just "competitors".

Also things like the API fiasco, and also small annoyances like the fact that when you click on an image on reddit, it now goes to a wrapper html page instead of just the actual image (this was one of the reasons reddit was better than most social media...).

It'd be somewhat hilarious if google bought reddit just to archive it and shut it down.

It's like what happened to personal websites when things like Blogger, Tumblr and Facebook popped up.

It's hard to beat something that is easy to set up and pays for hosting but still let's you control moderation. It's a no brainer.

Managing your own domain where users post content is a minefield of problems these days even if you didn't mind the cost of running it.

Something like this might also explain the move to things like Discord over IRC.

IMO, something federation is very good at is solving one slow-moving problem - enshittification of social platforms. It's not immune, of course, but an Elon Musk-style takeover is much harder with Mastodon than Twitter, and it would be hard to run it into the ground in other ways because the platforms are owned by different people and groups.

I just read they have 2000 employees which is also puzzling to me

let assume apple is forced to allow people to make a choice of the default search engine in safari then google might stop paying apple, and ...

Firefox has always allowed people to make a choice of the default search engine, since before it was even called Firefox. I know. I was there building it.

And as for Firefox, Mozilla being forced off Google's teat would be the best thing that could happen to the browser.

Is Google's contract with Reddit exclusive, so that other search engines aren't given the opportunity to also pay?

I highly doubt that, especially since the DOJ would go after that immediately because of antitrust.

So no, pretty sure the blame here is 100% on Reddit unless you have evidence otherwise.

whether or not the deal is exclusive OR companies have to pay to index reddit it's still bad for competition. money has a barrier to entry preventing newcomers.

I can blame reddit for creating the deal and I can blame google for accepting the deal if the effect is bing, ddg and others cannot display reddit results without reaching some deal.

Reddit can charge smaller companies less money. So if there's a problem, again, the problem is 100% with Reddit.

Google is absolutely blameless here. You may not like Google, and you can certainly blame them for plenty of other things. But in this situation, literally all of the blame is with Reddit for deciding to remove their content from all search engines unless they pay. Reddit didn't have to do that. Google didn't make them do that.

Reddit did this. Not Google.

Things are more than the sum of their parts. If you have a ton of beneficial things which can be cobbled together into one bad detrimental thing, the existence of the latter does not remove the benefits of the former.

No such thing, and definitely not the AP.

Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways.

If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.

You don’t even need >90% market share for that to be the case. e.g. Standard Oil only controlled 64% of the US market at most, it was still broken upz

Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg

Eventually it becomes expensive to scrape reddit's data and most people will stop.

Even HN can get a licensing deal if they want to.

If you are producing content, you have every right to do what you want to with the content.

That was a bug, apologies. It should be fixed now.

Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.

[1] https://free.law/recap/faq

(full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)

When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.

Is there a way to export my history? How?

It doesn't solve the problem, but if money is the only thing preventing search engines from accessing Reddit, then what goes for Google also goes for Microsoft.

Monopolies are entirely consistent with free market economics. After all, if there's clearly a "best product" for a particular niche, it's entirely rational (free market actor) behavior for everyone to use the same product, leading to its monopoly in that market segment.

I don't understand why people think this isn't/won't be/shouldn't be a common result of "free market forces".

Having Barriers to Entry != Anti-competitive

Yes, large players have advantages of Economies of Scale.

Just because you can't run an Airline because you don't have money to buy an Airplane isn't anti-competitive.

Today Microsoft, Apple, OpenAI, Google, Amazon all can afford those piddly $60m to license from reddit.

Not Anti-competitive at all.

But saddened by how much corporate-hate by HNers destroys their credibility in debating these thing.

Go ahead downvote

I’m happy to continue this debate if you’d like to start supporting your posts with citations but probably won’t engage further unless you do. Have a great day!

[0] https://www.mojeek.com/about/why-mojeek

https://www.crawlson.com/ https://search.marginalia.nu/ https://wiby.me/ https://searchmysite.net/

And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively.

And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above.

So realistically there's just one search engine for the full web that actually does the work.

Endless iterations of the discussion about how copyright should/shouldn't be are meaningless without considering the larger social context.

The vast majority of creators (artists!?) are not compensated at all, and the users' (content consumers'? the public's?) attention is already 100% fully saturated.

So GenAI chrurning out more kitsch hardly changes that. (It got popular when it was novel, and ... that was it for now. And it became one more brush in the evergrowing workshop.)

And if/when a company creates a product out of it, then that product needs to be scrutinized and we should consider the ethical, social, and political problems. (Because ... politics is a blunt tool whereas ethics is as infinitely nuanced as we make it, whereas actual social considerations ought to be pragmatic and fair.)

And that's the problem. Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.) So what's the cost of this? Does it make some people jobless? Is that good or bad? (Is it good that plant breeding programs, fertilizers, tractors, and irrigation systems made a lot of agricultural workers jobless? Well, in some sense yes as allows a few people to feed many, freeing up time for others to become doctors and artists. In some sense bad, because our current socioeconomic system does not provide real social support - despite enormous redistribution of GDP. [Because brutal inefficiencies in allocation of that surplus. And that's again a political problem.])

... and of course here the status quo bias leads to "technological progress chipping away at inefficiencies" in a capitalist reality translates to "even more things get commoditized", and that in turn in the current shitty socioeconomic system equals "lots of externalities are not priced in, and lots of people are forced to drastically change their lives to adjust to new prices" (ie. price of their labor and/or products going down, so they need to change jobs, yet they barely get any support for doing so) ... and of course ethically it's bad that most people virtually uncritically accept and enjoy the results of progress (new products and services) without giving a fuck about the costs.

Except that isn't what happened. Within the context, 'almost none' referred to size of the group, not to the amount of crawling.

I was discussing how much crawling is being done - outside of the 'almost none' sized group.

The 'almost none' sized group has 'their own results', so they do crawl. Based on that, the rest do not. Ergo, they do not crawl.

A search engine that never crawls seems non-intuitive.

I've no reason to doubt this so I'll attribute it to which results get pushed higher in the rankings. Different entities hold power in the E & W hemispheres and that can influence search results to a degree.

Past that, there are some things I get at Yandex because Kagi seems incapable of searching for them. ex: Long strings like MD5 and larger hashes.

edit:30 seconds after posting I searched a hash and Kagi sent back a ton of results. As recently as last week I got zero results (diff hash).

That's the 2nd time in a week Kagi disproved a complaint of mine. I hope they continue to do so.

Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money.

I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate?

As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.

do you have any reference for your claim?

i use brave search and find it very useful. very rarely there is something i can't find, and when i run into that other search engines are not much better.

This was sarcasm. That system would never happen. If birthday card requires no effort it is worse than nothing, literal noise.

> So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)

Feels like you misunderstood. It is not "10x better" or "advantage" because literally the easier it was for you the less it is valuable.

Value is relative, of course, that's why exchange/trade works, right? And the same applies here too, with the added complications of explicitly "unknowable price", after all I don't know apriori how much value my friend will assign to the card!

In general there are many situations that map closer to your understanding than mine, but I think you are dismissing also a lot of perfectly valid ones where your model simply doesn't apply.

For example if said friend values effort, but "apparent concept vs. actual results fidelity" is not part of effort for them.

Yet still we don't know if they value effort as in time spent (because if they value time spent then it's the same value in both of my scenarios) or if they value some "visible handcraftedness" (effort to produce something that I am known to be very amateur at). And even then the question arises, isn't picking the best tool for the job itself effort?

Of course if this friend simply abbors GenAI shit then they will think it's "low effort" even if I put hours and hours into it (that I wouldn't have put into a traditional image manipulation software, because there I realized soon that I would need months of learning, etc.)

I hadn't really thought about that topic in that way before. Really explains why some of those older MMOs have no desire to really make any improvements, the owners are happy to just keep them powered up and collect a check but have no incentive to invest in making them better.