Even worse, they state that Brave search won't index a page only if other search engines are not allowed to index it. It is morally not their right to make that call. A publisher should have full control to discriminate which search engine indexes the website's content. That's the very heart of why the Robots Exclusion Protocol exists, and Brave is brazenly ignoring it.
Even worse than that, the Brave search API allows you (for an extra fee) to get the content with a "license" to use the content for AI training? Who allowed them the right to distribute the content that way?
I wrote about all this here:
https://searchengineland.com/crawlers-search-engines-generat...
and more references elsewhere in this thread:
https://news.ycombinator.com/item?id=36989129
Amusingly, while I was writing my article, this got posted to their forums, asking about how to block their crawler:
https://community.brave.com/t/stop-website-being-shown-in-br...
No reply so far.
If you post something to the open web, what's it to you who reads it and how? You can block some IPs but that's about it.
I don't know if Brave has a knowledge graph - if they do, I would understand objecting if they filled it in with “stolen” content. But I don't see what's the problem with search.
By the way, isn't everyone's favourite archive.is doing the same thing?
I have no strong opinion on this, curious to hear counter arguments.
> A publisher should have full control to discriminate which search engine indexes the website's content
If you want someone to not see what you publish block him yourself. Also why would you want to do that? Do you want google to own the web or something?
I share your concern about Google having this much power, and I'd add that Microsoft Bing is equally bad but gets away with it because they're smaller. Still, the final decision about which search engine indexes a website is purely the publisher's.
It simply doesn't sound right to say which tool a user can use. It's literally the same as arguing that you should be able to block Firefox from accessing your website and it's Mozilla's fault that they don't respect your wishes as a webmaster to block Firefox exclusively. Or that a VPN doesn't publish its IP addresses so that you can block it. Or a screen reader that processes the text to speech in a way that you disagree with.
Philosophically it seems intuitive to say "I should be able to block a third party that is abusing my site" but it's ignoring the broader context of what "open web" and "net neutrality" actually mean.
I run a service for podcasters. There are podcast apps and directories that either ignorantly make unnecessary requests for content or have software bugs that cause redownloads. I could trivially block them, but I don't because doing so penalizes the end user who is ultimately innocent, rather than the badly behaved service operator. The better solution is primitives like rate limiting, which I use liberally. Plus, blocking anyone literally has a direct effect of incentivizing centralization on Apple, Spotify, etc. and making the state of open tech in podcasting even worse.
> the Brave search API allows you (for an extra fee) to get the content with a "license" to use the content for AI training? Who allowed them the right to distribute the content that way?
I don't think there's any court at this point that would back you up that freely published content annotated with full provenance cannot be scraped and published for a fee. Services like this have existed for decades. If you don't want your content scraped, put it behind a login. Especially considering this only applies when you allow other search engines and if you think Google and Bing aren't using your content to train AI, you're off your rocker.
1. User agents should identify themselves
2. A crawler is not a User agent - it's an agent for Brave
>I don't think there's any court at this point that would back you up that freely published content annotated with full provenance cannot be scraped and published for a fee.
You can't end-run copyright like this: just because something is publicly available doesn't mean anyone can redistribute it. Look at the legal issues & cases relating to Library Genesis.
What if I consider (some or any of) my ideas to be un-indexable, not directly suitable to representation in any hierarchy other than those I may set them in?
To me as a search engine end user, this kind of behavior is undesirable. Why would I want a website to selectively degrade my experience because of my choice in search engine or browser?
Brings back horrible flashbacks of “this website is only compatible with IE6”.
Also, these search crawls by the browser do not identify themselves beyond the Brave standard UA header, namely a plain Chrome user-agent string.
How many Chrome users have opted in to sending data to Google.
Sometimes uninformed consent is not actually consent. These so-called "tech" companies love to toe that line.
Baidu
Bing
Brave
Google
Yandex
https://www.gnod.com/search/?engines=p,o,br,n,q&nw=1
(If you want to also search image libraries like Flickr and Pexels, click on "more engines" to select all places you want to search)
I'm so skeptical that I'm just now starting to develop a feeling of trust towards DuckDuckGo.
In the browser domain, Mozilla is the only company of which I feel that it is genuinely pro-customer.
So I give all my stuff to Google and hope that they at least just protect it from hackers, while I am aware that they analyze my data in order to see how they can monetize me better, but at least with anonymity in regards to 3rd parties. I just hope I'm not wrong.
On the other hand, the Google lawyers seem to have found an excuse to link some proprietary code into Chome (that's not part of Chromium). Does anybody know what that excuse is, and if it provides a loophole large enough to close off Chrome development?
It also explains why Brave Search is slow today I suspect... :)
Usually google+tineye+bing+yandex will yield some useful results
It surprisingly comes up for image searching. Google for example has been known to censor images of Tiananmen Square.
(though I've recently switched away from Brave Search since the goggles subscriptions (the reason I've switched to Brave) have a big bar right at the top and the whole top settings+goggles shifts search results on load!)
Now... I would just like to see the full https url on searches.
I already love the look, Brave summarizer AI and general results!
The current policies don't allow it, sure. But then there will be gradual changes to those policies, and all sorts of dark patterns that make 99% of users leak their data (think "enabled by default but opt-out").
Is there any guarantee against this?
You have to realize data is worth less over time -- a lot less. So the issue would be whether users stick if (heaven forbid) a change of control left Brave in the hands of unethical owners. By design, open source, network sniffing, and other auditors would catch wise and flame such a corrupt Brave into the ground. That's the best we can do here. There is no safety property ("X" holds for all future program states) enforceable on a company.
Bamboo sign
Give it a try on Google images. You'll see that nearly all the results are x-rays of people with ankylosing spondylitis, a form of which is commonly referred to as "bamboo spine".
I tested out brave search - it correctly shows 90% signs however it does also show a few spines.
Google still shows incorrect results months later. It's by far the worst of all the search engines in the list for this simple and obvious search.
I actually like Brave here for my test better than Google. I typed in a few cities, just wanting to see the skylines and such.
Brave gives me good photos, some stock photos, etc. Google gives me pictures from recent news articles, which isn't overly helpful IMO.
That already makes it worth of support.
But Google having become so bad of late has made switching quite easy, even if brave is not getting better super fast, Google unfortunately is getting worse and making up for it.
What are the censored image searches you found?
See: https://help.kagi.com/kagi/why-kagi/kagi-vs-google.html
c.f. yep.com
Brave search was my default for quite a while, until a few weeks after they got rid of bing results. As soon as that happened, stuff just wasn't showing up that I'd expect to be there, and 90% of the time I'd follow up searches with !g or !ddg just to get something decent to show up. The index just felt severely lacking, or the relevancy was pretty far off base.
Would you say search has greatly improved over the past month or two?
A recent example from my search history, "doors of stone release date". The author has announced a new novella releasing Nov. 2023, but not the actual book Doors of Stone. The google infobox gets this wrong, but the first result is correct. Brave accidentally gets it right that there's no release date for the book, but misses the novella announcement and all but one of the results are blogspam.
My god, there is so much telemetry in FF now, and it's tricky to hunt down all the about:configs to disable it. Not friendly or privacy conscious at all. Do you really want Mozilla to get pinged with your IP address every time your browser process starts and exits? Yuck!
Quick FF enshittification example from 2 months ago:
Alert HN: Mozilla puts advertising into Firefox AGAIN
https://news.ycombinator.com/item?id=36351322 (48 comments)
Mozilla stops Firefox fullscreen VPN ads after user outrage
https://news.ycombinator.com/item?id=36085642 (220 comments)
They literally can't win. One group of vocal users is outraged by how much money Mozilla takes from Google while the other half screams about how Mozilla is trying to gasp gain additional revenue streams that isn't taking money from their biggest competitor.
> Do you really want Mozilla to get pinged with your IP address every time your browser process starts and exits? Yuck!
No, but I really don't give a shit either. At a certain point, I looked in the mirror and said life is too short to care about stupid shit like that. If I was a spy or a journalist in some state like China or Iran, maybe I would care. But this feels odd to hone in on when any website you go to is collecting all sorts of info of this sort.
Mozilla doesn't go about it in as upfront way as Brave does, IME, but stuff like VPN, Pocket and other browser-related services I mind not at all.
I have no sympathy to the current political shitfest that Mozilla is as an organization, but as makers of Firefox I feel like Mozilla is in an impossible bind: Their users expect a fairytale of an independent, donation-funded browser that people spontaneously adopt, and go nuts about stuff like the inclusion of Pocket. I know, I used to be one of those people back when Pocket was introduced. But reeing about Mozilla trying to have independent funding by giving people useful services is just strange. It's exactly what they should be doing, and Brave setting up revenue streams like Talk and Search is great. Especially because they operate in the normal money universe for those of us who aren't terribly enthusiastic about crypto.
I'm curious to see what you think about this. If you're not okay with Firefox telling Mozilla your IP address every time you connect, does the same go for Brave sending entire pages of your search results to them? This also includes which results you've clicked on.
The thing is that it will always get worse. Every company needs to grow in order to stay alive and eventually quality and your user experience will suffer because of this, but here's the thing: Always choose the upstart competition but just be prepared to jump to the next up-and-comer after that. For me, I found DuckDuckGo getting worse over time (probably not on purpose, just spam) and somehow Brave is better so I'm sticking with that, but as soon as Brave decides to fuck me (and they will!) then I'll be jumping to whoever the new underdog is at that time.
There is no rule that this is true, and many user agents exist _specifically to not be identified_. See Tor and other privacy-centric user agents.
> A crawler is not a User agent - it's an agent for Brave
You know, I thought "what does Wikipedia have to say on this matter?" and sure enough:
> Examples include all common web browsers, such as Google Chrome, Mozilla Firefox, and Safari, as well as some email readers, command-line utilities like cURL, and arguably headless services that power part of a larger application, such as a web crawler.
I can't even make that up.
> just because something is publicly available doesn't mean anyone can redistribute it
You're mistaking reselling content with providing access to it. By your logic, caching proxy servers would be illegal on the grounds of copyright. The physical act of downloading files necessarily creates copies of the data every step of the journey from the source server to you. There's a material difference between paying someone for a copy of some content and paying someone to fetch content for you on your behalf. Nothing about copyright law specifically requires the person physically acquiring the content is the one who ends up consuming it.
And to use that analogy even further, if you want to block Chinese visitors you block Chinese IPs. You do not add a file called "countries.txt" containing "China block" and then expect Chinese users to see it and voluntarily cease to use it, and threaten to sue them if they don't.
Repeatedly asserting that "the final decision is with the publisher" is stupid. That is the point you seem to want to defend. Defend it! Give us a reason. Just saying the same thing over and over again doesnt make it true.
Much of the problem with search today arises from websites showing googlebot what it wants to see and showing real users. I have to manually remove entire domains from google search as they often appear 1st yet don't show any content without me signing up for an account. Clearly that's not what they are showing to google.
There should be no differentiation between a crawler and a human being with regards to what is being served.
First I've heard of a "metasearch" engine. I just tried SearX and it gave me no results for my projects and only a couple for "Mastodon", but the animal not the software.
Brave offers a simple deal: if you believe or have audited their technical claims, you give them fully anonymized snippets of URLs and web pages, and they improve their search index for everyone
Nobody's asking you to personally fight for everything. Life truly is too short for that. All we're asking for is your tacit support, or failing even that, your abstinence from the conversation.
I don't personally have time to look after stray dogs in my area, for example. But I sure as hell don't come out with "I looked in the mirror and decided I don't give a shit" when I meet someone who does care about that. Not online and not offline. Instead I'll be supportive and tell them how amazing they are for spending their valuable time on this.
Even if it's something I don't personally care about at all, or even if I think it's a massive waste of time but I can see it's important to someone else, I'd still never tell them that I don't give a shit.
Is it too much to ask you to have the same respect for people who care about important issues like online privacy? If it's not important to you, that's fine! Go be somewhere else instead of interrupting people who do care about this. There's about 500 million other conversations happening on the internet at this very second. Surely one of them is something you actually do care about enough to engage with in a positive way?
At this point, Brave is a more promising bet than MZ.
example.com/correcthorsebatterystaple
Taking either step you suggest (along with robots.txt or eqiv.), it would seem fair to expect that Brave, Bing, whomever, would not feel it their neutral/natural domain to include in a public index.
You're certainly allowed to try, but I don't see why indexers should be mandated to collaborate with you. They serve their users, not you.
I mean, "allow" or "regulate" don't _really_ apply here - there was never any enforcement regime around robots.txt, just a convention based on the general expectation that you don't claim ownership of whatever passes your line of sight.
The issue is the metadata ranking is garbage, the documentation extremely lacking, and the support is nonexistent. The software is confusingly laid out, with various options strewn all over the place and absolutely no clue given as to what they do.
If you want a peer driven search engine, perhaps the quick start guide shouldn’t involve having to read the source code.
That tends to upset some engines including Bing I think.
Click two links down in the same menu:
https://help.kagi.com/kagi/why-kagi/kagi-vs-brave.html
Kagi Search includes anonymized requests to traditional search indexes like Google and Bing as well as sources like Wikipedia, DeepL, and other APIs. We also have our own non-commercial index (Teclis), news index (TinyGem), and an AI for instant answers. Teclis and TinyGem are a result of our crawl through millions of domains, focusing primarily on non-commercial, high-quality content.
Our unique results combined from all of these sources help you discover the best content you can possibly find online, sometimes from the quieter places on the web.
It's not as good as Google was at its peak (and Google itself has degraded severely in quality, IMO), but it's good enough that I can generally find what I'm looking for with a minimum of effort.
I run maybe 1 in 40-50 searches with "!g" because Brave is insufficient, for context.
Exactly my experience. I hadn't connected it to them getting rid of Bing results, but it makes sense. I've had to use the bang redirects to other search engines a lot more too, to a level I haven't had to in more than a year.
The difference might be that they (including myself) don't ask search engines for facts like "doors of stone release date". They'll search for "doors of stone", find personally reliable sources like Wikipedia, Fandom, Goodreads, browse them and decide on an answer. When sources fail to appear, they'll either refine the search (like "doors of stone rothfuss") or call it a failure and maybe try a different search engine.
This is one the reasons why Brave has been good for me so far. When a relevant Wikipedia article exists, it shows it, even if the title doesn't match. Whereas lately DDG and others don't. In fact, you can see this with "doors of stone". Brave shows "The Kingkiller Chronicle", DDG doesn't at all, Google has it low down in the results.
It also shows Reddit discussions without needing to explicitly filter for it. And I use ad block to remove the AI summariser that takes up half the screen, it's not what I want from a search engine.
I will say there are times when it just falls flat. Like I will search for a brand or specific thing , expecting to get to the home / login page for that brand, and it just flat out gives me weird results.
But when I put the !g in front of the query, the first result is always what I wanted.
On the other than, when doing more general searches, Brave is on par or better than google.
Conjecture: that might have to do with filtering SEO'd results. It is probably difficult to get rid of the pages that are meant to look a whole like the legit brand pages but not get rid the brand page itself.
The position of Brave of not accepting to be blacklisted if not all search engines are blacklisted is a pragmatic one. It works against Google's search monopoly, but still gives them some legal coverage since the robots.txt is not completely ignored, in case that indeed matters somewhere. I think it's an elegant approach well suited for the current state of the web, one that serves the greater good. And Brave, but that's completely fine.
To massively (over)simplify the argument to its essence (and ignore other important points): the publisher goes through the trouble and expense of creating the content The publisher then allows its content to be copied by a search engine only because being shown in search results gets it traffic back. The traffic it gets in return has value, and the publisher is happy for this arrangement to continue as long as the value of the traffic is more than the cost of producing and serving the content.
Brave offering a "license", for its own financial benefit, to "allow" others to use the content for LLM training gives zero benefit to the original publisher. This is why I use words like "sleazy" to describe Brave's position.
This argument applies to Google and Microsoft. Right now both are failing at citing sources in their generative AI search results. That is terrible and I hope it's fixed soon, as otherwise they're being sleazy scrapers as much as Brave is.
Finally, I wholeheartedly disagree they what Brave is doing is for the "greater good". The fact they charge extra for the "license" to use the content for LLM training shows that.
Then you need to gate your content such that it is not available openly to the public.
This falls inline with many objections to Google's WEI. If you host content openly and allow access freely, then don't be surprised when people access it at will and use it for free.
https://stackdiary.com/brave-selling-copyrighted-data-for-ai...
What possible justification could Brave have for this policy?
Loosely related XKCD: https://xkcd.com/927/
My problems with Mozilla are:
- Misuse of money: the browser team have brought in lots of money over the years (we talk billions) and the foundation is milking it dry. If the income created by the browser had stayed with the browser team they would have had funding for years to come.
- Being dishonest: Mozilla has sought donations for Firefox and I think many of us have donated thinking we supported Firefox, while in reality the Firefox team funds itself and the rest of Mozilla and Mozilla isn't even allowed to send money the other way.
- Not being up front about what they do: they more or less lied about their relationship with Pocket. I like Pocket, both the product and as a way to bring in income, but whenever it comes up, everyone who was there starts thinking about their lies.
- Nerfing the extension API.
- Writing "dear community members" in emails begging for money while simultaneously being rude to us in responses to real issues in Bugzilla.
Now, if anyone think I use Chrome, think again.
I am still optimistically waiting for authorities to wake up and punish Google the same way they punished Microsoft - huge fines and browser ballots - but that does not mean I give Mozilla a free pass ;-)
Can you elaborate? I don't really remember much about those kinds of things.
> - Nerfing the extension API.
This I think was necessary if you want a modern, secure browser. Wouldn't be against a more extensive extension API that double-checked with the user wrt access to more dangerous permissions, which is partly what the current ones do already.
Today however Mozilla stalls at giving us back basic primitives like for example a programmatic way to hide the top tab bar when an extension provide an alternative tab bar like Tree Style Tabs or Sideberry.
Meanwhile Mozilla the organization boots people for politics, wants "more than deplatforming" and uses as one of their examples organizations deciding what I should see on the Internet - preferring outlets that Mozilla themselves like, naturally. They publish stuff with the gist of "did you encounter other politics on YouTube, how scary".
Meanwhile the organization spends money on getting a sneaker designer to make time-limited color themes for their browser and writes a pile of copy about how cool it is that a sneaker designer painted the browser blue.
Meanwhile, Brave releases a user-customizable filtering function for their search engine so you, not someone else, can decide from what POV you see the Internet and added native vertical tabs to the browser.
One organization is blatantly political, the other just makes a good browser.
You can run a similar set of comparisons for Vivaldi, who are also clearly a much more product-focused organization and it shows. They have some more visible political leans as an org, but still far milder than Mozilla's, a lot of those leans being just being vehemently anti-crypto, with the biggest focus going to user control and privacy, as it should.
But yes, Eich and his horrible treatment of gays, which he still hasn't renounced (instead doubling-down and saying that the "deal" was that they could have civil unions but they went too far to get marriage too — they can ride on the same bus, but they can only sit in the back), as well as his nonstop promotion of conspiracy theories is something no sensible person would want to support. If you think Mozilla "[booted Eich] for politics" instead of for taking public actions that tarnished the reputation of the company, you seem like the type of fellow who would have cheered Stephen Douglas in the Lincoln-Douglas debates. As Douglas repeatedly stated, laws concerning the sale of "negroes" are no different from laws concerning the sale of dry goods or liquor, and the people of Illinois should no more tell Missouri how it can sell people than it should tell them how it can sell dry goods — that's Missouri's politics, and Illinois should mind its own business.
A search engine index is a search engine index. It can have an economic impact, but it can not be an economic exchange, since it is a technical artifact.
Though I think I understand what you are trying to say - this is also a commercial relationship where both sides can profit. You are free to interpret the relationship between publishers and search engines with such a capitalist lense, that does not mean those mechanisms govern the actual rules. That a publisher is happy with what happens here is of no real concern. If any rules apply we are talking copyright, maybe media law, where happy is not a relevant category (ok, that of course can matter, it wouldn't here if the search engine uses a right).
I did not touch the LLM training data in my comment, as I did not read up on what Brave is really doing there. If Brave were really to sell complete texts from others, that would not be legal under copyright laws I'm familiar with, so I kinda doubt they do that.
You do if the word has a legal definition and you refuse the benefits tied to that legal definition. Eich is free to call it whatever he wants as long as people get the benefits of marriage, something he paid to stop from happening. What kind of asshole thinks it's OK to treat someone poorly because of quibbles over how everyone else is allowed to use a word? For that matter, Douglas is free to call black people "human goods" so long as he doesn't allow them to actually be bought and sold by controlling legal definitions that allow for it.
I specifically gave the example of selling it not being covered by what I'm talking about. But other parties can definitely "xerox" a book and read it. Use its content. Talk about what is written in it. Quote it. There are limits to the controlling rights of a publisher.
As far as making that the law, that's exactly what happened. Eich and his fellow ethically-challenged smallbrains then changed the constitution so that the law no longer applied.
> You just justify it by tortured logic based on your traditional superstitions.
Why are the American left so obsessed with religion? You seem to think it is the basis for all disagreement with your absurd views. Guess what: almost everyone in the world disagrees with you. And it isnt on religious grounds.
> As far as making that the law, that's exactly what happened. Eich and his fellow ethically-challenged smallbrains then changed the constitution so that the law no longer applied.
So you are making an argument that one group of people are allowed to unilaterally pass laws that redefine common terms, but you say that anyone trying to undo that illogical change is "trying to change the definition"?
Do you not see how hypocritical that is?
Also FYI, insulting people doesn't make your argument look stronger. It just makes you look very desperate. I know that American leftists almost never have to justify their views to anyone and have no practice defending them, but try a little harder.
Thank you, Judge Douglas.
> Why are the American left so obsessed with religion? You seem to think it is the basis for all disagreement with your absurd views.
Because you're using your superstitions to tell other people what to do. Guess what? Everybody with at least half a post-enlightenment brain disagrees with you.
> So you are making an argument that one group of people are allowed to unilaterally pass laws that redefine common terms
No, my argument from the start has been that one group is wrong to do so, just like Lincoln said that one group is wrong to make laws allowing slavery to expand. This type of consistent thinking is difficult for superstitious people to follow, but with practice, it will come more easily.
> Also FYI, insulting people doesn't make your argument look stronger.
I'm just saying what everyone is thinking. If you don't like it, maybe Eich (and you) should stop being publicly wrong. For that matter, you should see the insults that Lincoln hurled at Douglas. A hint for you: your insults would land better if you were right.
There is no "legal definition".
I have no fucking clue who this person is. Stop bringing him up. It is insanely cringe.
>Because you're using your superstitions to tell other people what to do. Guess what? Everybody with at least half a post-enlightenment brain disagrees with you.
I am not religious you moron. Stop making random assumptions. How many times does this need to be explained to you? Why are you fucking stupid American leftists incapable of understanding that people can disagree with you for GOOD reasons. You aren't automatically right about everything just because you're on the left and you were born in the USA. You're an absolute fucking dolt.
Figures you would have no knowledge of American history nor even the basic ability to learn about a topic that has been provided for you. And you're the one calling people stupid dolts. Talk about cringe.
If you aren't relying on the superstitions passed down from your parents, that is an even stronger indictment of your mental abilities. At least Eich has the excuse that he's too uncomfortable removing himself from his community in order to think critically.