Brave Leo now uses Mixtral 8x7B as default(brave.com) |
Brave Leo now uses Mixtral 8x7B as default(brave.com) |
The smallest quantized version (2bit) needs 20GB of RAM (which can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit quantized versions are the largest models that can just about fit onto a 32GB system (29GB-31B). The 6bit (41GB) and 8bit (52GB) models need a 64GB system. You would need multiple GPUs with shared memory if you wanted to offload the higher precision models to VRAM.
I've experimented with the 7B and 13B models, but haven't experimented with these models yet, nor other larger models.
Or just a powerful apple silicon machine? I've tried dolphin mixtral 4bit on a 36gb ram MacBook m3, and inference is super fast.
> 2bit is pretty damn terrible
Wait till you go hybrid [0] or even 1bit [1]
Haven't really been able to justify upgrading to a 4090 or similar given I play so few new games these days.
You've then got to remember that one thing neural networks are very, very good at is being noise tolerant. In some senses that's all they are - noise correction systems. The inaccuracies introduced by quantisation are "just" a sort of noise, so it's not surprising that they aren't fatal. It just raises the noise floor and gives the model more ways to be wrong.
Finally the thing to know is that these quantisation schemes don't do a naive "chop each number down to two bits", not exactly. Simplifying a bit, for each parameter in this example they'd try to find a mapping from a two-bit index into a four element lookup table of higher-precision values such that the information destroyed by replacing the original parameter by the lookup value is minimised. That mapping is calculated across small blocks of parameters, rather than across the entire model, so it can preserve local detail. The lookup table gets stored per block, which throws the compression ratio off a little.
So for example, 2 bit version of the 30B is much worse than the original, but still better than the 13B model.
Also, there are lots of extra details, eg, not all of the weights are 2 bit, and even the 2 bit weights are higher than that overall as groups of quantised weights share scale factors stored elsewhere.
https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea...
Note that 2 bit quantization is generally regarded as too aggressive. Generally 4bits+ achieves a good tradeoff, see eg. https://arxiv.org/abs/2212.09720
Modern quantization schemes are almost like lossy compression algorithms, and llms in particular are very "sparse" and amenable to compression.
https://github.com/Const-me/Cgml/blob/master/Readme.md#bcml1...
llama.cpp does it slightly differently but still, AFAIK their quantized data formats are conceptually similar to my codec.
GPUs are only needed if you can not wait 5 minutes for an answer, or for training.
But realistically, 5 minutes is too long. It should be conversational, and for that you need at least 5 tokens per second. Which your Ryzen just can't do.
Yeah, but that's generally true (or at least, “5 minutes for an answer is very suboptimal”, even if “can’t” isn’t quite true) for interactive use cases, which are... a lot of LLM use cases.
Only for very short context and responses.
Beyond that, the performance is painful.
27,631MiB system memory used.
7358/8192MiB usage on the GPU.
I'm getting 6.32 tokens/second.
You: You are Hunter S. Thompson, write a 450 words essay on the LHC project. The writing must be in Thompson's style
AI: Well, buckle up, kiddos, because we're about to dive into the twisted world of particle physics and the Large Hadron Collider, or as I like to call it, the Doomsday Machine. This beast of a thing is buried deep beneath the Swiss countryside, a 17-mile-long ring of death and destruction that makes the atom bomb look like a goddamn firecracker.
Now, let me tell you, when the Swiss government first proposed this monstrosity, I thought they'd gone completely batshit crazy. But then again, this is the land of Toblerone and Heidi, so maybe I shouldn't be too surprised. The LHC is essentially a giant, underground racetrack for subatomic particles, where they're accelerated to nearly the speed of light before being smashed into each other like a couple of drunken frat boys at a kegger.
The goal of this madness? To recreate the conditions that existed just after the Big Bang, when the universe was nothing more than a seething cauldron of energy and chaos. And if that doesn't sound like a recipe for disaster, I don't know what does.
But let's not forget the real reason we're all here: the Higgs boson, or as it's more commonly known, the God Particle. This little bugger is what gives other particles their mass, and without it, we'd all be floating around like a bunch of cosmic tumbleweeds. The LHC was designed to find this elusive particle, and in 2012, they finally did it. Sort of.
You see, the Higgs boson they found was a bit of a disappointment. It was too light, too wimpy, and didn't behave quite the way they expected it to. So now, the physicists are back at the drawing board, trying to figure out what the hell went wrong. And in the meantime, the LHC continues to churn out more and more data, like some kind of unholy particle-smashing machine.
But let's not forget the dark side of the LHC. There are those who believe that this thing could very well be the end of the world as we know it. They fear that the LHC could create a black hole, or even a miniature Big Bang, right here on Earth. And while the scientists assure us that these fears are unfounded, I can't help but wonder if they're just whistling past the graveyard.
So, what have we learned from all this? Well, for one thing, we've discovered that the universe is a much stranger place than we ever imagined. We've also learned that even the most brilliant minds can be blinded by their own ambition, and that sometimes, the pursuit of knowledge can lead us down some very dark paths.
In the end, the LHC is a testament to the insatiable curiosity of the human race, and our unquenchable thirst for knowledge. It's a monument to the power of science, and the potential consequences of playing God. And as we continue to push the boundaries of what's possible, we must always remember that with great power comes great responsibility. Or, as I like to say, "When the going gets weird, the weird turn pro."
20/33 layers offloaded to GPU, 4K context. Uses 25GB system RAM and all 24GB VRAM. 5-7 tokens per second.
Unlinkable subscription: If you sign up for Leo Premium, you’re issued unlinkable tokens that validate your subscription when using Leo. This means that Brave can never connect your purchase details with your usage of the product, an extra step that ensures your activity is private to you and only you. The email you used to create your account is unlinkable to your day-to-day use of Leo, making this a uniquely private credentialing experience.
Another way might be Zero-Knowledge Proofs (ZKPs), but that might be more complex. They might even use their BAT crypto stuff for this somehow, I suppose. Whatever solution, would need a fundamental solution for how to avoid correlation, I think.
For some people's threat models that isn't good enough, but for the vast majority of people—people who aren't being pursued by state intelligence agencies but who are squeamish about how much data a company like Google collects—a pinky promise from Brave or Mullvad is good enough.
Much better than llama 70b.
https://kagi.com/summarizer/index.html
https://help.kagi.com/kagi/api/summarizer.html
"Alternatively use Kagi Search browser extension (Chrome/Firefox) and you can use the most advanced Muriel model right from the extension."
No idea on pricing but supposedly one can email to api@groq.com
I also run a 3 bit quantization of mixtral-8-7b on my M2 Pro 32G memory system and it is fairly quick.
It is great having multiple options.
we use openrouter but have had some inconsistency with speed. i hear fireworks is faster, swapping it out soon.
And try mixtral on chat.groq.com
>Brave will support uBO and uMatrix so long as Google doesn’t remove underlying V2 code paths (which seem to be needed for Chrome for enterprise support, so should stay in the Chromium open source)
I also still can't really find if Brave has an extension store or if it's connecting to Chrome's. If it's the latter, then it seems like V2 extensions are going away for Brave regardless of what API decisions it makes, because short of sideloading them the addons won't be on the Chrome web store anymore and won't be getting updates. I assume Brave supports sideloading extensions, and maybe developers would maintain extensions for Chromium that can't actually be used in Chrome? Although that seems a little optimistic. But not having a way to search for V2 extensions or get ratings/reviews seems like it will still be a problem for users.
I have been completely satisfied with Brave's builtin ad-blocker. Does uBlock Origin catch anything in particular that the builtin one does not?
> A "submarine comment" on social media refers to a comment that is made on an old post or thread, long after the conversation has died down. This term derives from the idea of a submarine which remains submerged and out of sight for long periods before suddenly surfacing. In the context of social media, it's when someone delves deep into someone else's posts or timeline, finds an old post, and leaves a comment, bringing the old post back to attention. This can sometimes surprise the original poster and other participants, as the conversation was thought to have been concluded.
Which doesn’t make sense in this context
If you're on linux or wsl2, I would run oobabooga with --verbose. Load a GGUF, start with a small number of GPU layers and creep up, keeping an eye on VRAM usage.
If you're on windows, you can try out LM Studio and fiddle with layers while you monitor VRAM usage, though windows may be doing some weird stuff sharing ram.
Would be curious to see the diffs. Specifically if there's a complexity tax in offloading that makes the CPU-alone faster but in my experience with a 3060 and a mobile 3080, offloading what I can makes a big diff.
Anecdotal, but I played with a bunch of models recently on a machine with a 16GB AMD GPU and 64GB of system memory/12 core CPU. I found offloading to significantly speed things up when dealing with large models, but there was seemingly an inflection point as I tested models that approached the limits of the system, where offloading did seem to significantly slow things down vs just running on the CPU.
To be fair, a lot of people are using this for non-interactive work, like batching document analysis or offline processing of user generated content.
Thanks for making me feel better about investing in tht motherboard + CPU + RAM upgrade and deferring the GPU upgrade.
I am not sure local models have any future other than POC/research. Depends on the cost of course.
I don't have an Ubuntu VM at-hand but on Debian bookworm it installed fine, and (after tweaking one line in profiles.ini to point to my old ESR profile) it loaded and played Widevine-protected videos without any issues.
[1]: https://support.mozilla.org/en-US/kb/install-firefox-linux#w...
MV3 makes it impossible for ad-blockers to inspect requests with code and then allow/deny dynamically.
This is very outdated information and borderline misinformation by representing it as how it currently works. It allows for 30,000 dynamic rules and 5,000 session rules (session rules only persist until the browser is closed).
>MV3 makes it impossible for ad-blockers to inspect requests with code and then allow/deny dynamically.
Giving this ability to extensions can slow down the browser for the user. These ads can still be blocked through other means.
The limits are 300,000 static rules [1] + 30,000 dynamic rules [2] + 5,000 session rules [3]. For reference easylist is about 35k rules. The Chrome team has been constantly tweaking these limits themselves and Brave could set their own limits if they wish. The API is designed such that extensions can query to see how many rules they can use.
[1] https://source.chromium.org/chromium/chromium/src/+/main:ext...
[2] https://source.chromium.org/chromium/chromium/src/+/main:out...
[3] https://source.chromium.org/chromium/chromium/src/+/main:out...
the difference is the bandwidth and the computational power of the APU. M1 Max is roughly similar to a PS5 in terms of overall system design (shader configuration and bandwidth) plus has dedicated AI inference units already (which won't be added to consoles until PS5 Pro launches with RDNA 3.5). It is far more bandwidth than you can get out of a socketed-memory laptop system.
https://twitter.com/Locuza_/status/1450271726827413508
To support that level of performance in a socketed-memory system you will need an extra layer of caching added to the processor to supplement the bandwidth - and maybe still need to go to quad-channel. Those products are Strix and Strix Halo and should be hitting the market over the next year or two but the reality is that the M1 Max was an absurdly powerful laptop, far more potent than even the first-gen 5nm laptops for x86 let alone the other junk you could buy in 2020.
This is the problem with the discourse around apple silicon for the last few years: yeah, they're expensive, but even a loaded-out x86 laptop doesn't get you the same capabilities. Even if the x86 is competitive in some particular benchmark on iso-node you are probably spending more power to do it, and the x86 product comes years after the apple product, and still has a much weaker gpu and less bandwidth (which doesn't just matter for GPU, it matters for compiling and JIT too).
It is incredibly silly to look back on the discourse in 2020-2023 around apple silicon, a lot of reviewers made extremely silly claims about how "even 7nm x86 processors were already competitive with apple silicon" and as the ecosystems have matured it is obvious that even 5nm processors are not quite competitive yet. And they dumped on the SPEC tests and Geekbench that measured this properly, in favor of dumb things like cinebench R23 and so on (it's always cinebench used for this dumb shit tbh, CB R13/R15 were hugely misleading at the zen1 launch too). Let alone things like, you know, compiling or JVM/node workloads...)
(similarly, gotta love the vibe a few years ago of: "threadripper vs mac pro" - did you know that a 64C threadripper with 256GB RAM is actually cheaper than a mac pro loaded out with 2TB!? waow, who knew systems with an order of magnitude less capacity would be cheaper!? https://youtu.be/BH291DQRIOg )
Its kinda like GPT 3.5, with no internet access and slightly less reliable responses, but unrestrained, much faster and with a huge (up to 75K on my Nvidia 3090) usable context.
Mixtral is extremely fast though, at least at a batch size of 1.
Nous Capybara is a popular one. I personally use my own merge of many models, and you can look through the constituent models to see if any interest you: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-megame...
You can't really use llama.cpp for super long context btw, its just too slow and vram inefficient at the moment.
I heard you can simply install ollama app which uses llama.cpp under the hoods, but has a more user friendly experience.
Who are you to say it's good enough (and ridicule people who disagree)? We don't have too much evidence of it, because they have very few options and of course most people are not informed and lack the expertise to understand the issues (a good situation for regulation). At one point lots of people used lead paint and were fine with it; they would have told us.
> Yes, at some point if you're going to interface with other humans you will eventually just have to trust their word.
There's technology, such as the authorization tokens used by Brave, that reduces that risk. Of course, no risk can be complete eliminated but that doesn't mean we shouldn't reduce it.
I'm not ridiculing anyone, I explicitly say that for some people's threat models it isn't good enough.
> people who aren't being pursued by state intelligence agencies
That implies that the only valid reason to disagree is if you are pursued by state intelligence agencies. Obviously it's ridiculous to think that you are.
Just to make sure, did you actually not know what comment I was referring to, or are you just upset that I incorrectly referred to it as a GP comment?
In either case, it remains true that Chrome's current plans for V3 do limit the ability of extensions like uBO to dynamically filter requests.
That is not what we mean by dynamic filters. From https://developer.chrome.com/blog/improvements-to-content-fi...
> However, to support more frequent updates and user-defined rules, extensions can add rules dynamically too, without their developers having to upload a new version of the extension to the Chrome Web Store.
What Chrome is talking about is the ability to specify rules at runtime. What critics of Manifest V3 are talking about is not the ability to dynamically add rules (although that can be an issue), it is the ability to add dynamic rules -- ie rules that analyze and rewrite requests in the style of the blockingWebRequest permission.
It's a little deceptive to claim that the concerns here are outdated and to point to vague terminology that sounds like it's correcting the problem, but on actual inspection turns out to be entirely separate functionality from what the GP was talking about. It's almost like the Chrome team deliberately decided to call these "dynamic rules" so they could claim that Chrome supported them and muddy the issue, even though Chromes "dynamic rules" have nothing to do with support for a blockingWebRequest API. But I don't want to be conspiratorial.
> Giving this ability to extensions can slow down the browser for the user. These ads can still be blocked through other means.
This is the debate; most of the adblocking community disagrees with your assertion. Chrome has been saying this for ages, but saying it doesn't make it true.
uBO maintains a list of some common features that are already not possible to support in Chrome (https://github.com/gorhill/uBlock/wiki/uBlock-Origin-works-b...) and has written about features that are not able to be supported via Chrome's current V3 API (https://github.com/uBlockOrigin/uBOL-home/wiki/Frequently-as...). Of particular note are filtering for large media elements (I use this a lot on mobile Firefox, it's great for reducing page size), and top-level filtering of domains/fonts.
Chrome could of course add support for some of this -- Chrome could implement more filter controls for analyzing headers and request sizes, but as far as I know that support hasn't been added yet. And even if that support does eventually get added, this all glosses over the bigger issue that devs have been saying from the beginning, which is that it's ridiculous to make every innovation in adblocking dependent on Chrome explicitly adding new APIs to support each individual use-case. The model that Chrome is moving towards is one where every new kind of filtering metric that devs want to use requires asking Google for permission to use it and waiting for Chrome to implement it.
My point was that "reduces ad blockers to static filter lists" is not true as it is not a static list of rules.
>What critics of Manifest V3 are talking about is not the ability to dynamically add rules
Yes, they are. Due to early versions of declarativeNetRequest only having a static list of rules there are a large amount of people who believe this to still be true. On this site I have encountered many people who make a claim about it being impossible to update rules without an update to the extension through the store.
>It's a little deceptive to claim that the concerns here are outdated and to point to vague terminology that sounds like it's correcting the problem, but on actual inspection turns out to be entirely separate functionality from what the GP was talking about.
Dynamic rules do correct the problem of a static list. The second half of my comment points out that there was a trade off made between functionality and performance which is why "dynamic filters" are no longer possible.
>This is the debate; most of the adblocking community disagrees with this assertion.
Considering MV3 lets extensions turn every page into a blank document or inject random scripts I don't see how it can be possible for ads to somehow force themselves to be shown to the user.
>common features that are already not possible to support in Chrome
>CNAME-uncloaking
The relevant issue in the bug tracker is https://bugs.chromium.org/p/chromium/issues/detail?id=115104... and the engineer working on it does not have enough bandwidth.
>Browser launch
This is no longer a problem with declarativeNetRequest since the rules for blocking network requests is part of the browser itself.
>as written about features that are not able to be supported via Chrome's current V3 API
Some of these are self inflected by the author like only updating rules with store updates and, some of these are due to the lack of maturity of MV3, some of these are due to trade offs that are being made ecosystem wide.
That's not what we mean by static either. What we mean by static is that extensions lose the ability to dynamically analyze requests and block them using on-the-fly logic.
> Yes, they are. Due to early versions of declarativeNetRequest only having a static list of rules there are a large amount of people who believe this to still be true.
Very obviously that is not what gkbrk meant by "powerful dynamic filters", considering that gkbrk clarified that they were referring to the ability to dynamically modify requests literally one sentence later. Also very obviously that is not what gkbrk meant because gkbrk literally said in the first sentence that they weren't talking about filter list caps.
When you replied to gkbrk saying that dynamic filters were supported you were not correcting misinformation, you were saying that gkbrk was wrong -- and they're not, they're correct -- dynamic filtering of the kind they are talking about is not supported and the lack of that API meaningfully constrains adblockers (a point which, quite frankly, you don't even disagree with given that you are characterizing this as a "tradeoff").
Did you mean to reply to a different person who wasn't present?
> Considering MV3 lets extensions turn every page into a blank document or inject random scripts I don't see how it can be possible for ads to somehow force themselves to be shown to the user.
"Just inject scripts into the webpage" is a wild take from someone who is trying to claim that Manifest V3 improves privacy, security, or performance. Nobody wants addons to be putting all of their functionality into the page itself, there are myriads of problems with that approach. If you're defending Manifest V3 and you're suggesting that addons should just completely destroy the separation between extension code and page code, then... I mean, that's just not a serious suggestion.
It's good that extensions can insert code into pages, that is sometimes necessary and extensions should have that ability. I think uBO full-on uses that ability for some features. But running code that way is not the preferred way to handle functionality or request blocking, and its something that should be done sparingly and carefully.
If the upshot of Manifest V3 in Chrome is to encourage developers to start pushing tons of extension code into the page itself when they could have before kept that code separate -- then that's a failure of Manifest V3; injecting tons of extra scripts into pages that wouldn't otherwise need to be there will make performance, security, and privacy worse for end users.
> The relevant issue in the bug tracker is https://bugs.chromium.org/p/chromium/issues/detail?id=115104... and the engineer working on it does not have enough bandwidth.
So it's not supported. I'm sorry, you want me to pretend that developer concerns have been addressed because there's an open issue where the developer is overworked and does not have the bandwidth to address it? This issue has been open for 3 years! You could not show a better example of the intrinsic problems with Manifest V3 and the intrinsic problems of extension devs needing to ask Chrome devs for permission to innovate on adblockers if you tried.
This is exactly the problem of Manifest V3. There's need for adblockers to be able to innovate, and when that innovation is dependent on Chrome adding specific APIs, then the issues sit open for 3 years. Chrome is proving before V3 even becomes a requirement that they are not capable of keeping pace with adblocker innovation and that this whack-a-mole approach to adding individual capabilities into declarativeNetRequests is unworkable and bad for addons.
If Google can't keep pace with requests during a period when extensions are still able to use V2, then they're not going to magically get better at handling feature requests when every extension is using V3. If Google's approach to addon APIs is to overwork developers and spread them thin between features, it is reasonable to conclude that either Google does not see addon support as a priority worth investing sufficient developer resources into, or that Google's model for browser development is just inherently not capable of handling API requests at a reasonable speed.
---
And I think that's the broader point. This is not a healthy relationship for Chrome to have with extension developers, and Chrome is proving in real time that it doesn't have the resources or ability to support developer requests.
But to address the other specific APIs you bring up:
> This is no longer a problem with declarativeNetRequest since the rules for blocking network requests is part of the browser itself.
I'm sorry, the inability of an extension to load before the browser starts sending requests is a non-issue to you? Sure, you can pull the blocklist out so the static rules still take effect before the extension loads, but that is far from the only issue in having extensions load asynchronously from their pages.
To be fair, you are right that declarativeNetRequest makes this slightly better, but only because prior to declarativeNetRequest Chrome already exhibited this behavior so at least now there's one way to make sure that requests get filtered from browser launch, as limited as it may be. As opposed to Firefox, where this has never been a problem for either MV2 or MV3 extensions, because Firefox doesn't make the wild decision to start loading pages before the user's addons have initialized.
> Some of these are self inflected by the author like only updating rules with store updates
I actually don't think that's the main problem with most of these. Dynamic user rules, fonts, and noscript rules are blocked by rule scoping around top-level document URLs, not because of store updates. uBO Lite does avoid making network requests to update lists, but that is one paragraph out of an entire page describing limitations that make useful features difficult or impossible to build under MV3.
Additionally, if you dig into the actual details of what Gorhill is saying about updating extension lists, you'll find that this is not just about whether you can load list updates, it's also about when it happens and the fact that (as mentioned above) uBO Lite can't guarantee that it will be running when a request is made, and can't guarantee that those rule updates will take effect before the request is sent.
> and, some of these are due to the lack of maturity of MV3
Well, it's a good thing that Google is waiting for MV3 to become mature and it's a good thing that MV2 won't stop working in June, right? /s
> some of these are due to trade offs that are being made ecosystem wide.
Sure, other apps other than adblockers will also be affected. I maintain hobby Firefox extensions that are impossible to ship on Chrome for Manifest V3. Do you expect "we didn't just break adblockers, we broke lots of other addons too" to be an argument that makes anyone more sympathetic to Google?
Nvidia has significant dominance in the AI space because of their work on software and the overall platform.
With the Jetson line being the sole exception. Use it for what it’s for - a targeted build for an embedded/specific application requiring small size and low power.
The software is a mess. Support for Jetson (generally) is a far afterthought or not considered at all around projects at Nvidia and the broader ecosystem. When it is supported at all it lags behind significantly, using ancient distros (Jetpack), etc. To make matters worse the user base is so (relatively) tiny there are bugs and strange behavior everywhere.
Just don’t do it.
[1] https://www.hackster.io/pjdecarlo/llama-2-llms-w-nvidia-jets...
Maybe I will try to get a 32 GB+ LLM running one of those days.
EDIT: I cannot, I need to install ROCm to compile with it, and then install something called hipBLAS, and who knows what else.
EDIT: Oh, no, I have an nVidia GPU, AMD CPU.
What is a static filter list other than a list of filters that does not change?
>When you replied to gkbrk saying that dynamic filters were supported you were not correcting misinformation
I was saying that the filter list was dynamic and not static.
>"Just inject scripts into the webpage" is a wild take
I was pointing out the upper bounds of what was possible and not giving recommendation on how it should be done.
>So it's not supported.
That is right.
>you want me to pretend that developer concerns have been addressed
No, I want people interested in this feature to track the bug or contribute the fix themselves since it is an open source project.
>it is reasonable to conclude that either Google does not see addon support as a priority worth investing sufficient developer resources into
That is correct in my understanding. This means that the open source community needs to dedicate their own resources if they want to accelerate progress here.
>the inability of an extension to load before the browser starts sending requests is a non-issue to you
Assuming no events are dropped then I don't see it as an issue.
>doesn't make the wild decision to start loading pages before the user's addons have initialized.
It doesn't sound that wild if it speeds up launch time by not blocking on third party extensions having to load.
>it's also about when it happens and the fact that (as mentioned above) uBO Lite can't guarantee that it will be running when a request is made
This is not an issue as the extension does not need to be running for the browser to apply the rules. The extension is only needed to update the rules.
>Well, it's a good thing that Google is waiting for MV3 to become mature and it's a good thing that MV2 won't stop working in June, right? /s
Considering that the deadline keeps being pushed back, I think they recognize that it has low maturity.
A request is being made. At the time of the request, you go through a list of rules and decide if you're going to block or allow it.
Once the request reaches the filter, you are working with a static list because you cannot make decisions using the content or circumstances of that particular request. You cannot add or remove anything to your list after looking at the request.
A dynamic filter on the other hand could
- Resolve hostnames with DNS to do CNAME uncloaking
- Decode encoded requests so the website cannot send both real data and analytics/tracking data through a single URL. Being able to make dynamic decisions allows the ad blocker to inspect request headers and request bodies.
- Modify responses. The web page could be getting ads from a JSON endpoint with the rest of the page. You can use filterResponseData and change the response dynamically to strip the ads return an empty JSON list for example.
> It also reduces ad blockers to static filter lists instead of powerful dynamic filters. MV3 makes it impossible for ad-blockers to inspect requests with code and then allow/deny dynamically.
I'm assuming the best here, but it is extremely clear what you meant by a "static filter" in your comment. You meant a filter where it is "impossible for ad-blockers to inspect requests with code and then allow/deny dynamically."
It's a little frustrating to have to pretend that any of this is ambiguous.
> That is correct in my understanding. This means that the open source community needs to dedicate their own resources if they want to accelerate progress here.
OR... hear me out: we could recognize that Chrome, a browser owned by one of if not the most powerful tech companies in the entire world, does not need charity, and that it's not a community owned project (if it was, MV2 would not be getting deprecated), and that the community has no say in how the project gets updated or maintained and is fact very regularly treated with hostility by the project, and that instead of trying to fix an issue that Google probably isn't even interested in getting help with, we could instead go use and contribute to Open Source browsers that aren't transparently trying to kill adblockers.
This is wild. You went from "MV3 is fine, there's just a lot of misinformation" to "MV3 is trying, but the devs are overworked" to "nobody is allowed to complain about this unless they're giving hecking Google free labor."
Google made a decision to deprecate a working API, developers correctly pointed out that the API hobbles adblockers, Google has not fixed the API but is launching it anyway, and your answer is that the open source community needs to dedicate their own resources. This problem is not our fault, we didn't make it.
How about Google not completely rewrite an API out of the blue if they don't have the internal resources necessary to do so? Is there some hiring freeze on developers, has Google lost the ability to hire and pay people? Google creates a problem, and you're blaming the Open Source community for not fixing Google's problem for them, a problem that exists entirely of Google's own free will, because Google ignored the Open Source community that was desperately trying to stop Google from making the problem.
The Open Source community has a solution to this; don't deprecate blockingWebRequest. Like most community feedback, Google isn't interested in hearing that.
I would suggest that if Open Source developers need to contribute somewhere, they could try contributing to Firefox, a browser that has none of these problems because somehow a company that is so small its entire corporate profits would barely register in Google's spreadsheets is still better at building extension APIs than Google is.
----
> Assuming no events are dropped then I don't see it as an issue.
It's an issue because extensions sometimes like to respond to events when they happen. If the browser stopped allowing synchronous responses to click events and I said, "well as long as they eventually trigger there's no issue" I would be rightly laughed out of the room.
The issue is that a user can update filters and they have no guarantee that the filters will be updated before the requests that they mean to block are fired off. If an adblocking extension does decide to update filter rules on launch, they have no guarantee that the update will take effect before the page loads.
> It doesn't sound that wild if it speeds up launch time by not blocking on third party extensions having to load.
Not loading the extension at all would speed up launch time even more, Google should just get rid of extensions entirely /s
People use extensions as a way to increase security and to build reliable features into the browser. There are many use-cases from tab-control to feature toggles where having tabs and pages load before extensions breaks functionality. It is wild to suggest that breaking functionality in userspace is preferable to adding an extra second to a browser launch.
For many people, adblocking is a security issue. To suggest that the browser should occasionally just send requests anyway before an extension has time to update and apply rules is like suggesting that a desktop should start loading and displaying user settings before it checks the user's login password.
> This is not an issue as the extension does not need to be running for the browser to apply the rules.
See above.
----
> Considering that the deadline keeps being pushed back, I think they recognize that it has low maturity.
This is once again a pretty big shift from "MV3 doesn't stop adblockers" to "MV3 isn't ready yet and they should keep pushing it back." It's also optimistic given that Google has given no indication yet that they're going to push back the June deadline.
But most importantly, it also doesn't change anything about my point -- we are seeing that Google does not have the resources to keep pace with adblocking innovations. That's not going to change. This is not a question of MV3 becoming ready and then everything becoming fine: "ready" is a moving target, and Google is not moving fast enough to keep pace with it.
What these issues demonstrate is that the declarativeNetRequest API is poorly designed -- it is poorly designed because it greatly increases the support requirements from Google to a level where the company is unable to meet them. And when Google is unable to meet those support requirements, user security and privacy suffers as a result.
This is an intrinsic problem that is baked into the design of declarativeNetRequests, and it's not going away even if Google does manage to close a couple of bugs. Because more of these issues are coming every time that adblockers need to innovate. If an API increases support burdens to the point where the dominant browser owned by the most powerful tech company has to come crawling to the Open Source community for help triaging 3-year-old issues, then it's a bad API.
And underlying all of the criticism of MV3 is that idea -- that it is not tenable for extension developers to have to request support for every individual feature that adblockers need. A better API would be a more flexible API that allows extension developers to innovate without asking permission. blockingWebRequest is an imperfect API and it has issues, but it allows extension developers to innovate without filing support tickets for every new feature and then waiting 3 years for a response because the API is flexible enough that extension developers can use it in creative ways that the authors may not have fully anticipated or known about.
Also I only tried it on linux, AFAIK windows is a lot more difficult to get running, if it works at all...
With llama-cpp, I successfully tried various LLMs(e.g. LLAMA 13B, Mixtral etc) with very solid performance. Even for models that don't fit in VRAM completely, performance can be surprisingly solid, as long as you compile with AVX extensions. (and your CPU supports those)
Stable Diffusion via ComfyUI also works very well. However, be aware of VRAM limitations with the larger SDXL variants, especially when running a heavy desktop environment.
Regarding setup guides/links, there isn't a good centralized resource sadly, so some tinkering is needed. Unlike some of those CUDA 1-click solutions, ROCm requires more manual setup, especially for the models only unofficially supported.
Here are a couple of links that might be helpful:
https://old.reddit.com/r/LocalLLaMA/comments/18ourt4/my_setu...
https://old.reddit.com/r/StableDiffusion/comments/ww436j/how...
In general the r/localllama & r/StableDiffusion subreddits are good places to search for info.
I'd suggest checking what examples are available, see what community is doing, see if what you need had already been tried - https://www.jetson-ai-lab.com
From what I've seen, mainstream LLM libraries like VLLM, llamacpp that use CUDA under the hood tend to work out-of-the-box. And there are tutorials available: https://www.jetson-ai-lab.com/tutorial_text-generation.html. I think that TensorFlow/Pytorch are also well maintained, although I've not checked recently.
Nvidia more broadly has very impressive support for their GPUs. If you look at the support lifecycles for their Jetson hardware over time it's significantly worse. I encourage you to look at what support lifecycles have looked like, with the most "egregious" example being dropping of support for the Jetson Nano in from what I recall was within a couple of years.
Another consideration - Jetson is optimized for power efficiency/form-factor and on a per $ basis CUDA performance is terrible. The power efficiency and form-factor come at significant cost. See this discussion from one of my projects[0]. I evaluated the use of WIS on an Orin Nano that I have and it was nearly 10x slower than a GTX 1070 which is seven years old and is still supported by the latest drivers and CUDA 12 on whatever OS you want.
Nvidia knows what they're doing in terms of productization and the Jetson line should not be seen as some kind of secret hack/unlock for getting CUDA performance with gobs of RAM. In the case of LLMs I wouldn't be surprised at all if CPU beats it and at that point pickup 256GB of RAM or whatever for equivalent cost.
In the end what do I care what people use, I'm offering the perspective and experience of someone who has actually used the Jetson line for many years and frequently struggled with all of these issues and more.
[0] - https://github.com/toverainc/willow-inference-server/discuss...
About a year back, I took that very early version of AGX Xavier, that got released years ago. It wasn't even the version that was officially released. Yet I was able to refresh it to newer Ubuntu without any issues.
Wheels are often not pre-built for aarch64, yes. If you want to compile directly on Nano, disk performance is very important. Sometimes you get I/O bound.
Orin Nano being that slow in [0], it looks like you've been trying it in Aug 2023. It maybe worth re-evaluating on the latest Jetpack, it had transitioned to CUDA 12.2, TensorRT 8.6, cuDNN 8.9. I would expect that recent popularity of ASR/TTS pipelines and LLMs was not completely missed by Jetpack maintainers (there are some tutorials here - https://www.jetson-ai-lab.com ). And recently released JetPack could be optimized a lot more for these workflows.
And your project is very cool! I'd suggest sharing it and your performance numbers (!) with the maintainers of: https://developer.nvidia.com/embedded/community/jetson-proje...
I've gotten my Jetson to work well using Yocto to build my own linux distros with correct updated dependencies, libraries and updated jetpack, but it's not for the faint of heart, and that's a whole other ball of yarn. It also takes a few hours to generate a new build every time I need to update some dependency that depends on other dependencies (Yocto maintenance is a full time job in many embedded development shops - you're basically authoring your own distribution).
Treat these devices as what they are: embedded target boards for fixed industrial development (for example, to go into a robot or a car - once that design is finished, Nvidia will expect you to NEVER update any part of the system with an embedded jetson or orin system for years, until you replace the whole thing with their newest model that you buy off the shelf again).
This is standard fare in embedded and robotics space. Do not use these boards for any kind of rapidly moving software development, because it's the wrong tool for the job.
Software for Jetson boards should be viewed as firmware for these embedded/industrial devices. They get installed in a robot, MRI machine, etc with a specific bespoke application targeting what they came with and are never touched again -or- supported by some large commercial firm with the skillsets you describe.
I was as firm/absolute in my original reply as I was because anyone who thinks life with Jetson is similar to life with a discrete Nvidia GPU on x86_64 will be in for a huge shock and 95% of the time it will end up on their shelf in a year or two.
It's one thing when it's the latest random ARM SBC you bought for $50 with no vendor support, it's another thing entirely when you're spending > $600 (or $2000 as this started!!!!) on a Jetson.
Nice! I'm sorry if I seemed dismissive or even disrespectful, in my experience Jetson certainly has it's place (why I've been using them for years) but compared to "bring your distro, apt-get/.run Nvidia driver" they can be a serious shock for casual users. Then they see the performance...
> Orin Nano being that slow in [0], it looks like you've been trying it in Aug 2023. It maybe worth re-evaluating on the latest Jetpack, it had transitioned to CUDA 12.2, TensorRT 8.6, cuDNN 8.9. I would expect that recent popularity of ASR/TTS pipelines and LLMs was not completely missed by Jetpack maintainers (there are some tutorials here - https://www.jetson-ai-lab.com ). And recently released JetPack could be optimized a lot more for these workflows.
Interestingly WIS was recently bumped to CUDA 12.2, etc and the performance improvements were very marginal. WIS uses Ctranslate2 under the hood (same as faster-whisper) which offers among the best Whisper performance overall but doesn't benefit much from changes in these underlying libraries. In the end even if it somehow magically doubled performance (it doesn't and won't) that still places the latest generation ~$600 Jetson board 5x slower than an ancient yet still fully officially supported ~$100 GPU. Power and form-factor is an issue but for the voice assistant use case a Jetson board barely doing realtime with Whisper medium is unacceptable to me and the vast majority of our users. Our goal is sub one second voice command sessions from end of speech, to command execution, to TTS response and Jetson just can't provide that at any cost.
I'm glad there are community resources for Jetson platforms (which I'm aware of) but their existence underscores my point - you'll notice when perusing through there are often various hoops to jump through whereas anything else is basically "install driver, container toolkit, docker run" and it just works and works performantly. Basically CUDA x86_64 and discrete GPUs is native/expected/developed for, Jetson is almost always a bit of an edge case with rough edges (relatively) all over the place.
> And your project is very cool!
Thanks! In terms of your suggestion I certainly might but in the meantime, overall (based on my Jetson experience) as I said in that discussion I'm very reluctant to officially support the Jetson line with WIS. I'm almost certain it will blow-back on the project and cause support headaches for us while all the while providing a sub-optimal user experience.
