A pure NumPy implementation of Mamba(github.com) |
A pure NumPy implementation of Mamba(github.com) |
Why are you entitled to have every single GitHub repo explained, tailored to your individual knowledge?
Many other people understood exactly what this is.
Maybe the submitter could add a comment on HN with an explanation, but the author owes you nothing.
I'm not going to name names because I don't want to throw shade at what are essentially good or even great projects but, as a recent example, I encountered a library in our codebase the other day where I simply didn't get what the point was, and the corresponding project page and documentation - whilst really detailed in some ways - didn't help. In the end I asked ChatGPT and also found a series of video tutorials that I watched at 1.75x speed to understand it.
It was worth doing that because the thing is already used in our codebase, and it's important in that context for me to understand why and the value it adds.
But if I run across something reading an article or whatever, and it mentions some library or project in passing, I'm semi-regularly left a bit baffled as to what and why and I probably don't have the time to go digging. Nowadays I probably would ask ChatGPT for a short summary because it's so convenient and it's often quicker than Googling, and maybe I'll start submitting PRs against readme.md files to add those summaries (with a bit of editing) to the beginning of them.
"""Simple, minimal implementation of Mamba in one file of Numpy adapted from (1) and inspired from (2).
Suggest reading the following before/while reading the code:
[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Albert Gu and Tri Dao)
https://arxiv.org/abs/2312.00752
[2] The Annotated S4 (Sasha Rush and Sidd Karamcheti)
https://srush.github.io/annotated-s4Even that first line you posted is unhelpfully circular, defining mamba as an implementation of mamba.
Call me old fashioned, but a best practice read me should concisely provide: what the thing is, and why it is, aka the problem it solves. (And not with circular definition.)
Which is the purpose of these doc comments.
If you have the time to gripe on HN, you have the time to click on the link and do some reading. The "Usage" section in the link above is enough to help one disambiguate; if not, then there's always the doc comment.
at this moment, in this time, if you see Mamba, either you know or you don't
I’m familiar with mamba, the conda like thing in python, but a numpy implementation of that makes no sense.
A numpy program will work tomorrow.
ALL of the machine learning frameworks have incredible churn. I have code from two years ago which I can't make work reliably anymore -- not for lack of trying -- due to all the breaking changes and dependency issues. There are systems where each model runs in its own docker, with its own set of pinned library versions (many with security issues now). It's a complete and utter trainwreck. Don't even get me started on CUDA versions (or Intel/AMD compatibility, or older / deprecated GPUs).
For comparison, virtually all of my non-machine-learning Python code from the year 2010 all still works in 2024.
There are good reasons for this. Those breaking changes aren't just for fun; they're representative of the very rapid rate of progress in a rapidly-changing field. In contrast, Python or numpy are mature systems. Still, it makes many machine learning models insanely expensive to maintain in production environments.
If you're a machine learning researcher, it's fine, but if you have a system like an ecommerce web site or a compiler or whatever, where you'd like to be able to plug in a task-specific ML model, your downpayment is a weekend of hacking to make it work, but your ongoing rent of maintenance costs might be a few weeks each year for each model you use. I have a million places I'd love to plug in a little bit of ML. However, I'm very judicious with it, not because it's hard to do, but because it's expensive to maintain.
A pure Python + numpy implementation would mean that you can avoid all of that.
For me pure X means: to use this, all you have to install is X.
"Yes, the comment you mentioned is fair and reflects a common perspective in the programming and data science communities regarding the usage of "pure" implementations. When someone refers to a "pure X implementation," the typical expectation is that the implementation will rely solely on the functionalities of library X, without introducing dependencies from other libraries or frameworks."
TIL.
So it’s just numpy and einops, which is pretty cool. I guess you could probably rewrite all the einops stuff in pure numpy if you want to trade readable code for eliminating the einops dependency
Edit: found the torch import, but it’s just for a single torch.load to deserialize some data
Torch is quite heavy though, isn't it? All for that one deserialization call?
Proponents of it usually highlight it's inference performance, in particular linear scaling with the input tokens.
It's an LLM.
I use Mamba for instance to build surrogate models of physics-based building energy models which can generate 15-min interval data for heating, cooling, electricity, and hot water usage of any building in the US from building characteristics, weather timeseries, and occupancy time series.
It has many other non-NLP applications.
I also assumed that "a pure NumPy implementation" meant that it was built purely with numpy, which it isn't smh
https://huggingface.co/learn/nlp-course/chapter1/1
And there's a pile of awesome. It feels pretty lame compared to GPT4o, ChatGPT, or even GPT3, but it's still super-useful a lot of the time, and not too resource-intensive.
(Disclaimer: That's the original Hugging Face course and is politely structured to work on reasonable machines too. They have other courses which require moderate GPU, and plenty of models which require crazy hardware.)
I can also share my master’s thesis which is similar but using CNN layers rather than Mamba and only for monthly predictions rather than 15-min interval data. There are some other architectural differences but the basics are the same. That work is also globally robust.
As you can imagine, the current work I am doing at a much higher resolution is a big step up, and Mamba so far is working out great.
I'm currently learning about machine learning and digital twin but don't really where to start