The Little Learner: A Straight Line to Deep Learning(mitpress.mit.edu) A new book in the same series as The Little Schemer! Here's a preview of the first two chapters: https://mitpress.ublish.com/ebook/the-little-learner-a-strai... |
The Little Learner: A Straight Line to Deep Learning(mitpress.mit.edu) A new book in the same series as The Little Schemer! Here's a preview of the first two chapters: https://mitpress.ublish.com/ebook/the-little-learner-a-strai... |
Given the wealth of information and the problem of appraising it all, I don't think it helps this book that it costs $55 and requires the reader to learn an esoteric language (for the field), when there are so many educational books and lecture series by experts in the field that you can find freely available online.
This is exactly the concern I had when I saw what it was. I'll definitely keep my eye on it and read the reviews.
So yep, must have. If you haven’t read the rest, they are really must haves imho.
EDIT: Half Life 3 confirmed: "Presents key ideas of machine learning using a small, manageable subset of the Scheme language"
I don't know what is in the book, but things are possible.
EDIT: I have not tried the OpenBLAS Racket bindings here (https://github.com/soegaard/sci) but perhaps the low level tensor and tensor ops book code could be optimized,
The Kindle version of it does not even have fixed table cell width for the layout of the conversation.
German book website buecher.de has "The Little Learner" to preorder as ePUB with Adobe DRM. I don't know, if you can buy digital books on this site outside of Germany. Also notice, that copy-protected digital books on this website are highly unusual; most of the books are with watermark instead, if at all.
I have no clue why a book with pedagogical background and custom layout, has copy-protection and no print-quality PDF version (but instead one with poor layout).
Have to think about this for a while :)
I had to think on the little schemer, but grabbed it because my ten year old said that the little typer was "interesting", and I thought he might learn something from it.
Probably should finish working through what I have before getting another, though.
https://mitpress.mit.edu/9780262561150/a-little-java-a-few-p...
I know _why wrote his books as a conversation between two foxes.
The Seasoned Schemer - More on Scheme.
The Reasoned Schemer - Minikanren in Scheme
The Little Prover - Proofs about programs
The Little Typer - Dependent types
The Little MLer - The ML language, not machine learning.
A Little Java - Java
I have the first 5 books, but never got around to finishing Little Prover or Typer (got married and moved that year, probably not the best time to try and start something like that). Definitely like the first three and what I read in those other two.
Read the preface. A very high praise coming from Guy Steele Jr and Peter Norvig.
At the same time, a warning! Scheme based introductions don't appeal to everyone. Some people feel way out of comfort zone in coding with it (which is sad because it is much simpler). Also, the utilitarian appeal is low: it won't right away see a step change in your Pytorch knowledge or whatever. The appeal of these books is to think deeply about fundamental ideas by implementing them in simplest language without too much help.
In short, YMMV. But if you have a long term view it might help you a lot than sort of currently fashionable trends. (Though I must admit that fast.ai is not just a flavor of the season resources but much better!)
I like the style of these books, and might read this one - I'm just worried that Java has changed quite a bit since then. Is it still, say, 90% OK?
The book itself takes you on a journey to reinvent the Visitor pattern, in a version of Java that predates generics. You can probably find a better explainer for the Visitor pattern in Java.
(trivia, he's also behind djvu format)
Personally, I also feel a bit sad the ecosystem has become a Python monoculture.
Lua was much nicer. Let's hope this book sparks some interest in alternative languages for ML.
I always had trouble with recursive functions when I was new to programming, and many recommended working through "the little schemer" to solve that problem. It was a tough read for me, but the investment was well worth it and it did for me what it said on the tin. I didn't have nearly as much trouble with recursion after that book, but an unfortunate side effect was developing an affinity for lisps which I haven't yet shaken.
Too little of that in this world nowadays. Too much ios and not enough arch linux.
It worked very well for me in learning functional programming and some computational theory ideas.
Worth it.
there is zero fluff, almost zero narration
the books are basically just input output pairs of "now do this, and now that happens"
they are basically a sort of brain data dump for people who can think with computer code
In particular, Scheme. If your language of choice is, say, Python, then you'll want to get a primer on Scheme before reading this particular book. Maybe by starting at the beginning of the series.
> Maybe, maybe not. But even if you use a machine learning toolkit like TensorFlow or PyTorch, what you will take away from this book is an appreciation for how the fundamentals work.
There are two books that do exactly this:
1. Deep Learning from Scratch
2. Data Science from Scratch
In these books, you implement each part of the ML/DL pipeline, from scratch, in Python.
There is also a GitHub project call Minitorch that teaches you the inner workings of a framework like PyTorch.
And then there are several other good resources for exactly this.
What he claims to have as a content is neithet new nor unique.
Do you reimplement matmul or other basics?
Do you reimplement auto-diff?
Maybe PyTorch or TensorFlow using auto-diff is a good "from scratch" basepoint, without using predefined optimizers, or modules/layers, or anything. Just using the low-level math functions, and then auto-diff.
https://github.com/zdimension/hm-infer-scheme
fantastic for research
My criticism with fast.ai, (I am part time educator), is that this approach is an information overload and poor sequencing. Their comparison with Teach whole game approach is flawed because a game of, say Football, is essentially simple. So you can say just start kicking around. But we don't teach chess this way. It is accepted that you have to spend some time upfront to learn the rules before you can play even simple game. Sure one need not learn castling or en-passant upfront. But you get the drift.
This book (looking at the preview chapters) is going to follow the lego blocks approach or bottoms up approach to build it. For me, this is correct way to teach supervised ML focussed on neural networks and deep learning. We have a problem of too many library plumbers in the ML field currently. People who can piece together library function calls without knowing why it is working. But this house of cards is not sustainable strategy to build AI based application over long term.
Long story short, the book will need patience but that patience will be worth it!
I'm a Lisper (used Common Lisp, Racket/Scheme, and Clojure) and a math graduate student, and am interested in learning more about ML and DL so that I could potentially use them in research in the future, to come up with constructions and counter-examples, so I may have some time in my hand. I have always been a fan of Dan Friedman, so I'm definitely thinking about getting this book.
I found out that fast.ai seems to be of this approach that you mentioned. Since you are an educator, if I'm interested in learning more about ML and DL (as well as the math behind and implementation), which book(s) would you recommend me to study (I have some experience with Andrew Ng's coursera course many years ago)? I did some research here and people seem to recommend different books...
Thank you!
If you have never used lisp, you need to be patient with the notation and resist the urge to balk at what is unfamiliar. I remember grumbling quite a bit when I first went through TLS, but that phase was over pretty quickly and within days I had no trouble following the code.
Are there better ways to introduce deep learning and neural networks? Maybe.. but I like "little" books and there is no better way to learn something than by building it yourself. For that reason alone I'd recommend the book sight unseen (but with the knowledge of prior little books). I do think choosing what is likely an unfamiliar language for most may be somewhat of an impediment for an already challenging topic, but scheme is a simple language and allows the author to focus on the ML concepts.
I'm highly confident you'll learn a lot, if you put in the work. For DL as a separate topic, the best resources I've found are:
- 3 blue 1 brown Neural Net playlist
- Karpathy's "The spelled-out intro to neural networks and backpropagation"Also word of advise from my experience (I'm not an expert in DL either): Think of DL field as a game of lego blocks. The ideas in this book / Karpathy's videos are the basic lego blocks: parameterised linear functions, non-linearities, auto-grad, cross-entropy / KL divergence loss and gradient descent. Then there is entire body of more complex legos discovered simply by practice (alchemy!): transformer blocks, layer norm, max-pooling etc. It is impossible to understand how the second kind were obtained from first principles. The trick is to not beat yourself up about the advanced blocks too much but just play around, read up things in papers. Just focus on fundamental blocks.
Even if you think so, Python is really an easy language, and you can easily port the code to something else.
If you already have the basic ideas about the parts of a Neural Network pipeline, you can just search google "implement part-X in Y language", and you will get well written articles/tutorials.
Many learners/practitioners of Deep Learning, when they have the big enough picture, write an NN training loop in their favorite language(s) and post it online. I remember seeing a good enough "Neural Network in APL" playlist in YT. It implements every piece in APL and gains like 90%+ accuracy in MNIST.
I also remember seeing articles in Lisp (of course!), C, Elixir, and Clojure.
I am writing one in J-lang in my free time.
So if you use a library for matrix multiplication, inverse, transpose, ... with a nice syntax, you're good to go.
Yes, in those books, you do implement matmul, auto-diff, etc.
I just wanted to point out that "from scratch" is not really well defined. There is always some arbitrary line. I just found it interesting to discuss and think about where to draw this line exactly. Obviously it's never really from scratch, i.e. you don't reinvent the hardware level, for example. Or you don't start with teaching quantum physics. So you start from somewhere.
And I was wondering whether auto-diff is maybe something which could also already be the starting point, or also matmul. Reimplementing an efficient matmul on CUDA is not easy, and might distract from the main deep learning content. But it depends also where you want to have the focus on.
Maybe I misread your comment.
Thanks for rewording.
Where do people draw the line? Where they want to.
Some people choose to go closer to the metal than the others. It's just people's choice.
Some just write stuff in Python, some write CUDA kernel for themselves (some among them had to), a friend even wrote his own compiler and programming language for Deep Learning.
So it depends on your choice. And how much deep you want to go also depends on what you want to do- i.e. you choice of career, direction of research, etc.