I still haven’t forgiven CGP Grey for changing the title to his 2017 ML video to “How AIs, like ChatGPT, learn”. The video is about genetic algorithms and has nothing to do with ChatGPT. (or with anything else in modern AI)

Knowledge is hard to attain...

This description more closely describes reinforcement learning, rather than gradient based optimization.

In fact, the entire metaphor of a confused individual being slapped or rewarded without understanding what's going on doesn't really make sense when considering gradient optimization because the gradient wrt the to loss function tells the network exactly how to change it's behavior to improve it's performance.

This last point is incredibly important to understand correctly since it contains one of the biggest assumptions about network behavior: that the optimal solution, or at least good enough for our concerns solution, can be found by slowing taking small steps in the right direction.

Neural networks are great at refining their beliefs but have a difficult time radically changing them. A better analogy might be trying to very slowly convince your uncle that climate change is real, and not a liberal conspiracy.

edit: it also does a poor job of explaining layers, which reads much more similar to how ensemble methods work (lots of little classifiers voting) than how deep networks work.

It does an excellent job of taking you from 0 to a decent understanding without dumbing down the content or abusing analogies.

1. "gradient descent" is like tuning a guitar by ear and listening to the beat frequencies ("loss") and then decreasing these by tuning a string up or down.

2. the best I can come up with for "backpropagation" is to imagine a clever device that can tirelessly optimize a Rube Goldberg machine for you but as a science, not an art.

Also reminds me of Ted Chiang warning us that we will torture innumerable AI entities long before we start having real conversations about treating them with compassion.

It seems like these sort of structures just regularly arise to help regulate the flow of information through a system.

Sure, I can explain deep learning using the analogy of mixing paint colors.

When you mix different colors of paint, you can create new colors that are a combination of the original colors. Similarly, in deep learning, you can combine different types of data, such as images, text, and audio, to create a more comprehensive understanding of the information.

In a neural network, which is the basic building block of deep learning, each layer is like a different color of paint. The input layer represents the initial data, which is like a blank canvas. As the data passes through each layer, the network extracts increasingly complex features, similar to how mixing different colors creates more complex hues.

Each layer in a neural network can be thought of as a different color that is mixed with the previous layer's color. As the colors mix, the resulting color becomes more complex and nuanced, just as the data becomes more sophisticated and detailed as it passes through the layers of the neural network.

At the end of the process, the output layer represents the final result, which is like a finished painting. The deep learning algorithm can then use this final output to make predictions or classifications based on the input data.

So, in essence, deep learning is like mixing paint colors to create new and more complex hues, but instead of colors, the mixtures are made up of data and the neural network layers are the different colors being mixed.

This needs expanding: it's the sort of thing that's easy for a programmer to say, but few non-{programmer,mathematically trained person} are going to see that an RGB value has 3 parts and so a collection of RGB values could be sliced into 3 sheets.

Cute.

1 - We want to model data, representative of some system, through functions.

2 - Virtually any function can be expressed by a n-th order polynomial.

3 - We wish to learn the parameters, the coefficients, of such polynomials.

4 - Neural networks allow us to brute-force test candidate values of such parameters (finding optimal candidate parameters such that error between expected and actual values of our dataset are minimized)

Whereas prior, methods (e.g. PCA) could only model linear relationships, neural networks allowed us to begin modeling non-linear ones.

I think this book is a shining example of that philosophy: https://www.buildyourownlisp.com/. In the book, you implement an extremely bare-bones version of lisp, but it has been invaluable in my career. I found I was able to understand nuanced language features much more quickly because I have a clear model of how programming languages are decomposed into their components.

One of the most interesting things about this topic is that the fundamental concepts and implementations are all really simple. It's the fact that it actually works that's mind boggling. In any case, the above is not a months like affair - but like one week of dedicated work.

I do think that maybe it feels inaccessible because we transition from discrete concepts easily digestible by CS grads into some complicated math with very terse mathematic notation, yet the math might not be as hard if presented in a way that doesn't scare away programmers.

I think it's actually worse than that - somebody who doesn't know actually realizes that he doesn't know, but somebody who _thinks_ he understands through analogies and metaphors will confidently come to the incorrect conclusion and then argue with somebody who actually does understand the topic - often managing to convince innocent bystanders because his reasoning is easier to grasp and the so-called expert seems to be getting more and more flustered (as he tries to explain why the analogy is actually correct, but oversimplified).

Initialization: The weights and biases of the neural network are like the starting colors of the paint mixture. Just like how you need to start with some base colors to mix new ones, the neural network needs some initial weights and biases to begin the training process.

Forward Pass: During the forward pass, the input data is fed into the neural network, and the network produces an output. This output is like the mixed color of paint that results from combining different colors.

Backward Pass: During the backward pass, the error between the predicted output and the actual output is calculated using a loss function. This error is like a measure of how close the mixed color is to the desired color. Just like how you would adjust the mixture of colors to get the desired hue, the neural network adjusts the weights and biases of the neurons to minimize the error between the predicted output and the actual output.

Repeat: Steps 2 and 3 are repeated many times with different input data points, and the weights and biases of the neural network are updated after each iteration. This process is like adding more colors to the mixture and adjusting the amounts until the desired color is achieved.

Validation: After several epochs, the neural network is evaluated using a validation set of data to determine if it's overfitting or underfitting the training data. This step is like checking the mixture of colors to see if it matches the desired color under different lighting conditions.

Testing: Once the neural network has been trained and evaluated, it can be used to make predictions on new data. This step is like using the paint mixture to create a painting.

In summary, just like mixing paint colors to achieve a desired hue, the neural network adjusts its weights and biases to minimize the error between the predicted output and the actual output until it accurately predicts the output for new data.

  y = a + bx + cx^2 + dx^3 + ...

So fitting polynomials is easy. The problem is that it's not that useful. Deep learning has to solve much harder problems to get to a useful model.

As far as my understanding goes, you can represent practically any function as layers of linear transformations followed by non-linear functions (e.g. `ReLU(x) = max(0, x)`). It's this sprinkling of non-linearity that allows the networks to be able to model complex functions.

However, from my perspective, the secret sauce is (1) composability and (2) differentiability. These enable the backpropagation process (which is just "the chain rule" from calculus) and this is what allows these massive mathematical expressions to learn parameters (weights and biases) that perform well.

- It requires some mathematical understanding so will exclude some part of the non-technical audience

- It is the incorrect analogy. Non-linearities in neural networks have nothing to do with polynomials. In fact, polynomial regression is a type of linear regression, and for the most part, it sucks.

Also, as someone mentioned, all the “serious” alternative ML methods prior to the deep learning revolution allow modeling non linearities (even if just through modification of linear regressions, like polynomial regression).

But there are many things that are not functions. Like circles. And they tend to crop up a lot in the real world, no pun intended.

Prior methods also allowed modelling of non-linear relationships, eg. Random Forests.