Differentiable Programming – A Simple Introduction(assemblyai.com) |
Differentiable Programming – A Simple Introduction(assemblyai.com) |
[1]: https://dl.acm.org/doi/10.1145/3236765
[2]: https://news.ycombinator.com/item?id=18306860
[3]: Talk at Microsoft Research: https://www.youtube.com/watch?v=ne99laPUxN4 Other presentations listed here: https://github.com/conal/essence-of-ad
Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form".
(Disclosure: I'm the other of another paper on AD.)
Also, just curious, what are your studies in?
Sometimes there are other better ways to describe "how does changing x affect y". Derivatives are powerful but they are not the only possible description of such relationships.
I'm very excited for what other things future "compilers" will be able to do to programs besides differentiation. That's just the beginning.
I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place?
I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? Could it be that your neural network ends up as an approximation of the Taylor expansion of your inverse function?
Computing the derivatives is not very difficult with e.g. Jax, but ... you get back to the memory issue. The Hessian is a square matrix, so in Deep Learning, if we have a million of parameters, then the Hessian is a 1 trillion square matrix...
I understand perfectly what it means for a neural network, but how about more abstract things.
Im not even sure as currently presented, the implementation actually means something. What is the derivative of a function like List, or Sort or GroupBy etc? These articles all assume that somehow it just looks like derivative from calculus somehow.
Approximating everything as some non smooth real function doesn’t seem entirely morally correct. A program is more discrete or synthetic. I think it should be a bit more algebraic flavoured, like differentials over a ring.
The Pontryagin maximum (or minumum, if you define your objective function with a minus sign) principle is the essence to that approach to optimal control.
So it's not really about NNs vs polynomials/fourier/Chebyshev, etc., but rather what is the right thing to use at a given time. In the universal differential equations paper (https://arxiv.org/abs/2001.04385), we demonstrate that some examples of automatic equation discovery are faster using Fourier series than NNs (specifically there the discovery of a semilinear partial differential equation). That doesn't say that NNs are a bad way to do all of this, it just means that they are one trainable object that is good for high dimensional approximations, but libraries should allow you to easily move between classical basis functions and NNs to best achieve the highest performance.
If you're talking about the specific cannon problem, you don't need to do any learning at all you can just solve the kinematics, so in some sense you could ask why you're using any approximation function,
let b = f a in g (a, b)
g ∘ (id × f) ∘ dup
https://www.assemblyai.com/blog/why-you-should-or-shouldnt-b...
Would definitely be interested in an article like that if you decide to write it
Unfortunately the CS field is only just rediscovering control theory while it has been a staple of EE for years. However, there haven't been many new innovations in the field until recently when ML became the new hottest thing.
For the ones interested there is a book that discusses both: 'Reinforcement Learning and Optimal Control', by Dimitri P. Bertsekas. It covers exact and approximate Dynamic Programming, finite and infinite horizon problems, deterministic and stochastic models, model-based and model-free optimization.
Aside from this book, Ben Recht has some interesting blog about Optimal Control and Reinforcement learning: http://www.argmin.net/2018/06/25/outsider-rl
It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine telling us about optimization techniques that control systems practitioners would know more about than applied mathematicians because they were developed specifically for the field, can't remember what the techniques were though ...
https://news.ycombinator.com/item?id=8417882
> It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing.
I agree, apparently Bellman's reasoning for calling dynamic programming what it is was because he needed grant funding during the Cold War days and was advised to give his mathematical theories a more "interesting" name.
https://en.m.wikipedia.org/wiki/Dynamic_programming#History
The generalised form of the Bellman Equation (co-formulated by Kalman of the Kalman filters fame) to control theory and EE is in some ways what the Maximum Likelihood function is to ML.
https://en.m.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%8...
That hilarious and sadly insightful. I remember thinking "what the hell is so 'dynamic' about this?" the first time I learned about dynamic programming. Although "memoitative programming" sounds pretty fancy too, lol
