This is (roughly) the same course but the lectures are from 2018. The audio/video quality is quite a a bit better:

https://youtube.com/playlist?list=PLqYmG7hTraZBKeNJ-JE_eyJHZ...

When it comes to education online I prefer these, where they publish actual university course lectures and assignments than what you can find on Coursera and alike. There is still a big gap in the quality and depth.

That's one of these moments in life where you see tech and you know it will change the world, but don't see the problem yet.

I used to be a bit more excited about RL. I mean, it's still definitely something I have to learn, but one aspect of it _seems_ lacking to me and is messing with my motivation to learn it. I'm sure someone will happily explain all the ways I am ignorant.

It seems like there is a lot of emphasis on "direct RL" or whatever where they don't even really think about the model much, but it's I guess often inside of the policy or something?

But it seems to me as someone who has just started learning about robotics, that I absolutely need to first verify that I have an accurate model of the environment which I can inspect. It seems like a lot of RL approaches might not even be able to supply that.

I mean what I am stuck on as far as creating a robot (or virtual robot) is having a vision system that does all of the hard things I want. I feel like if I can detect edges and surfaces and shapes in 3D, parts of objects and objects, with orientation etc., and in a way I can display and manipulate it, that level of understanding will give me a firm base to build the rest of learning and planning on.

I know all of that is very hard. It seems like they must have tried that for awhile and then kind of gave up to head down the current direction of RL? Or just decided it wasn't important. I still think it's important.

I watched this course and David Silver is a great lecturer, better than anybody else I’ve seen actually. I hope he does more publicly viewable courses in the future.

On one hand you're right, methods like Q-learning are model-free and do not necessarily encode much about state dynamics. The Q-function is a feature (function) of state and while ita may not say much about the model, it does encode the most important aspect of the model in terms of solving the task. Namely, it predicts the accumulated reward conditional on next actions actions. That makes it a somewhat narrow representation of state on its own. But, if you consider an environment that has many reward signals, and you learn Q functions for each, this ensemble of Q functions can consitute a rich representation of state. Depending on what the reward functions are, the associated Q functions may be sufficient to construct a full model. so I guess my point is that the learned quantities in RL encode key aspects of state, and when you expand beyond the single task/single reward RL setting the lines between value and model can become blurred.

Asking from a layman's perspective:

I've read a bit about genetic algorithms or evolutionary computation at some point. Apparently it achieves good results as it can find discrete solutions for complex, well defined problems.

Reinforcement learning is something I know even less about. But from what I gathered it is also most successful in well defined problems and systems (such as games).

So my question is: How do they relate? Is there overlap and what are the most significant conceptual differences?

I'm taking an adaptation of this class. My professor is simply reusing Silver's slides, so I'm watching the original lecture instead. Highly recommend!

Watched the course and it's great, probably the best intro to RL. Multiple watches needed as the subject is very deep.

David Silver is awesome! Loved him in the Alpha Go documentary as well

Just watched his interview with Lex, inspiring engineer

I did the course during my RC batch this year. Highly recommended if you're looking to learn RL.

Hrm so are markov decision processes RL?

While RL is about the problem it's also about the solution. Problems/Environments are formulated in a way where methods can be applied easily (i.e. Markov decision process) and thus the solutions are directly connected to the way the problem is formulated.

Deep learning is used for function approximation and is not in contrast with evolutionary computation. You can train a neutral network policy (mapping states to actions) with an evolutionary algorithm, but most of the success has come from methods that utilize the internal structure of the problem as mentioned earlier and evolutionary algorithms do not, which is what makes these optimization strategies both weak and powerful.

That makes sense thank you! It's easy to conflate the concept/problem with the tool as an outsider.