Local Minima, Saddle Points, and Plateaus(theorangeduck.com) |
Local Minima, Saddle Points, and Plateaus(theorangeduck.com) |
For example, I've heard elsewhere that great programmers possess the skill of being able to decompose large changes into smaller commits, making the code no worse as they go. Having the ability to see the small changes that get you where you want to go corresponds to finding the direction of change that gets you past the saddle point.
Random restart is a standard tool in the kit of stochastic gradient descent. But if local minima are rare, it may be overused. You don't have to rewrite your code if you can see the refactors and gradual improvements that get you where you need to go.
In real high-dimensional spaces, like life, exploring away from local minima and plateaus ("plateaux"?) is expensive. Figuring in the expense of exploration distorts the reward function intriguingly.
The tricky part in real life is figuring out the dimensionality and then reasoning about how changes in the gradient of one dimension appear to affect changes in the gradient of another.
What are local minima, in relation to the real life? Are those the "places", mentally speaking, where one feels like they're failing?
What, exactly, are saddle points in this context?
They are places where you're doing OK but can't seem to make progress to "better than OK" by moving a little away from them (e.g. for many steps away from that situation, things are getting worse -- you need to travel a lot to get out of the "pit" you're in).
Consider someone obese. They feel worse when doing exercize/diet (compared to e.g. munching a pizza), but the feelings they have are not the best they could have related to their weight situation -- they'd feel best if they could persevere and eventually feel fit/healthy.
Saddle points are those where you can't seem to make easy progress on one direction (one aspect, e.g. how you feel about your health) but you can on another (e.g. how much money you make).
It's a little more involved (because if you're obese you're always best to lose, whereas a function can try both increasing and decreasing values to get out of a local minimum) but that's the gist I think.
I'm still having trouble visualizing it, though. I reckon it's the kind of an idea where, if you don't give me a clear-enough image, I'm stuck no matter how well you pick your words others.
Not to put any kind of pressure on you, of course. I'm grateful for the explanation. It's my brain that does its tricks.