So a user who watches erotic videos might be recommended videos of women who become conspicuously younger, and then women who pose provocatively in children’s clothes. Eventually, some users might be presented with videos of girls as young as 5 or 6 wearing bathing suits, or getting dressed or doing a split."
Holy shit, just burn it down at this point. This is what happens when you just mindlessly optimize.
Retracing the browser history, it went something like: historic recordings of civil war/partisan songs -> later recordings by popular musicians of the most vanilla/famous ones -> current renditions of the old songs by extremist groups -> crazy political propaganda from those extremist groups.
As you say, it was a first hand experience of what unsupervised, mindless optimization looks like. Ethics are not factored in the merit function, so they just don't exist for the machine.