Is there a concept of safety trademark in ML black box models? How would someone trust a black box model that it is not biased or things like that specially when it comes to its deployment in mission-critical applications such as health care? How do we manage the tradeoff between accuracy and intelligibility? |
For instance, at the convenience store near me there is a sign that says "To buy alcohol you must (A) be 21 years of age and (B) not visibly intoxicated"
The first of those is easily addressed as a rule, the latter is a statistical kind of thing -- different raters are going to disagree whether certain people are "visibly intoxicated" and (in real life) it is contextual. It is one thing to send somebody to drive home drunk, it is another if you are not.
An interpretable model might tell you that it learned it is OK to serve people that are 20.95 years of age, but that is for chumps. You should just punch in it is 21 from the beginning.
Similarly, a model that is biased against black people might not have a reference to "X is black" but rather might learn to discriminate based on geography or other characteristics. On the other hand, in health care, there are some cases you do want to take race into account. Blacks react differently to drugs for heart failure, while Naltrexone seems to be highly effective for alcoholism in Asians, moderately effective in Whites, and barely effective in Blacks.