What I always wondered about is that this generally assumes that classes are assigned with an equal (human level) error probability. While this is certainly the case in heavily curated example dataset, many real world scenarios only consist of a considerably labeled positive set while the negative set is often drawn randomly from the background. Is there anything on how this can be taken into account (Besides weighting, obviously)?

"Cross-entropy is defined as the difference between the following two probability distributions"

Huh? No this is a mathematically imprecise statement (and not correct either). Most explanations use references to information theory, where a perfect knowledge of the desired probability distribution leads to a perfect allocation of bits in a binary encoding. The entropy is the expected number of bits when this allocation is done using the incorrect distribution, and obviously the goal is to minimize this, hence why it is suitable for use as a loss function.

Aside: I'm always surprised how few people notice that e.g. "cos" is rendered differently to "\cos" in TeX; for a discipline largely characterised by attention to detail, almost no programmers seem to notice.

Please do not put "Show HN" on blog posts. This is in the rules: https://news.ycombinator.com/showhn.html.

Why link to the disqus thread?

Nice read! Looking forward to more blogs.