Great read on the fragility of the "process" of doing science - the "right" way to do science shifts as knowledge shifts, but both of these are opaque leading to numerous disagreements on both. Doing science, and trying to scientifically improve /how/ we do science, are both very much akin to wandering around in the dark. It's a bit unfortunate these ideas aren't taught during a PhD.

The contrast between qualitative and quantitative research reminds me of how some old-school biologists are suspicious of p-values -- in their mind, the best kinds of biological research, and the best biological results, don't need p-values to "prove" their reality -- they are plain to see, qualitatively. Of course, modern-day statisticians or computational biologists can find this perspective a bit maddening.

from the article "A pattern we see in some interpretability and interpretability-adjacent ML papers is defining some metric which is claimed to correspond to some property of interest, and then very rigorously measuring this metric. We see this as a kind of Cargo-Cult Science."

During the first year of my PhD I tried building metrics for interpretability of Graph Neural Networks… kind of obsessed about having “measurable” properties because I thought that was the “science way”.

I failed every single try, and after some time I began appreciating qualitative indicators.

Also, Chris Olah’s articles are lovely (Anthropic and also distill.pub ones)

[deleted]