While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1].

One would think this is cognitively dissonant enough, but it gets worse:

This article, with the thesis that good arguments are more important than data, is based on, well, a good argument – not much data. On the other hand, the work by Meehl et al. claiming pretty much the opposite, is based on, well, a lot of data, and maybe not much intuitive reasoning. (There's some, yes, but the main thrust of why I believe it is that variants of the experiment have been replicated reliably.)

I don't know what to believe. Fortunately, as I've grown older, I've become more comfortable with holding completely dissonant opinions in my head at the same time.

----

Edit a few minutes later: This actually prompted me to refresh on the subject. It might be the case that Meehl is actually making the same argument as this article, only it gets distorted when repeated. Some things are reliably measurable; for those things be data-driven. Other things not so much, then use your expertise.

----

[1]: Here's just one relatively early example: http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...

> Are you prepared to do some very very fancy statistics?

I'd extend this with "... while understanding what you're doing?"

I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.

I have experienced this first hand, so this article resonates a lot with me.

I worked with a manager who prioritized work which was easily measurable, so he could report the good numbers to leadership and get career points out of this. Unfortunately the project we took on was a demanding and technically challenging problem, and in almost a year of work of a team of engineers we made barely any real progress or made any actual difference, but the numbers were great and people were satisfied during presentations. I ended up feeling completely disconnected from my job and losing all motivation to work there.

> I originally claimed that data-driven culture leads bad arguments involving data to be favored over good arguments that don’t

This is symptomatic of the deeper problem of thinking in terms of bumper stickers and slogans, instead of thinking from first principles. When it afflicts educated people, usually you hear slogans like "an anecdote is not data", or "that's the slippery slope fallacy". Instead of grappling with noisy reality, they have sharp cognitive categories with firm boundaries between concepts, then they try to squeeze things into these categories in order to make cognition easier because the relations between the categories are already understood. This gives them the illusion of rigorous and clear thought.

This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy.

In this case the topic of value is the often fraught relationship between empiricism and rationalism, and the impacts each have on the scientific process, research, education, and how we go about understanding the world.

To operate with one with a complete absence of the other is to expose yourself to huge, often fundamental gaps in your thinking, your arguments, and your plans. This is what the author is ultimately getting at from the direction of the empirical: data, in the form of a large collection of discrete observations, can be used to justify a sea of mutually exclusive claims that may or may not be in accordance with reality, and that's to say nothing about the quality of the data itself.

I often experience the inverse: people come up with hypotheses and theories that should see expressions in observable data - but no-one bothers to look and instead everyone argues around logical constructs etc.

This reminds me a lot of the discussion of the scientific method by Karl Popper, and David Deutsch who was very influenced by Popper. "Being data-driven" sounds very empirical. Just look at the data, and see what you find in it.

But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data. Popper in Conjectures and Refutations:

> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems.

Deutsch, in The Beginning of Infinity, emphasizes the importance of conjecture, and the role of observation as refuting or criticising those conjectures:

> Where does [knowledge] come from? Empiricism said that we derive it from sensory experience. This is false. The real source of our theories is conjecture, and the real source of our knowledge is conjecture alternating with criticism. We create theories by rearranging, combining, altering and adding to existing ideas with the intention of improving upon them. The role of experiment and observation is to choose between existing theories, not to be the source of new ones. We interpret experiences through explanatory theories, but true explanations are not obvious.

To bring this back to the subject of the article, I might suggest that it's possible to be "data driven" without a sound explanation or theory that the data is either interpreted through, or used to criticise. Or maybe such theories do exist, but are left implicit.

I won’t belabor the point because others have already made it: this article assumes there is some way to sort through good and bad arguments in the absence of data - a pretty big leap. The reality is all of our arguments are appealing to some sort of data (eg previous experience), it’s just that it doesn’t always fit in a neat definition of data.

Obligatory: https://en.m.wikipedia.org/wiki/All_models_are_wrong

The related problem that I see actually more often is the "you don't have big data" problem.

You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis. You see datasets of a few gigabytes being processed with spark when SQL would be fine. You see ML techniques being thrown at questions that could be answered simply and reliably with basic statistical tests.

Especially in the B2C space a lot of companies, departments, products don't actually have a lot of customers and certainly not many decision makers. The N number is always going to be low. You can just talk to people. Let's say you are doing pretty well and running a SaS with 1000 corporate customers paying a million each - that's a billion dollar revenue - you can just talk to them. Certainly you can just talk to every single person who signs the cheque and those are the only people that matter.

And which is easier - putting together a thorough suite of A/B tests or getting some real customers to use your app on video and talking to them about what they are finding annoying, useful, missing? I see less people do that than you'd think.

To use Clayton Christensen’s theory of innovation here, to sustain innovation, businesses tend to be purely data driven. They continue to grow and make more money based on choices made with pure data.

For disruptive innovation however, there needs to be an “argument” or opinion to help drive that data based on the industry trends. Companies then take a risk of delivering something new and good enough to the market. Also known as disruptive innovation.

This has shifted the idea of being data-driven to being one of “data-inspired”.

Anyone can make the same dataset fall into their favor. That’s the problem with being purely data-driven. Another way to think of it in the US especially is that our two party system makes wildly different conclusions from the same data. What’s preventing businesses from doing the same?

To the author... I'd suggest a rewrite of what you're trying to communicate because your usage of "good-argument-driven" is a textbook example of Begging The Question: https://en.wikipedia.org/wiki/Begging_the_question

For discussion's sake, let's go along with excluding data/metrics/science in pushing for arguments. In this framework, what exactly is a "good" argument based on? Gut feel? Opinion?

There was a famous quote by Jim Barksdale, the former CEO of Netscape: "If we have data, let’s look at the data. If all we have are opinions, let’s go with mine."

(So the tie-breaker in competing arguments in that case was "hierarchy-of-arguer-driven".)

So Jane and Bob disagree on the next action to take. Jane thinks her argument is a "good argument" but has no data. But Bob thinks he has a "good argument" but no data.

How does this thread's blog post help resolve the above scenario? (Blog's answer: you're driven by the one that has the good argument.) ... which is circular.

This is not what the data shows

https://www.google.com/search?q=data+driven+companies+more+p...

Any good-argument-driven based argument you attempt to make is almost always based on political motivating factors, rather on what is good for the business.

Intuition driven decisions work when the market is behaving normally, however, are generally too slow in a fast changing market like we have been since the start of COVID.

I think the problem is that people chronically underestimate how hard good science is.

Professors get this wrong all the time, despite being some of the smartest people we have around, despite decades of experience and education, despite a career and reputation on the line, and despite a system of peer review to catch mistakes before they get published.

Designing experiments is really difficult.

Interpreting experiments is difficult and unintuitive.

Statistics is difficult. You can't just look at whether the number went up. You need to have a deep understanding of significance, power and effect size, you should probably be doing ANOVA or some such.

> A weak argument founded on poorly-interpreted data is not better than a well-reasoned argument founded on observation and theory.

So a good argument is founded on...good data and good understanding of data?

The article more seriously makes the mistake of begging the question: it presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. But how do you know good arguments from bad arguments in the first place? What makes a good argument if not empirical data?

In Range, David Epstein talks about about NASA and some of their disasters, like the explosion of Challenger. NASA is the entirely encased in specialized knowledge, and has a completely data-driven mindset, with no room for logic. If you can't prove it with data, they wouldn't even consider it. He explains that, “Reason without numbers was not accepted. In the face of an unfamiliar challenge, NASA managers failed to drop their familiar tools... The Challenger managers made mistakes of conformity. They stuck to the usual tools in the face of an unusual challenge.” Even though the mistake that led to the Challenger disaster could have been caught, it was the uniformity of thinking that lead to an organizational blind spot, and that uniformity was to be too focused on data-driven arguments.

There is a famous call prior to the disaster on which engineers had raised the concerns but it was based on intuition and a few cherry picked samples, not a full set of data, and this was the night before the launch. Because of the lack of data, they went ahead with it and we all know the tragedy that ensued. Moreover, other engineers who agreed that there was an issue didn't speak up, because they too lacked the data, and knew that management wouldn't care.

One of the big reasons why data driven approaches are so seductive is, it's very difficult in the moment to distinguish between a good argument and a well crafted rationalization.

1. If there are no good arguments in the collective - there's no retrospective and it's primarily a management and psychological issue. No one is able to fully self-reflect and it breaks the existing delegation / escalation chains, respectively.

2. If there are no viable data sources, when it can be proven that there's a correlation with an actual business processes, - it's a management problem. People Can't establish viable metrics, once again, mostly due to 1.

This is something any company of any size and any budget can struggle with due to lack of XP and the usual collective XP-accumulation / knowledge sharing deficiency. You can't self-reflect onto something you haven't learned about, yet. And due to 1 this is a closed loop because lack of XP can't be escalated accordingly, most of the time it's also a Workplace Deviance factor.

3. Practically, it ends up in a bouquet of Workplace Deviance because no one in the end will be willing to take the blame and actual responsibility to fix anything.

Any Problem vs Solution type of culture will worsen things a lot i.e. "All the blame and no Compassion". Companies are usually forced to adopt some Teal stuff in the end, maybe for really no other good reason, but just to keep on growing.

The idea of hiring HR that can "work by the booK" and actually build up a personal profile of how anyone could fit into all this mess is impossible by definition - due to Employee Silence and broken retro no one will be willing to expose all the shit that is happening, in the first place... So, most of the time I see Kitchen Sink companies with volatile outcomes where there really no one who could even be able to listen to any arguments, in the first place.

Google's internal ML-driven productivity metrics became a meme already for all the reasons described above. You can't reason with Toxic and Inadequate people.

Also Asana claim that Social Loafing is a myth and everything else is a retro deficiency really wrong - retro can prevent and display certain glorious occasions, but it's not a root cause of any psychological effect by definition.

Good article. When your only tool is a hammer, every problem looks like a thumb.

While we're at it: I've actually been in scrums where the "burndown rate" was analyzed as if it was actually A Thing. It is not A Thing.

Key idea is the "data maturity" of the topic under discussion.

Where there is data, you should use it and be smart about it.

For a lot of big decisions, especially in companies doing something new, there is no good data at first. You have to reason about it based on experience and analogy.

Then, once you commit to a path, you can start gathering data to see if your hypothesis was correct. The further you go, the more you can rely on data, assuming you know how to think about it.

Discussions about being data-driven that don't take into account the "data maturity" of the situation are nonsensical.

Being "data driven" when you're considering something radically new is either delusional or a cop out.

Ignoring data when it could correct your biases is either lazy or wrong or both.

And finally, lots of people who claim to be "data driven" are not smart about data. To paraphrase Wilde, "data is rarely pure and never simple." It doesn't just reveal truths you can treat as dogma. It's ambiguous and takes a lot of work to interpret. A lot of "data driven" teams aren't doing that work.

I'm surprised there's no mention of Goodhart's Law [0].

Even if the metric is "well understood and free from human/social factors", once you start using it as a target that will no longer be the case.

[0]: https://en.wikipedia.org/wiki/Goodhart%27s_law

I couldn't agree with this more. I feel like the author took some of the arguments straight from my brain—I'm exhausted by pseudoscientific "data-driven" arguments.

From my experience, most of these try to distill an incredibly complex problem space down to a one-dimensional black and white decision. But the real world doesn't work like that–it's full of grey area, and things we can't effectively measure. If you're trying to slice and dice data down to a happy one-dimensional decision point, you're often missing or ignoring important detail.

At work, I'm far more happy with postmortems with general, open "good/bad" lists of after the fact feedback, that we use to consider how we prioritize and design what comes next.

Being data-driven for the sake is being data-driven is indeed becoming an issue. The resources spent measuring and analysing data are overwhelmingly larger than they should in most cases. Cohorts of "data scientists" and "managers" dive head on into data without much (if any!) first-principles thinking. People tend to replicate metrics without much thought into their relevance to the specific situation. Thinking properly is a very hard skill to acquire (the hardest?), and most do everything they can to avoid it.

"What you measure affects what you do. If you don't measure the right thing, you don't do the right thing." -- Joseph Stiglitz

Great article, but I think it somewhat misunderstands the impetus for the concept. "Data has its place" sounds obvious precisely because "data-driven" has been such a successful concept. The alternative perspective, which used to be very common in our industry and still pops up from time to time, is that metrics are something you write for debugging and business decisions are made by gut feeling or abstract philosophical analysis. (Most software companies had to make decisions this way in the pre-cloud era, because it wasn't usually feasible to collect usage metrics.)

The hidden assumption here is that things go well if and only if (you think) you understand all the factors that influence your metrics, can do experiments and are prepared to use fancy statistics.

Which I reckon is a bit iffy. Special relativity was thought out well before any experiments to test it were feasible, and if understanding everything that influences your metric is a prerequisite then you can blame all failures on insufficient understanding without having any way of knowing when you have enough understanding.

I’ll probably be buried in all these comments, but my position is that data is only as good as how it is collected. Sloppy data collection gives rise to sloppy conclusion through unknown biases.

The key is to understand the ‘data generation process’ so you can identify biases. My experience suggests that doing so side-step some common pitfalls.

I recommend reach out for ‘The Book Of Why’ by Judea Pearl. He includes many real life examples that’s surprisingly applicable to modern data science.

Most argumentation we do as on questions that are worth debating aren't based purely on deductive reasoning, but more on informal reasoning and heuristics with limited evidence.

Toulmin identifies the three essential parts of any argument as

    - the claim
    - the data (also called grounds or evidence), which support the claim
    - the warrant.

The warrant is the assumption on which the claim and the evidence depend. Another way of saying this would be that the warrant explains why the data support the claim.

Toulmin says that the weakest part of any argument is its weakest warrant. Remember that the warrant is the link between the data and the claim. If the warrant isn’t valid, the argument collapses.

Example:

    Claim: You should buy our toothwhitening product.
    Data or Grounds: Studies show that teeth are 50% whiter after using the product for a specified time.
    Warrant: People want whiter teeth.

Notice that those commercials don’t usually bother trying to convince you that you want whiter teeth; instead, they assume that you have accepted the value our culture places on whiter teeth.

https://www.blinn.edu/writing-centers/pdfs/Toulmin-Argument....

I would start by simply putting everyone through a course in deductive reasoning at the earliest age possible: https://en.wikipedia.org/wiki/Deductive_reasoning

From there you can go into the whole spectrum of critical thinking approaches, and then on to what's basically the liberal arts e.g. philosophy, social sciences etc. as you desire. But the value you get from all of those things depends heavily on the framework you have for thinking about them going in.

Claiming random things are "fake news" would be a lot harder if people could work out what is and isn't fake by themselves!

I agree so strongly with this.

The point I would add is that hardly anyone uses the empirical process directly. It is all 'this article claims this' or 'this study says that'. It's very 'meta' with little to no personal verification or testing of the claims - ie, theories based on theories or models based on models, or maps based on maps.

Very few check the terrain itself to confirm that the map applies. We trust education, experts, peer review etc. We're drowning in models, especially as these are easily represented on computers, but have no ability to check the models against reality.

PS this disassociation from reality will not improve as we move forward technologically. No doubt, in the metaverse we will be able to create ever more elaborate models, or is it that we will be ever more disassociated from our own anecdotal experiences? (Where 'anecdotal' is something to apologise about).

> This entire discussion makes a good case for why the general populace would benefit from being taught the basics of philosophy.

But our entire education pipeline is optimized for loading people into the “system”. Philosophy etc. has little market value (unless it aligns with the system).

While this is indeed an issue that falls within the domain of philosophy, philosophy is also home to fields in which the absence of empirical evidence is regarded as an irrelevance, or at best a mere detail that can be deferred to an indefinite future. Take, for example, the resurgence of enthusiasm for panpsychism, and the enduring appeal of armchair metaphysics.

I am doubtful that academic philosophy has much enthusiasm for pursuing and inculcating the practical aspects of reason (any more than does theoretical physics or mathematics), though there are exceptions.

Idk how if studying philosophy helps. Most philosophers were/are themselves committed to one school or theory, with gaps galore.

In any case, I think empirical science's defeat of rationalism ( eg Galileo Vs Church) has all sorry of ramifications. Social sciences like economics and psychology have a lot of trouble bridging the gaps.

The problem with this is that philosophy isn't a magical panacea that illuminates the way towards a more ideal state. It can be used to justify a sea of mutually exclusive claims that may not be in accordance with reality, and that's to say nothing about the quality of the arguments themselves.

Doesn’t the scientific method specifically say you can’t start with the data, you have to start with a hypothesis otherwise you are subject to all sorts of selection/hindsight biases. I mean you can start with data, but then you have to develop a hypothesis and use that to create an experiment that generates new data in order to reach a conclusion. It seems like that is the compromise the author is looking for, start with a good idea, then see if you can verify it with data.

> But you can't just let the data "speak for itself" without an explanation or a theory that interprets the data.

If you look at the heart attack data, and you ignore smoking you end up inventing the mythical Type A personality — but it was data driven.

https://en.m.wikipedia.org/wiki/Type_A_and_Type_B_personalit...

A single metric is just one very thin dimension from the temporal development of a complex process involving many factors. You need to watch a multitude of metrics to devise an explanative theory, and even then, that theory can be rendered flawed when new and unexpected factors come at play.

The fact that empiricism is false was a revelation to me as a young adult, after reading so much about the triumphs of science and reason and getting very excited (mistakenly) that you can get away without bothering with pesky things like epistemology. Of course, Quine and others pointed out that empirical observations are useless without explanations both of the phenomenon being measured and the measurement device itself (including, for example, the human vision system). And I believe it was Deutschmark who pointed out that empiricism is itself an epistemology which had to be invented. It turns out that it tended to be a significant improvement upon previous widespread epistemologies, but that doesn’t mean it’s not false. :)

I agree, pure empiricism can lead to superstition. If you only learn from experience, and do not have any theory that ensures the consistency of the model, it's easy to infer wrong causal connections.

> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems.

Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.

I was about to post that.

Here is a good talk https://www.youtube.com/watch?v=EVwjofV5TgU

frankly, there’s only a tiny handful of these mythical saas “1000 users each paying 1 mullion dollars” companies. the vast, vast majority of saas startups are serving millions of “users” - i put that in quotes because these aren’t real users or customers. they are real people checking out your product - but they aren’t users or customers.

if you set up a gas station near the off ramp of some major interstate, say I-65 North, you will see cars pulling in to fill up on gas. maybe buying a coffee. now, these aren’t your customers in the traditional sense of a Target or Walmart customer. Because you will never see them again. They were driving from town A to town B via the interstate- they started running out of gas and needed to refuel, so they are in your gas station now. Once they gas up, off they go. They aren’t going to come back to you and establish a customer relationship or something. We’ve all been to tons of gas stations on the interstate and we’ll probably never go back to the same one twice - unless we are plying the same route everyday like a truck driver. So the task is to find and convert these truck drivers, who are the true repeat customers.

I was working on an android app which had like millions of unique cookies. When they hired me they said we have million of users. No you don’t. If you put out an android app in some popular domain, say news, entertainment, tax accounting etc- people will download and “use” your app. they are checking it out. they aren’t users, in the sense they aren’t using it everyday or want to have a relationship with you, pay subscription etc. conversion stats are minuscule, like 0.01%. So maybe 1 out of 10000 users is the truck driver. The vast majority will never ever use your app again. To do data science with these millions of rows of user interactions and find some nuggets just because you know your way around pandas or sklearn is a fool’s pursuit. To ask foolish questions of your data, like why are all these people churning, is silly - they aren’t your users, they haven’t converted, they are just checking it out. In that sense, its a waste of time and resources to do so much data crunching. Look at actual conversions, which are probably a few thousand people, not millions. Reach out to those thousands and maybe a few tens will give feedback and then continue to iterate on the product based on that.

Unfortunately, your reality-driven approach has ~zero emotional appeal for most managers, exec's, and alpha-data-scientist wanna-be's.

Talking to people is not going to help you either. You end up getting a lot of noise and making sense of what you hear is difficult. When you keep probing you will get to hear stuff thats not really critical and just often made up because you ask too many questions. Classical trap of market research.

> You know, in data science, you see people spending hours writing pandas scripts that replicate a few clicks in excel for a one of analysis

I mean, having an Excel doc at all usually implies hour(s) of work formatting the data in structured manner. Sometimes collective decades of work depending on how much heavy lifting your 15GB .xlsx is doing.

I say this about every other day at work (we even have only internal users so it's part of their job to talk to us). So far impact: zero....

Would you say the big data threshold moves every year?

That would explain why people think a <1TB is big data.

Why not both…

This is a textbook example of The Strawman Argument.

I'm pretty sure the author is talking about "data" in the context of "databases", i.e. repositories of digital information that can be queried, transformed and displayed (dashboarded).

In other words the author is assuming the value of human's more natural data processing: common sense, personal experience and conversing with others (empathy).

If a process/feature/etc doesn't make sense within how you understand your product, then you can make an argument based on that. The argument will involve data (i.e. the current architecture) but not data in any database.

A simple answer to this is that good explanations are hard to vary.

More here: https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanatio...

> It presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments.

It does indeed assume that there's a way to learn bad arguments from good; and so the focus should be on learning what are good argument and what are bad.

> ...What makes a good argument if not empirical data?

Consider the following conversation:

A: We've done some numbers, and we've determined that there's a correlation between the number of firemen at a fire and the total damage done by the fire; with the fires handled by a single crew of three firemen doing the least damage. So we should limit all fire responses to a single crew to minimize damage.

B: That doesn't make any sense -- of course we send more firemen to bigger fires, and bigger fires cause more destruction! If we take your advice, those big fires will cause even more damage!

A: Hey, my argument is backed by empirical data; yours is just theoretical!

Like, sure, it might be even better if B had empirical data to back him up; but even without that data, B should be winning the argument here. And the argument of the article is that many people espousing "data-driven" approaches end up being like A: Not scrutinizing the logic that they're using to analyze the data, and not acknowledging the limitations of what the data collected can say.

Good arguments (explanations) are hard to vary.

More here: https://www.lesswrong.com/posts/jcTsbaQ8hNc7qxwaQ/explanatio...