The Forgotten Job of a Data Scientist: Editing

The Forgotten Job of a Data Scientist: Editing(john-foreman.com)

22 points by rouli 12 years ago | 7 comments

novum 12 years ago |

Data Scientist (noun): A statistician who lives in San Francisco.

(only half joking)

I feel like "data scientist" is a title that grew out of the Fundamental Theorem of Employment, which states that you're usually hired to do a job that either (1) the boss man can't do for himself, or (2) the boss doesn't want to do. Type 1 work gets you respect and autonomy. Type 2 work will have you commoditized.

Software companies are satisfied with the job they've done at commoditizing programming talent but, at least for now, having a half-decent grasp of any specialty (e.g. machine learning, information retrieval) requiring mathematical firepower puts one solidly into Type-1 employment, which is where one wants to be.

"Data scientist" seems to be a way of saying, "yes, I code but I also know math, so use me for Type-1 work only".

eshvk 12 years ago | | |

> "Data scientist" seems to be a way of saying, "yes, I code but I also know math, so use me for Type-1 work only".

You make it sound like a bad thing? Despite the rah-rah I hear from programmers about how they are unique snowflakes, being only a programmer is like being a janitor. A prime way to get discarded at the age of 40. If I can make sure that I am valuable because I bring other things to the table (Math, Product vision, people skills), why on earth wouldn't I rebrand myself to better reflect that?

michaelochurch 12 years ago |

"Data scientist" is a mess of a job title. It seems to be as much of a reaction against the commoditization of software engineering (which leaves the smartest, and by correlation, usually the most mathematically literate, 10% of programmers ill-suited for the average software job) as it is a real distinction.

There are plenty of "data scientists" who use canned tools and play around with parameters because that's all "the business" thinks it needs.

You want to trim complexity for a reason that any data scientist worth his salt (and there are plenty of celebrity engineers in SF making $500k who aren't worth their salt and don't know this) should already know: bias-variance tradeoff (see also: underfitting and overfitting). If your model is too flexible/complex, it will begin absorbing noise. That leads to a model that performs extremely well on training data but fails miserably on unseen data. There are well-studied techniques for preventing this, but I'd guess that fewer than 20% of self-described or titled "data scientists" are familiar with them.

eshvk 12 years ago | |

> There are plenty of "data scientists" who use canned tools and play around with parameters because that's all "the business" thinks it needs.

As with a software engineer, it is a role that is different in every place. Every place has its own definition of the role. This is not bad. It is a mere reflection of the market conditions where there are a lot of people are simultaneously bad at Linear Algebra, Probability and Statistics and dangerous enough to write production code fast. (Your standard C.S. grad SWE).