Not my area of expertise at all but, 35 engineers for WhatsApp sounds like a lot
Most of the successful tech companies he is referencing are using Python. And no one saw a big successful website/app in java. Most of the time corporate or gouvernement shitty web applications that randomly crash with backtraces when you are using them.
Sometimes that means picking technology that is not mainstream. Choosing mainstream technology is one of the best ways to make your production environment decidedly not boring.
I would argue Erlang is one of the most boring pieces of tech you can pick. Look at how boring your production systems can get when you choose that.
Sorry, you drank too much of the kool aid. Let me clear it up for you: you're a cost center that the corp would rather not have.
I kinda thought coding might be immune, because it's about information. But of course, you build on lower levels. Today, you can get outsized returns by applying LLMs to real problems. Also about information.
If you want a different view on your code: write one. Code is the model. Presentation of code in an editing UI is the view.
Tools like gofmt normalize the model so that it's easier to build a view that re-represents it.
A number of years ago, the uptight cultural conformity of the valley bubble branded itself with some of the imagery the author attempts to borrow. The collective consciousness hasn't forgotten how that image was power-blasted away. It was done as cover to sneak in a bunch of orthogonal changes while nobody was looking, but that part's neither here nor there. Forget I even mentioned it. The end result is that we're here now and we have the tools to evaporate this article with one shot at 50 paces.
I recognize the author is a professional developer, and seemingly has worked on teams to build complex things, but it appears as if they forgot how the human machine works, and are now stuck on some kind of nostalgia trip where everyone involved in the process is somehow like them and only able to gain energy through self expression in their software engineering work.
I do wonder sometimes if there's something about living in SF or NY that makes you forget how tiny your perspective actually is.
> we prevent any Taylor Swifts from ascending
So... success?
Almost the same specs, but the one in the startup had better scalability and fewer serious bugs...
So, spend 10-50x less and get higher quality?
Worst is I now consider case b) quite lean and efficient compared to what I see tech consultancies doing towards public sector, banks, etc...
-- My point being: There definitely IS a room for "rockstars" (don't like that term, but interpret this as developers competent enough to carry a lot of weight on their own and work in a smaller company) to deliver a lot of value.
The problem with the model is the lack of predictability (what if the few developers you trust don't deliver), the bus factor if one of the few devs quits, etc
In a sense inefficiency is a goal, because otherwise you loose redundancy. It costs a lot to hire 100 to produce the output of 10, but much lower risk.
What absolutely does not mean that teams are useless. But that choosing the correct size and composition for them is one of the most important decisions for a project (right there with "are we solving the correct problem?"). And "more is better" is a completely wrong way to decide on it.
Also, this doesn't make that one developer a "rockstar" or whatever else you call it.
Same goes for "rockstars", only its easier not to "lose your groove" if there are fewer people pinging you. You still run the real risk of two "rockstars" playing out of sync, and causing a mighty mess down the line.
I've seen 5-person teams of raw bootcamp grads run rings around conventional dev teams with a broader experience base, repeatedly. Part of it is that there are no bullshit status games going on when everyone knows everyone else is just as in the dark as they are, but (in my view) an equally large factor is that the grad teams know they're going to have to learn whatever tools the problem needs. A more tenured team is far more likely to waste time on trying to build java in python, or fighting over not being able to use their favourite libraries.
But then there's also the wider situational context: an environment that has 100+ developers in a big company will have governance tarpits and coordination overheads they almost certainly haven't measured the cost of; 10x wouldn't surprise me at all just from those two factors.
Grad teams in the same org are more likely to have standalone, lower-risk projects to work on so they're going to be more insulated from those effects.
Every more junior person on a project is going to be less efficient than a senior person but they need to level up and go through the apprenticeship phases to become a senior person.
Very small companies or teams can tactically choose to pay more and only hire top talent but most places will need to develop internal talent due to budget, recruiting, or bus factor (business resiliency) constraints.
20 developers is a lot, but could in theory be worthwhile. I'd really try to get all 20 be very-effective ones and structure the responsibilities and communication so that they don't get bogged down. If you do it a naive way, and aren't lucky, it'd be easy for 20 1x..10x developers to become 0.01x developers.
Or, if you get people who just want to call themselves a "dev" and grind concrete task assignments in a straightforward way, then I think you either need to be doing something bog-standard and naturally amendable to just going through the motions (e.g., scalable Web CRUD, or yet another forum/chat app), or you need a brilliant approach to structuring the work.
The 'good' (whatever that means) devs will waste more of their time hacking around the above... so more devs are added to compensate... repeat a few times: Fred's prophecy becomes manifest.
The problem is the velocity hit you take by doing this is its own risk that's not tracked in the same way predictability and bus factor are. I'll bet one of the core reasons OpenAI was able to release a ChatGPT style product so much faster than Google, despite Google coming up with the fundamental Transformer breakthroughs, is this difference in velocity. And when a startup begins to erode a calcified, highly redundant larger company, then the larger company has the risk of becoming irrelevant along with all of its internal redundancy.
I think there's a better way to think about risk here.
There's lower risk with the predictable team. Those are so common, especially in established business, that we can even treat this as the default state.
If you so wish, you can take on more risk and seriously slim down and get work done faster. And it will almost certainly be faster on a large enough time scale, but because it's less predictable, you're now playing schedule chicken.
The kind of risk that you bring up, risk of being less risky and going slower . . . is really just opportunity cost. And it's ok to take that on as a tradeoff for something else, too, if you do it with eyes wide open.
Risk is not a thing to be avoided. It is a thing to harness to gain an edge on competition. After all, without risk there is no profit.
Yep, this is known as McNamara Fallacy - https://en.m.wikipedia.org/wiki/McNamara_fallacy
Mythical man month in play. In my experience, teams working on a single project peak at around 3-4 people, depending on how well you can silo the responsibilities. You can have 200 developers on a massive project, but the planners need to be able to silo the pieces off properly and define interfaces very well early on, which requires a massive planning phase. That usually doesn't happen the way people want it to either.
https://en.wikipedia.org/wiki/The_Mythical_Man-Month
Currently on my team, we have one guy responsible for the schema, back end webapis and automation. Another guy does exclusively UI, those are the two main people. We have another guy responsible for auth and misc stuff and the third guy is a full stack that came in after major development was complete who can add features pretty quickly. We have a completely separate team that handles billing operations/applications of 2 people. Our team members communicate directly with business for requirements and understanding their needs, so nothing gets convoluted passing information down through middle channels.
It's not the only way to do it, but it works pretty well for us. We write all the programming required for operations of a pretty complex medical company. This was a complete Greenfield project and we have about 10 major and minor applications so far. We also have to adapt to incoming partner companies who have their own special needs and changing regulations, so the system and team is very flexible. We're about 4 years in and we had a working system up and running from "File -> New" after about 6 months. All this during COVID.
Another problem with big companies that I've seen is that everything is done by committee and about 10 people need to sign off on any little thing. It turns into an administrative nightmare.
The Microsoft Band golfing on device UI was built by 1 developer in 2 or 3 months.
The entire notification experience was 2 or 3 developers on and off again for the product's lifecycle.
The onscreen keyboard was a couple ML engineers and a software engineer for a few months.
The runtime it ran on was ~8 engineers, including all drivers, board bring ups, custom file system, everything.
Small teams can do amazing things, but they rely on good leadership and on every member on the team being able to pull their own weight. We had 1 person design+code the entire file system, and as you mention, that put the bus factor at 1.
One theory is: If you have just a little too many developers (redundancy), they become bored and find ways to introduce auxiliary complexities to keep things interesting, thus things explode to 100 developers being needed...
That is, the first project wasn't 15x as productive so much because the developers were so (intrinsically) -- or even when they were objectively all above average. But because beyond a certain optimal team size, per-developer productivity drops substantially -- even plummets precipitously (due to the factors articulated in TMM: communication, politics, etc).
Plus there's a huge survivorship / selective bias example involved here (we're not look at pairings of other projects where the smaller team failed, even with better developers, because it was just too small, etc). I also suspect that if one were on the "inside" of both projects (which of course no one was) one would see a host of other factors involved (from knowledge of the requirements to market conditions, etc) not visible to an outsider.
We can definitely see how anecdotes like these help promulgate the 10x myth at least. People hear about the fates of the respective teams and think "Of course, it's because those developers were individually that much more productive. I better start making my interview process 10x more difficult now ..."
(BTW, no need to drill down on what I just said about the "10x myth", please. It's not that awesome developers don't exist or aren't important. It's just that for a whole lot of reasons, it is both incredibly overhyped, and objectively wrong in its fundamentals -- especially when we look at the original studies whence it supposedly derived).
This is why I say "rockstar" is a synonym for mug. Why won't they pay those 6 developer 10-50x more?
I've seen it many times. Young developers working their bottoms off, creating a product that makes millions only to earn average developer salary and then being laid off when company gets new investment round.
At the same time, through progressive taxation, taxman takes (depending on the country) majority of earnings, so you won't be able to amass capital to start your own business.
This is the way how the rich captured the means of production (developers) in wage slavery.
You won't make it, unless gatekeepers let you.
I spent a decade in the public sector of Denmark and I've since got into helping non-SWE startups transition into enterprise organisations as far as digitalisation goes. So I've seen quite a lot of this. I've also had the pleasure to be on basically every sides of the table, from developer, architect, manager to buyer and consultant. I think it's always down to the business processes and the management.
Sure "rockstars" are a thing in SWE, I'm frankly one of them. However I don't think the difference between a 10x and a 1x/rockstar/whatever buzzword developer is necessarily too much about our technical abilities or talent. In my opinion it's often mainly defined by your organisational prowess. If you're the type of developer who can get things done, make an impact and create business value, you're going to be given a lot of freedom and you're eventually going to settle into "rockstar" roles. This is soooo much easier in smaller organisations where the restrictions you face on how you can work are often small-non-existant. Maybe you're even going to build some of the restrictions yourself, by defining how your organisations wants to lint, which techs you want to use and so on.
As things grow, more and more people are going to enter the work processes. Some of them are going to be hired to work on the work processes themselves, and eventually you're going to end up with this quagmire of bullshit red-tape. It's there with good intentions, of course it is, nobody goes to work meaning to make the world a worse place, but all the control and the business intelligence is usually an illusion of sorts. At least in my experience. Now, if you have good managers they'll build teams and manage the processes in a way that doesn't get too much in the way, but over the years, you're going to have a lot of bad managers involved in the processes. You're likely also going to work for an organisation where upper management doesn't care about digitalisation, at least not really, despite the fact that 90% of their work force is on a computer 8 hours a day. So eventually you're going to end up with an environment where it's hard to get things done.
What is worse, often the best developers. The people who are able to deliver a lot of visible business value, are going to be plucked by management in attempts to try and use them to make other developers produce more value. Meaning that many organisations tie their best development resources up in advisory roles and waste them away in endless meetings.
So it's not that those 6 developers are necessarily better than the 100. It's that sometimes, a team of 6 developers get to do more actual work than 100 developers combined do in a year.
Obviously that's only part of it. There is a whole range of other things that impact why big organisations suck. Because the quality of the 6 man team isn't necessarily on par with the 100+ developer team if you look at the application at scale over a decade. At least not in my experience. This might not matter when you're small, but it will matter when you're IBM because a few bad years will wreck your reputation for decades.
> So, spend 10-50x less and get higher quality?
You're never going to convince businesses that it's less risky to buy from a small company. It's not like CEOs are stupid. They might not care about IT too much, and they certainly hate it when IBM fails to deliver on time and that it'll cost extra to get the thing to actually work. But they do know that it will eventually work when they go to the 100+ developer company. They don't necessarily know this when they go to the 6 man team.
In fact, one of the primary reasons you hear so much more about the 100+ developer teams is because often the small startup will simply go bankrupt when the contract fails. I'm currently working on fixing one such thing, I can't tell you much more than that, but lets just say, that the 100+ developer company would've been the wiser option.
Now, sometimes you do get the better quality, the faster deliverance and the better product form the 6 developer startup, so it's not like you're wrong either. It's just that most decision makers hates risk.
ps. Sorry IBM.
That said, I think this article is essentially correct. I would put it like this: building information machines out of pure information is a creative endeavor. Human performance tends to fall on a bell curve, and IMO it's true for developer output in particular. So if you want to create a hugely valuable solution to a big problem, you should probably hire the best people you can find and organize them in a way that lets them max out their creative power.
The main obstacle to this IMO is poor leadership attitudes & practices. This article's author said it best in another post:
"Boring" is a good strategy if you think the bigger existential risk for your company is that Product & Engineering will [...] fail to ship a reliable product on a reliable schedule. But if you're more afraid of the business risks of shipping average product, at an average cadence, over the course of years, you should consider deviating, at least sometimes, from the playbook whose entire definition is "optimized for safety."
Right now the biggest thing bringing down developer wages is the threat of being replaced with an LLM. That's not going to work out in the long run - giving developers productivity increasing tools isn't going to lessen their value. However, it's the most plausible wage suppression hustle for quite some time.
Until the limits of LLM generation get tested and felt, the downward pressure on wages is going to continue. Maybe when the first wave of LLM driven startups feel the pain of maintaining code that doesn't have an author, or maybe high profile malicious training data attacks will do it. Even afterwards, the effects of the wage curve being bent downward for a couple years are still going to be there.
not really sure if this is helpful, but i see this take way too often on here. i’m convinced people who say this about the film and media industry don’t go out of their way to find good movies and listen to good music because the amount of high quality, hard to find stuff being made today is honestly amazing.
Same thing now days, but replace Java/accounting with React/CRUD. The rockstar code ninja 10x mentality of 2009 is maybe gone for good, but I see more crazy coders doing rockstar level stuff now than ever before. Asahi Linus and alyssa rosenzweig being great examples from the Asahi project, but these people are all over.
People talk about the complexity of the infrastructure making it impossible to do everything yourself, but in my experience is exactly the opposite. The degree of abstration, automation, and high-level tools means that you kinda can do everything without knowing too much.
To piggyback on this, I see many coders of this level of talent in many different fields all working on their own projects that are sponsored by Patreon or similar, rather than taking traditional employment with companies in general.
I much prefer ninja, rockstar, magician, sorcerer, codebro, gnarly dude, big daddy for loop, Little Bobby Tables, SSHelley and Nadine.
Nadine doesn’t have a buzzword she’s just really good with Kubernetes.
Anything but 10x-er.
I get why this happened. A startup has to deliver features now if they want to get acquired. It's a completely different mentality from the one we have now: no worries about funding, and we want to do things right in a way that will be robust long-term. Realistically, this is how it was always going to go down.
But I can't say that this is ideal. You don't want a couple rock stars carrying your project if you also want that project to be around in 10 years and can't guarantee that all your rock stars will be dedicated for life. You need somebody else to be able to pick up the torch. You want "commodification" (although I don't think it's a very accurate term; it's actually hard to get this right, not easy, but the alternative is much worse over time).
Pure creative programming work have always been a small subset of programming in general. Were COBOL programmers creative? Data Base programmers? IBM programmers? Those were the majority of software jobs in the past.
I have been mostly a pure C,Verilog, VHDL, analog digital electronics -low level engineer for a long time before becoming entrepreneur and using much higher level languages in our company. Was it creative? A lot. I interacted with machines like robots that did things like moving 20 tons or lasers or robots or whatever.
Was is painful, tedious and boring?. It was, also.
You need to make it rain using active work of whatever is necessary for your job. The fact that something is not "sexy" is a great advantage for a job as it removes most of the competition. Being hard scares most people.
The best advice I could give young people is to find hard problems and to find the techniques and tools that could make those problems easier. We use psychology and things like lisp as secret weapons.
Thinking in the 35 people that created Facebook is extremely misleading, because fb got extremely lucky, and they were hundreds of thousand of programmers working in other companies. The monetary success of fb has a lot more to do with free money created by central banks than anything the founders created as their technology was easy to replicate but its financing was not.
There are intense opportunities today like they were in the past. But they are hard, like it was hard in the past.
Some of those jobs surely had creative aspects to them … far more creative than the recent YAWS (yet another website) using ASWF (another shiny web framework) with all the bells and whistles included.
Employers don't want rockstars, they want underpaid, overworked workers.
First web browsers were a thing. Napster was a thing. Altavista was a thing. Skype was a thing. One of the first mobile apps (anyone remembers Symbian?) was a thing.
Things that some people enjoyed using and other people enjoyed creating.
Now, what can excite you as either an user or creator? Your average LOB app? A random chat app that takes ages to start and eats all your computer memory? Uber? Netflix app?
i had to let this person go because they did not know how to work with others. beyond their brilliance, they could simply not empathize with the organization's POV. everything was about them, supporting them, making them feel special, unique, and important.
this individual eventually returned the company laptop, which was bent in half. they told me that they had smoked enough DMT to understand that they had created their own god and that if my org needed further investment, to reach out to them.
i hired this person from HN "who wants to be hired." they still (to this day) add a post to the monthly thread.
I guess they were a rock star in the "Led Zepplin trashing a hotel room" sense.
Now it makes sense
Not that you can't work as an independent "creative" in programming though.
I've gotten to a point where I can work on Marginalia Search full time, and live off grants and donations for at least a few years. Dunno if I'm a rock star for it, I don't even have a Wikipedia page, but I at least reckon myself a fairly successful busker.
If I don't qualify, then certainly Serenity OS Andreas who made like $300k in a week with his browser shenanigans. Him and a other people far more talented than I am are able to live this way.
But it's a weird game. I think it's more or less all about the right sort of visibility. You distinguish yourself working on some large problem in a public fashion, and people inevitably will start throwing money your way; this enables additional work and additional visibility, it gives even more opportunities.
While maybe not reaching the John Romero levels of rockstar we had in the '90s, this manner of working is certainly closer to the success of a Taylor Swift than being the most talented developer in the IT department.
Application development technology has considerably improved over time - most applications simply do not need to reinvent the wheel! Yes, over-engineering by designing something that will never see more than 1qps to scale infinitely is bad - I’m sure it happens but I think it’s more a strawman. If you need a simple CRUD application with a good-enough UI you have no need to introduce additional complexity (and potentially maintenance, reliability issues) with custom tooling.
Two, the software talent market is bifurcated. There is basically commodity development of crud apps, and technically complex novel development. If you think there are no rockstars you might just be in the commodity development scene. There literally are these so-called “rockstars” being flown into SF to work on new stuff in the ML sphere or into NYC/Chicago to work on bleeding edge performance. Maybe the dissonance here is that the commodity developer market has grown a lot, and that over time some technology (like web applications - a lot harder to do at scale in 2005 vs now) shifts from rockstar to commodity as it matures.
Reverting to pets-not-cattle and statefulness can be appropriate at low scale. But honestly this is more of a “choose the right solution to the problem” thing and not a rockstar thing. Following this model as you reach large scale allows for cool production heroism and cowboy coding that makes you feel like a true hacker, but that doesn’t mean your users are getting a more reliable experience, or that your dev time is being spent efficiently.
My minor quip is that, I think as you get more experienced what you used to think of as rockstar development just looks routine because you’re better at writing software.
Another minor point: you can’t just engender a rockstar culture at a company that hires commodity developers as easily as asserted here. The big thing not mentioned: PAY. Nobody wants to get paid like a commodity developer to have to perform like a rockstar. Being a commodity developer is more chill and there is less day to day risk and stress. Once you start getting towards the bleeding edge or reinventing the wheel your work becomes riskier and requires more mental effort and attention.
https://en.wikipedia.org/wiki/Winner-take-all_market
I agree with bullet points about keeping it simple and not using crap tech for your MVP.
It also exists in a much more banal state lower down the ladder. I have friends who landed in the low end of the developer market. They literally cannot do most of the work I do. They don't understand DB isolation levels, they don't have the skills to design systems around eventual consistency, they are not familiar with basic distributed system problems like clock drift, etc. I'm not even an infrastructure engineer, this is just stuff I and my coworkers need to get through our days to ship product code.
Could my friends do this stuff? If they applied themselves, sure! But they're not going to. It's also not enough to learn on the job. We occasionally hire someone who doesn't actually have the skillset and they usually flounder.
"Knowledge worker productivity is the biggest of the 21st century management challenges." - Peter Drucker (the founder of modern management)
It's a false dichotomy to say you only have rock stars and as this person smugly tip toes around "normal people" when in reality you don't need rock stars anymore to make good software and let's be honest... Most rock stars didn't make good software they just make it in a time when software was generally even more crap than it is now.
You want to stop suffering among us plebs? Don't advocate for goofy rockstar developer propaganda, advocate for healthy work life balance and reasonable deadlines for things that truly don't matter. Stop letting sales and marketing write your software and stop taking opinions about systems design from your project managers and "technical leads" when they do not work in these systems day to day.
If you treat engineers well and respect them before a client who will drop you the moment a new product fits their need then yes you will lose clients from time to time but if you focus on making good software and happy people then you will attract stable clients who do the same and maybe the stock holders at the top don't get the ridiculous return per year that they expect out of more shameless companies but at least you have a half decent chance of sleeping at night...
I am well aware that we live in a world where this will be borderline impossible but the first step to solving a problem is admitting it
Some industries you either are the top 0.01% and make millions; in some industries being average means a decent living.
Software has long transitioned from one end of that spectrum to something more towards the middle. Super star developers simply aren't productive enough for the demand of software
Like other sports, it's fun enough that many play it for free at local tournaments, and some families can afford pay expensive flights and stay abroad.
I recall meeting with sportsmen and travellers who do months-long trips across continents. Half of them were pensioneers and just one single guy was a 34-yo die-hard traveller riding a bike across Eurasia for its own sake. Others were professionals, and they'd say they'd been preparing the travel for 2 years, seeking sponsors -- some in equipment, some in money, -- and signing contracts and clarifying sponsorship activity details -- photos here, report and booklet there, etc.
This contrasted a lot with our IT industry "let's do it for fun or learning" spirit.
Makes it much more difficult to parse when you say "either" but only provide one half of that statement.
Why is it they can follow a recipe to kind of make a mint birthday cake but they can't point at a picture of a cake on a picture menu and say "that one!"
Sometimes I think they don't actually know what they want before they start. And as the script evolves, like finger painting, they start to kind of like the blobs ...
But, I dont work at a software company. I just work at an enterprise with a large IT department.
They point at the picture of the birthday cake, and they receive an apple pie and cannot figure out why. You have to read the documentation to understand that standing on one foot while pointing makes a difference.
Declarative tools are easier when you understand the domain. Otherwise, problems arise because small changes in the requirements can result in huge changes in the outcome.
They could. They just don't want to, because it's not as sexy as writing code. If you can't do it in Python, from scratch (lol with a lot of modules and very little exception handling), it's clearly inferior and a waste of time to learn. Their intent is to write code, not get things done!
They don't learn how to write shell scripts either. I'm sure they will go their entire lives without ever reading the man page. But will read a million bad blog posts about Python and take from them the worst conventions from mediocre programmers trying to get eyeballs on ads.
unlike creative industries, there's a fuckton of toil work that just needs to be done automating processes. the playbook style excels there, and unlike artistic work there's not really room for memorable standouts that break the rules.
music will always have the quality where good novelty is impressive and breaks boundaries. business processes less so. prince can (could) deliver a stellar rendition of purple rain that breaks the bonds between heaven and earth even though you've heard the song before, but ain't nobody delivering a stellar automated check deposit workflow execution that stands out from the rest.
commodity services bring commodity work. this is fine. i do not want my mobile check deposit to be a tour de force, i want it to be something i don't give a second thought. there is beauty in the blase' being blase'
The obvious conclusion is that market conditions matter more than the technology stack for a space of "reasonable" choices about it. I expect, until the next big thing hits, the market conditions to be dominated by consolidation. Bitcoin was supposed to bit it, but that was a scam. VR was also a scam (Zuck tried to sell us non-existent real estate), but maybe it'll turn into something more useful soon.
I'm hoping higher interest rates may revive the older style: https://news.ycombinator.com/item?id=34565816
Don't know how many musicians you know, just ask one...
Now there are no hard problems. Just throw more hardware at it. Just write another CRUD app. Just wrap it in JavaScript. It is commodified because it's now a commodity. Software is ubiquitous and easy. There is no need to be creative, any more than for filing your taxes.
The people who like to program are, almost universally, nerds who get off on logic, solving problems, building things. But there aren't really new problems to solve. So they try to re-solve the same problems, over and over, without improving on what came before. Never satisfied. Software will always be the realm of these people that are obsessed with reinventing their toys in a sandbox. So the products will always be kind of toy-like.
That's why hardware gets better while software stays about the same. Hardware people can't just play with toys. If the hardware doesn't get better, nobody will buy new hardware, and they'll be out of a job. But people will buy different software and call that "better". Even though it's doing about the same thing as before, less efficiently.
> That said, he and I will likely never see eye-to-eye on this because he more-or-less invented Choose Boring Technology, and most of what I advocate is to break playbook, embrace narrative, and be interesting.
It really comes down to what you prioritize. Do you prioritize the output and the value added? Or do you prioritize making your engineers feel like "rockstars"? If you don't care about your end product, then yeah sure let your engineers break prod using untested technologies.
The reason playbooks have sprung up, is because 99% of use-cases for businesses are for the most part "solved." We're not in the era of renting colos and manually setting up MySQL anymore (which the author advocates for in order to return to "creatives," btw). We have managed solutions that you can throw money at. So, do you leverage those solutions to help you solve your problem? Or do you let your engineers re-invent the wheel so they can feel smart?
The "rockstar" treatment is a result of supply and demand. Talent in Hollywood/music can get the rockstar treatment, because the talent alone is the difference between millions of dollars. If you have a really talented coder who can be interchanged with another really talented coder, you do not have a "rockstar."
The "rockstar" era of coding was the result of a massive imbalance in supply/demand for engineers, so the rockstar treatment would help attract talent. Now that the supply has began shifting up to meet the demand, it makes sense that the "rockstar" treatment is starting to go away.
And good riddance to all that too. I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone just because they can implement a CRUD app. If you want to be a rockstar, go actually be a rockstar, don't make software engineering worse.
Stars are a thing, they are way beyond what the average dev can ever accomplish and they would only be held back by a team. They should be catered to and really be accountable to nobody as far as technical decisions go. Their time and energy is just too precious to be spent soothing your fragile egos. And that's not just my opinion either. That's why companies have positions like fellow, distinguished, etc.
The whole "rock-stars are toxic" is just the inability of some people to accept their limitations. Most of us are nobodies, easily replaceable code monkeys who will never accomplish anything of note. If you're of the opinion that a nice personality somehow is worth more than being excellent you are lying to yourself first and foremost. And you're not that nice anyway since you are willing to spend your time digging through stars' closets in the hope of finding any skeleton to bring them down.
Being agreeable only qualifies you to easily being worked in what are essentially sweatshops who produce crap, the subject of the article. Take note though that said sweatshops would not exist without the stars that CREATED this industry. Never in a million years, you and your team of "nice" people would have produced any of the foundations of current IT.
Same thing with math academy honestly.
There are 10 types of people in the world . . .
Well, sure, but to the contrary: DC has had a much harder time imitating Marvel's success, even with great IP. Universal tried to create the "Dark Universe." And even Marvel is having trouble cranking out the hits with the middling success of Ant-Man Quantumania, Thor: Love and Thunder, and Wakanda Forever.
I suspect making these things more streamlined and more of a flat cultural paste through the Disney Machine is a part of the lower outcomes.
I think there are plenty of examples of amazing software coming out of small teams or even single developers and plenty of examples of large corporations producing garbage with huge teams.
> we want to do things right in a way that will be robust long-term.
If you don't have the right people that know what "right" is and have seen projects through the long term then this is often used as an excuse to over-engineer and do work nobody cares about. At least that's my experience.
Never seen a place with worst technical debt, complexity and just difficult to work in.
I am convinced that commodification is what is actually causing all the issues we are seeing.
A bunch of mediocre developers getting in the way and creating complexity without any creativity, out of the box thinking or even holistic view of the system.
But management wants more people so that everybody can leave and not holding the company hostage.
Guess what? It is only worse. Very few people can actually make progress and those people are constrained by the whole team around them.
she showed me the very nice hotel that they would be putting me up at until I found a place. no hurry really. one of my new coworkers was still there after a few months.
we went back to the office for tastings. coffee. wine. beer. whiskey. it was important that they knew my tastes and would have my preferred food and drink on hand.
i know that was the point - but I certainly felt like a rock star.
I once got flown out to Shanghai from NYC, all expenses paid for 3 nights to interview for the China operations of an European tech company.
I guess everything is now different with remote work but before the pandemic companies, especially your prospective manager, would go out of their way to make you feel good if they really wanted you. Those were the places you wanted to work, even if the salary was higher somewhere else, because you knew you would be privileged and respected before even stepping in the door. That meant getting more resources, having more leeway and freedom to do what you want.
I also get the feeling hackers and "creative programmers" are not as valued as much in today's market. Managers want "I" shaped as opposed to "T" shaped hires.
Any sources for this claim?
https://news.ycombinator.com/item?id=36377805
Ok technically 10 days, but still.
I’ve been making this point for years. I think it’s telling that other nearby disciplines are bifurcated. Electrical engineers vs electricians. Architects vs builders. Etc. The virtues of a good electrician are that they’re reliable, have good customer service and they work efficiently. A good building company can bring a building up in time and in budget. But beautiful architecture work doesn’t share those values. The best architecture is bold and creative, and still has people talking years later.
I think this split already exists in programming. We just don’t admit it to ourselves. It’s different work inventing React vs using it to make a website. Inventing react, or rust, or redis requires considered boldness. It takes time to nurture the right insights to make it right. In contrast, the virtues of a good web consultancy team look more like the virtues of an electrician: Good customer service. Clear estimations. Work delivered on time & on budget.
But we’re so enamoured with the idea of ourselves as “elite hackers” that clock in / clock out programmers can’t give up the mantle. I get it. But it’s stupid. There’s no point evaluating candidates on their ability to reimplement a btree from scratch if their job is to style buttons. You’re evaluating people for a different job. Test their software estimation skills. Their social skills. Ask about workflow and productivity.
Essays like this one ask “where did all the hackers go?”. They’re still around, they’re just harder to find now. They went into weird crypto projects. Llama.cpp. Obsessive database projects like scylladb. They’re inventing rust, and finding security vulnerabilities in io_uring for that sweet Google bug bounty money. They’re in the demoscene or making programmatic art for burning man.
Do you need real hackers at your company? It depends. Are you making an apartment building (on time, on budget) or the Sydney Opera House - which is iconic now, but was almost never finished because of delays and creative disagreements? They aren’t the same kind of work.
Damn, that burns. It’s true though.
> Two, the software talent market is bifurcated. There is basically commodity development of crud apps, and technically complex novel development.
This I think is the source of most of the wheel invention. Someone pays a good deal of money for 'talent' they do not actually require, and they end up working at cross purposes. Assign someone with cleverness to work on a crud app, and they're bound to try to reinvent something, if not to keep their resume fresh, at least to fight the boredom.
But it's not bifurcation, it's trifurcation. It's not build or invent, it's build, buy, or invent. We are too many of us writing applications that were never really needed in the first place. Not for any single reason, but a whole host of them, from empire building, to rent seeking from people who could have made a tool that adapted well to customers, but saw much more money in keeping them engaged with you instead of operating on their own steam. Open source is also driven by a lot of motivations, but 'your own steam' is a pretty compelling one.
Nothing personal, but you are who I'm talking about when I say nerds who just like to build toys in the sandbox. I don't think you intend on improving technology, or making a better product, as much as chasing that tickled feeling of sitting and writing. It's just amazing to me that people still get paid very well for effectively writing a copy of a copy of a book, while most fiction writers barely scrape by. You'd think the people paying us would just use the last book and not pay you to write it again, or demand it be better. But we're all fine with photocopied monotony.
> It really comes down to what you prioritize. Do you prioritize the output and the value added? Or do you prioritize making your engineers feel like "rockstars"? If you don't care about your end product, then yeah sure let your engineers break prod using untested technologies.
First, the post is about sentiment. It's a response to a piece called "Software and its Discontents," about how people feel. So it's going to focus on that.
But I also think the narrative of people working on a project does impact its success. I don't think you pick EITHER "make developers feel like they're doing great, innovative work" OR "do you care about the end product for customers"; the key is to find a way to achieve both, because they feed off each other and are related.
It's as if I asked "do you want company growth, or customer happiness?" Try both!
> The reason playbooks have sprung up, is because 99% of use-cases for businesses are for the most part "solved." We're not in the era of renting colos and manually setting up MySQL anymore (which the author advocates for in order to return to "creatives," btw). We have managed solutions that you can throw money at.
I disagree they're solved! Again, this is in response to a 3-part series basically saying "everything is expensive and everyone is miserable and fewer businesses are succeeding." From a profitability perspective, when was the last Google or Facebook made?
> And good riddance to all that too. I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone just because they can implement a CRUD app.
To be 1000% clear: I also hate(d) the machismo/showboating we got from the rockstar/ninjas era. I'm a former theatre artist and I'm mostly about embracing people and creativity.
"Rockstars" make tens/hundreds of millions or even billions. A good software engineer can make as much as a doctor (in the US) or accountant. You don't have to be a rockstar to be a 400k/year accountant.
The pay in tech has never been high enough to justify the sort of extreme talent competition in sports and hollywood. Even some of the greatest software engineers (e.g. Guido Van Rossum) are paid pretty modestly in the low seven figure range.
Real world Rockstars like Taylor Swift live a glamorous lifestyle, travel the world, and tend to work short, 2-3 hour gigs. Nobody with a rockstar personality is going to sit in a flourescently-lit office building staring at screen for 8 hours a day. No matter how many ping pong tables and free kombucha kegs there are, it is a very dull environment and does not attract the sort of ambition that rockstars have.
I feel like the tail is wagging the dog a bit here because if what you’re actually trying to optimize for is building >$100b business, hiring rockstars or creating a rockstar culture is just one aspect of getting there which you may not even need.
Obviously, both are "good," but again, what's the end goal? To truly be the problem solver, you must make the hard decisions and tradeoffs. Companies trade-off growth versus customer happiness all the time. Right now, a lot of the largest companies are the ones that prioritize growth.
To loop back to the point, obviously employee morale and a good end product is "good" and can have synergy, but again, what's the end goal? At some point, you must prioritize. A lot of companies have been very successful prioritizing the end product versus making their engineers feel good, which is why I think this topic recurs.
> I disagree they're solved! Again, this is in response to a 3-part series basically saying "everything is expensive and everyone is miserable and fewer businesses are succeeding."
I some ways, I think the 3-part series and your article are basically saying the opposite. The 3-part series acknowledges the market conditions and overabundance in the past, and says that in order to get the same results, we need to be smarter moving forward. Whereas your post is (in general) saying that it use to be really good in the past when we were treated like rockstars, we should return to that.
And returning to the point, technical problems being "solved" is not mutually exclusive with a general market downturn. That is to say, you can't point at the general market downturn to say that deploying a CRUD app is not a solved problem.
And this gets to my main point (which I think the author of the 3-part series would agree with): returning to the "rockstar" era will absolutely not bring software companies back to the crazy valuations and money. The "rockstar" era was the result and not the cause of the crazy valuations of the past.
We built a lot of in-house infra. We had few runbooks. We ran tons of high-scale services with a pretty low corresponding headcount. And we had a great culture, that was definitely more personality based than the rest of Big Tech. Individuals had Opinions (TM). On-call was about having a durable understanding of the system and sleuthing around to figure out what was happening (along with a strong emphasis on learning from failures and putting systems/practices into place to fix those.) And we, as engineers, were weird. My tech lead rolled into work at 11 AM every morning on their skateboard. I used to take a 2 hour daily break from 2-4 PM, go home at 6, and work again between 8-10 PM (ish, this was flexible) before I went to bed and was known to enjoy coffee with fried chicken. Our infrastructure and our services reflected our personality quirks and the interpersonal relationships we had in the office. On-call was one of my favorite things at the company because all the different personalities would weigh in with their different quirks and would offer different perspectives on big problems, and together we'd solve tough issues.
Somewhere along the ZIRP-age this all changed. We needed to become a Real Company (TM) now. We needed to have runbooks. We needed to have Boring Technology. We were poised for huge growth after all (aka our stock value shot through the roof.) The new managers we brought in needed to build out new teams to grow fast, fast, fast. How can we grow when we had such immature processes? We started building runbooks and hiring teams to manage them. Incidents started requiring 3-pages minimum of paperwork to document, which was enforced by an incident management team. Teams dreaded these incidents and where once we collaborated to fix problems, now teams became defensive and combative during incidents. Now we needed the incident management team to force engineers to cooperate during incidents because each team was trying to be as defensive as possible. Managers stopped thinking in terms of individuals with strengths and weaknesses and started thinking about headcount, both cost and allocation. Headcount became everything. With this change, attrition also spiked.
What used to take 1-2 engineers to spin up over a week or two now took months. We load tested our new services, but now we needed to make load tests repeatable and runnable in a runbook. Teams became extra defensive about features because the cost of every incident became so high. Nobody wanted to be the team that missed an integration test which came up as a cause of an incident. Our net reliability didn't actually change though the thinking was that repeatability would allow more seamless swapping of headcount. Program managers managed migrations and we started creating status meeting meetings which rolled up statuses of multiple ongoing initiatives cascading over multiple teams.
My own experience at the company went from having fun writing code and interacting with quirky folks at work to dealing with engineers who were dotting their is and crossing their ts in every aspect of their job. The managers treated us as headcount and so headcount we became. It's been a highly depressing arc to a job I loved and where I built a lot of high-scale code at, but perhaps the most frustrating has been watching our velocity decrease despite our headcount ballooning due to the overhead of programs, migrations, and incident management that developed their own bureaucracies. The saddest part is that the oldest parts of the company have become so essential and the bureaucracy so thick around them that replacing them has become really challenging. The parts of the company that were developed with the least care are the hardest to replace.
The age of toiling away with weird tech in a dark corner and then expecting the team to be amazed at your contraptions is 2000% over. I've definitely been through this lesson a few times myself. It works and works until it doesn't and then one day you find that you are left holding a really big bag. My new ego trip is watching junior team members quickly ramp and become productive. Maybe you don't have to turn that ego off at all. Perhaps you just need to find a way to redirect it and get it hooked onto a new, more productive target.
If you want to go try crazy new stuff - do it on your own time. I really don't understand why this is controversial. It's the best of both worlds. I've heard some excuses for why this is infeasible but they're super-duper bullshit to my ears. If it works on your home lab and it is obviously a cool/valuable thing, then maybe put together a 5 minute pitch deck and present it to your manager/cto/etc.
Effective teamwork in this setting is just as foundational as skills they acquired in school but they only begin learning it after they graduate.
Very rewarding to contribute to the growth in jr devs teamwork when it happens.
I'm not close enough to the metal for those services/infra teams, though, to be able to completely tell if the FTE hours spent on all that cloud stuff are necessary. That is - can you throw stuff into the cloud and set-it-and-forget-it and not deal with ongoing cloud/infra/config maintenance? But one of the main things those teams seemed to often focus on was spend - if you leave it unattended is the cost just going to eat you up? But then there's a big irony.
The main costs in terms of labor end up being making sure that backends services are updated. Same pain you’d feel in a colo, except you cloud provider may force an upgrade you’d otherwise wish to put off. Now I intentionally kept most things at the VM level of abstraction, because it was clear that the added complexity of something like Kuberetes wasn’t worth any savings you might get by needing one less server, and I had enough granularity of healthchecks to just automatically spin down any server that was causing problem and let autoscaling work it’s magic. This was also 10 years ago, some choices might make less sense today, YMMV.
Also, don’t think I’m saying all those people aren’t necessary based on the current needs of the business. I just knew I was in a very constrained environment and prioritized anything that allowed for the constant shrinking headcount my department was facing.
In the very quote you lead with, they link to another post[1], which says "...MySQL is boring. Postgres is boring." Our author says they don't like that Boring approach, but then they advocate for a "pets, not cattle" approach, which I think is in line with administering a MySQL server, or such.
Maybe this all comes down to different people's definitions of Boring. Maybe modern developers think of "Cloud everything" as the boring/playbook approach, while other people (like me) see the older (in my mind simpler) idea of administering a few machines, maybe even on bare metal, to be Boring.
You wrote:
> So, do you leverage those solutions to help you solve your problem? Or do you let your engineers re-invent the wheel so they can feel smart?
In my charitable interpretation of TFA, I think they are saying to let your engineers use smaller-scale solutions/technologies that address the problem at hand, as needed, rather than getting pulled into the Cloud-everything solve-all-your-future-problems-you-don't-even-have-yet ball of wax that's often advocated these days.
I can get behind that sentiment, but yeah I'm not a big fan of the way it's framed in the article, and I have never liked the "rockstar" term.
> 99% of use-cases for businesses are for the most part "solved."
History is over, technology is done advancing, every business idea has already been had and executed well, everything that can be done has already been done. Sure. And you would have said the exact same thing last year, and 10 years ago, and 50 years ago, and 400 years ago, and you would have been monumentally wrong each time. But June 29, 2023 is different. This is the first day of the end of history.
By the way, the quotes around your word "solved" stand for many things, including paying through the nose for ineffective B.S. jobs that only exist due to inertia.
> do you let your engineers re-invent the wheel so they can feel smart?
The Curiosity Rover's wheels, which were re-invented for that purpose, are getting holes in them because the kind of rocks they expected to find are slightly different. Perseverance had its wheels re-invented again to fix this. Someone should have told those so-called rocket scientists at NASA that they're only trying to make themselves feel smart! If we never re-invented the wheel, we'd all be driving around on carved stone. "But you're not NASA, loser!" you say. Why not? Why aren't you doing something new too?
> the talent alone is the difference between millions of dollars
This is still true. One single talented coder can be the difference between a multi-million dollar business succeeding or failing. I have no idea why you think this is not true anymore; my guess is that you're completely unable to recognize these people when you're around them. Don't worry, most people can't recognize them.
In before: "wow, you really think you're an underappreciated rockstar, huh?" -- potentially. Of course I can't prove it over the internet. But rockstars are as much of the product of the environment you put them in as they are their own brains. Maybe rockstars actually are more common than you realize -- maybe you're the reason they're not rocking out.
> If you have a really talented coder who can be interchanged with another really talented coder, you do not have a "rockstar."
Or maybe the shackles and chains you're putting around these talented coders are forcing them into the same sub-optimal modes where they can't shine. I suspect if you really fostered an environment where software engineers could grow, learn, and shine -- where you fucking TRUST them, in other words -- you'd find they're not so replaceable as you think.
> I don't need to work with any more software engineers who think it's okay to be a massive dick to everyone
It's never okay to be a massive dick to everyone. In my experience, dickishness is negatively correlated with skill and talent, not positively. This is not an argument against highly talented programmers.
I had the pleasure of pair programming with ye olde rockstar for two weeks and we added more value in those two weeks to the company than the rest of the team did in a year.
I'm the cto I should know.
I only rockstar when my incentive structure has an unbounded ceiling. If you're paying me a salary, plus meager bonus, plus some bennies, plus some lottery tickets -- you're gonna get me at my 40-60%. Just enough to be above average.
Throw in some profit-sharing, and now we're talking, baby. What ever happened to that? Share some of the pie and I'll jump as high as you need me to.
Anything else is just foolish!
Nothing wrong with them doing this, of course. And they're very smart people. But you really can tell the difference between "software developer who takes pride in their work and wants to build something great" versus "software developer who wants to pay their bills". It's the difference between the developer who responds to a packaging problem by learning their build and dependency management system inside out to understand what's going on, versus the developer who copies and pastes from SO/ChatGPT until it works for some reason and moves on with their life.
Granted, not all software development companies deserve the former. But a tightly focused startup with 50 developers from the former category can absolutely dominate the market, at least until they get acquired, as we've seen so many times before: e.g. WhatsApp, Instagram, OpenAI.
I don't think it makes sense to talk in terms of absolute numbers. It depends on the size of the company and what they're trying to build. It's more like "n good developers are faster than 10*n shit developers". If you have enough essential complexity in what you're selling to your users you could realistically have 50s or 100s of developers where every single one of them is like "startup grade" and has their own projects with their own responsibilites, their own autonomy and low communication overhead.
The problem isn't companies that have 200 devs. It's companies that have 2000 devs that could run with 200 (or companies that have 200 devs that could run with 20, or companies with 20 that get outcompeted by 2 people in a garage). I've seen all of them at some point in consulting.
it's possible in theory
Curiously, the company with the higher cloud bill also had the lower ratio (spending more on both, but proportionally more on engineering); my diagnosis after the fact is that they were building for a "do 10x the scale" future that was never realized but which would've pushed the cloud spend higher without needing more dev spend. In the future, I wouldn't spend so much that far in advance of actually needing to scale.
On what basis are they smart?
I like this framing, thanks for it. Typically, when something is suboptimal, there's one argument that says "go back to before we [X]-ed" and another that says "no, we need to [X] smarter."
Illustrating this, [on another post of mine, I discuss how Alex Russell effectively says "SPAs/React are terrible" and Laurie Voss has a rebuttal of "no, we need better frameworks, smarter frameworks!"][1] Another example is when FB was in trouble of Cambridge Analytica stuff, it seemed like the answer to Bad Facebook was always More Facebook.
I'll look over it again, but I didn't get a giant message from Kellan's posts of "we need to be smarter moving forward," more about "how we got here." And I certainly don't remember how we're supposed to be "smarter moving forward."
But I also don't think Kellan and I disagree too hard on anything actually concrete. From [the footnote][2]: I think we both want technical decision-making to be grounded in business reality. I think a lot of scared leaders took the "Boring Technology" slogan to mean "something that doesn't scare me," which is IMO in practice was variations of "we won't innovate" / "we'll operate like money will always be free" / "we will use the most popular things I see in the blogosphere."
---
Said it in another comment, repeating here: I'm regretting that "rockstars" is in the title . It was meant to support the idea that tech was more of a "creatives" industry (because the organic use of "rockstars" to mean "great talent" is indicative of this); but I think my "break playbook and embrace creativity" is being read as "let's go back to rockstars/ninjas! Genius hackers who don't sleep and just HACKHACKHACK!!!!" which is not how I feel. What I'd like to go back to is taking some technical risk when there's a good justification for it, even if it's not what Uber did for their problems.
---
> That is to say, you can't point at the general market downturn to say that deploying a CRUD app is not a solved problem.
Well, many developers who were around in the Heroku era think it's harder to deploy a simple CRUD app than it felt like it was then :-p
But I'm also not talking about CRUD apps, I was talking about building a cost-efficient technology org for a better business. That's why the examples I used (deployment pipelines that costs tons of money and entire teams of developers to build and run) tended to be expensive. Many in the industry called these things "solved," but mostly ran giant bills. Your initial comment said "you could throw money at it," and I never thought that was the way, but especially now that money isn't free.
---
> And this gets to my main point (which I think the author of the 3-part series would agree with): returning to the "rockstar" era will absolutely not bring software companies back to the crazy valuations and money. The "rockstar" era was the result and not the cause of the crazy valuations of the past.
Again (and it's on me that this isn't more clear): I'm not saying "RETVRN TO ROCKSTAR," it's more "I think we should move the slider a little back to the "creatives" era." But while I don't know if it will create crazy valuations, I think it _can_ lead to happier developers, doing better work, and probably more profitable businesses, even if their market caps are lower.
[1]: https://morepablo.com/2023/02/little-piece-on-the-great-divide-of-javascript.html
[2]: https://morepablo.com/2023/06/creatives-industries.html#footnote1> Nobody wanted to be the team that missed an integration test which came up as a cause of an incident. Our net reliability didn't actually change though
This caught my eye because I’ve seen similar phenomena as well. No matter how well meaning those processes are very rarely do apps get significantly more reliable. It still seems entirely down to how complex the software is, it’s very hard to integration test your way to multiple 9s if it isn’t simple (some companies do it, but at bigger cost and slower velocity).
As for some other points…
I’ve come to believe change in culture is really something inevitable after a certain number though. And that number for an engineering team is not even hundreds, but may be only dozens. You operate less as a team and more like an army, with all the personnel issues that come with it.
What interests me lately is the idea that, for many companies, 80% of the value/growth seems to be delivered when the company is relatively small at X headcount. Then increasing the headcount to 5X does not deliver much more than incremental changes built on top of the core product.
Now if you have that many employees you need to find things for them to do, like managing themselves, creating processes or going off on new market bets, and you risk losing focus.
But at the same time a company with 5X headcount seems more attractive to investors than one with X. The measure of success being only an IPO means there isn’t much pride in being well-run, small and focused, admitting that you’ll never be the top player in the market. That precipitates changes in developer culture.
I agree. We should have had a cleansing cycle during the financial meltdown and cleared off all the madness. Instead money was printed and the party had restarted with even more stupidity.
During that time I was earning almost 2000 euros, in a time where earning above 1000 euros was a dream for many Portuguese.
Wasting money on CDs, theatre, going out, travelling, first time I managed to buy a desktop PC without being on credit,....
Then came the dot-com crash.
I remember a ridiculous TV commercial where a stereotypical pointy-haired boss guy was talking about hiring programmers, and he said "a candidate told me he dreams in code, and... I hired him on the spot!" And it was unironically framed as if the boss guy was doing an amazingly smart thing!
It was a crazy time.
These are different skills and responsibilities from being an elite rockstar developer.
Oh, and I still code. Because I like to code. A lot.
Now, I’ve also hired internationally from all over world and that has usually been fine.
What would be the difference between offshore and hiring internationally?
It's the same term that was used for transitioning textile and other factory work to third world sweatshops.
Hiring internationally is just that. You might get someone for a bit cheaper (somewhere in the neighborhood of 20ish percent), but that's usually in exchange for the extra hassle of managing a remote team and dealing with timezone issues. You don't usually crank through these people every quarter, they're hired for the longterm (or, as longterm as anyone in a software job is hired for).
Both have their benefits. In my experience offshore teams are basically an army that you need to train and provide a lot of direction. It can be very frustrating but if you have code generators, Sonarqube, linters and they are only working on CRUD applications it sometimes works out. Whereas international hires are usually very competent because you personally selected them.
Speaking very broadly here, my impression is that those offshore development shops bidding for US contracts often don't have the greatest working conditions. Those talented enough to know their worth and find better options will do so if the opportunity presents itself, which could mean either applying for international positions on their own or immigrating to where the jobs are. Thus there's an outflow of talent away from those offshore companies.
Hiring internationally, you might have TZ and language issues but none of the other stuff. If you do your hiring right that is, you'll get somebody who actually wants to be part of the team and is product oriented.
It's mercenaries vs. paladins.
One thing that I have found with "offshore" hiring: The very best have _mostly_ migrated to high cost locations because they get a huge salary increase (far more than cost of living increase). There are _some_ exceptions where the candidate is very high skill, but needs to live in a developing country. Usually family or cultural reasons. However, they are very hard to find and harder to keep.
In comparison to hiring individuals internationally to work with an internal team.
In the first case the entire project is "offshore." In the second case only individual team members are not in the primary country.
As relevant today as when it was first published, in 1975.
I consider him a giant in software, but because he was "IT Focused" he was not as loved as others.
You’ll be switching jobs quite a bit then.
Expectations goes both ways. The companies expect the ICs to ship code, and the ICs can expect to work with competent managers. Pay attention when selecting your team.
Definitely agree on this.
> You can’t just hire rockstars to work on something low margin or without significant demand and expect because they’re amazing engineers that you’ll make a lot of money.
This isn't what I'm trying to argue though — I think when you reach for the ceiling instead of avoid a floor, you're more likely to find those tremendous business value opportunities. Consider what Google shipped/built under Schmidt, "20% time," "Don't be evil," and "we're a different kind of company" vs… the last 10 years?
I'm not saying "being wild and creative makes a bad business good," I'm saying "embracing some narratives of a creative industry may increase output and be more cost-effective, even if playing "outside the playbook of the 0% era" is scary."
---
On another note, I'm regretting that "rockstars" is in the title . It was meant to support the idea that tech was more of a "creatives" industry (and the organic use of that to mean "great talent" is a symptom of that); but I think my "break playbook and embrace creativity" is being read as "let's go back to rockstars/ninjas! Genius hackers who don't sleep and don't care about HR!!!!" which is… not how I feel haha.
To your point about Google, that approach did lead to gmail, but also many abandoned projects like Buzz and Reader that led to Google’s infamous reputation for canceling products.
My original point remains: I think the business requirements and goals should inform the culture and hiring processes of a company. If you don’t need to deviate from the playbook to get things done, why should you? And not every developer wants to deviate from the playbook either, some prefer stable predictable work - while I’m not personally in that camp I think it’s not a moral failing, doing unpredictable things is more stressful as more things can go wrong, you might have to work more or suffer the consequences of delays/starting over, etc.
If you’re a bank, or a web consultancy, you want boring and predictable solutions. Conversely if you’re OpenAi you want people willing to reason from first principles and invent crazy new things because you’re shipping cutting edge technology at massive scale - and I think they’re doing that. So I guess I’m left wondering where exactly you think opportunities are being left on the table due to overly rote software development practices that would be improved by creativity?
Maybe I'm just old, but when I think of a rockstar I think of a lead singer or guitarist that has spent most of their life practicing their craft and loves it so much that when we see them perform it's just a taste of what's inside of their head and heart.
They are just rarely mentioned on HN because there's already enough tech billionaires to exhaust the patience and attention span of most readers seeking out that kind of content.
Most, if not all, Google fellows? Just randomly pick one.
I ask because while I don't reject this position, isn't there something to be said for intrinsic vs extrinsic motivation: one's internal notion of excellence, pride, discipline, respect, mastery, craft, grit... insert various honoring-self words.
Seems precarious to me, to let external dictate ones concept of value. Btw, I suffer from "doing my best" so that's why I'm curious, I am not shilling for BigCo.
I do not have notions of excellence, pride, discipline, respect, mastery, craft, grit, or anything of higher substance inside of me. At one point in my life, I'm sure I did, but they have been beaten out of me. Some have been taken advantage of by the ulterior motives of others; and some have simply not been conducive to achieving my goals (i.e. having fun, putting bread on the table, not ending up in the streets, living a leisurely and relaxed life, etc.).
In another vein, one could make the argument that the precarious thing is to tie one's intrinsic motivations -- those little devils inside of us -- onto a "job." That job will one day end, and you will have to find another as an effective outlet for those little devils; otherwise depression and all sorts of psychological illness will start encroaching. If one wishes to keep one's values strong, why make them dependent on uncontrollable things? Unless you are independently wealthy (if that is the case, I can offer nothing), there will be times you will be required to make sacrifice of your values to "get the job done." You will have to be sloppy, amateur, lazy, selfish, and maybe even villainous to do what needs to be done. At that point, you can hold strong and suffer the consequences, or you can relent, tell yourself you are temporarily setting aside higher ideals (it never is temporary), get the job done... and suffer the consequences.
Tying your soul, your essence to such a thing is precarious. Putting your heart into things that will require you to abdicate your integrity is foolish. You will end up hurt, broken, and missing pieces of yourself. It is better, and more sustainable, to save that sort of thing to matters that aren't directly related to your survival. Otherwise, to charge steadfastly into what one knows will be suffering is noble, but foolish.
Excellence is not in contrast to fun, leisure, and relaxation; it's made that way via Capitalism and hustle culture.
> In another vein, one could make the argument that the precarious thing is to tie one's intrinsic motivations -- those little devils inside of us -- onto a "job."
Well said. I suppose it does take more energy, effort and soul really, to constantly decouple one's job from identity and worth. That one's internal values _ought_ to permeate through one's life "authentically" is certainly easier said than done.
Thanks again for responding in earnest. I'm convinced lol.
to ignore incentives is folly. they drive the behavior of others around you as well.
bad incentives lead to endless kingdom building and mediocrity. good incentives lead to a brighter future.
We had very, very little overlap to have meetings (unless one of us wanted to wake up early or have a late night meeting, which happened a few times). We did everything async, even meetings and decisions.
The best part would be learning how to open a WIP PR, with enough detail that someone else could understand what you were going to do. You'd wake up in the morning to find a team member had contributed to your PR, either by fixing nits (we didn't have time to go back-and-forth on shenanigans or it would take weeks just to merge a single PR), or by actually implementing some idea they had. So, if you didn't like it, you'd just remove that commit from your PR.
It was fun, but it takes a lot of trust!
It was awesome. I'd turn up to the customer site to collect their issues and requests, do some investigation, formulate the precise requirements, and send it off to him.
The next morning, magically, things would be done.
You don't necessarily need to quantify that exactly. If you've worked on a team with people like that, then you know it. Their managers know it too.
To your main point, the size of the markets is drastically different. There is only so much room for headline rockstars and entertainers, but there's way more room for talented technologists. Having "rockstars" in the engineering world doesn't really reduce the size of the pie for others.
If you can't prove your superior productivity, you are not going to get paid for it. Which is going to incentivize you to optimize for your company's promotion rubric instead, if you don't get fed up first. This leads to the lamented "promotion-driven design".
Or you can ditch the middle man and take your talents straight to the market, through consulting or incorporation.
The person who understands and can solve your problems effectively is more valuable than a clueless person who cannot. It doesn't matter how many hours they spend in the office. What matters is technical knowledge on the sometimes confusing topics that arise at work.
We can take or reject projects, so if you are a bad predicter of use/time to completion, you are going to do worse.
Or I'm a manager. I help my engineers get work done. Let's consider the happy case and assume we can measure our team's revenue. How much of that revenue is attributable to the manager's superior skill, and how much to the rockstars under him/her? What would an experiment look like; the engineers manage themselves, or we compare with the previous manager?
No company that I know goes this far. In their defense, it is a challenge. An open problem, even.
You wouldn't run an A/B test to quantify your impact, but you could pretty easily do before/after metrics (or just collect them on the way during a graduated rollout) for any feature you're rolling out. It should be straightforward to quantify productivity benefit for a large initiative.
> How much of that revenue is attributable to the manager's superior skill
The output of a manager is the output of their team. We can't A/B test managers (or at least, we probably shouldn't), but we can look at a cohort of managers and see how well their teams are generating business impact. Combined with other data (e.g., upward feedback surveys, level-skip 1on1s, retention metrics, etc.), we can measure the impact of a manager.
It helps that we are often saving ~400 hours/yr spending ~80 hours or less.
We can also see it in overtime hours being eliminated, and the (real) engineers are making more complex designs but the size of the team hasn't changed.(30% more complex in 2.5 years, same team size) They also havent replaced engineers that left the team.
Basically: We save enough hours that it is obvious. There are added benefits like not having humans do manual checks, so less prone to errors. (Although I argue that our bugs are equally as dangerous)
Making sure those $ saved didn’t screw things up somehow is the hard part to measure.
And - this is going to blow your mind - but sometimes you can actually make things more efficient by spending more money. Cutting costs can be the exact opposite of the right thing to do.
So while I am jealous that your productivity can be measured so simply, I’m dubious about how many programmers’ contributions can be so easily reduced to a simple metric.
Luckily for the throwaway person above, it says the same for them. But does still show results for some, maybe only big accounts now?
https://news.ycombinator.com/item?id=27568709
edit: ahh, I think it just took a one-time snapshot of the three years leading up to its 2021 creation, as my old username that I changed to swores does show up. So it's an interesting proof of concept that someone could do similar analysis, but not an ongoing tool. So it won't be telling us which CTO to avoid :P
Edit: I think it was this one that has been shut down: https://stylometry.net/
> We need to keep track of the time your spending on your opex KIPs. We can't pay your expected bonus if you don't.
IME when I've seen a "1/10th engineer" it's almost always been a total management failure.
Turns out reading HN comments is not a good way to run a company. Who would have thought?
Just to confirm, does that person code all day there? (real question)
Next to that is a constant overhead of talks with leadership and supporting other engineering efforts throughout the company through things like doc review, chats, pairing, etc.
What he doesn't generally do is "productionize" features - code is generally exploratory in nature or fixes a serious bug (or class of bugs).
I have seen engineers at companies 1/1000 the size of Google generate $4 million in differentiated costs savings (definitely would not have happened without them). It seems plausible to me that someone could generate 10x that at Google.
Hmm, does come up every now and then.
> eventual consistency
Have seen these event-based maintenance nightmares, trying to steer clear wherever I can.
> clock drift
Never met a developer who would at least hear about some other developer for whom that would possibly ever matter.
I know a dude who got a nice package from a pizza company to be CTO + lead programmer. I think he has a few people under him, but the bigger deal was that he could get a percentage of the $ saved.
He worked all day and night to get LLMs to take orders, in a month 40% of their orders are taken through the LLM. For a company with 100s of stores, this is a huge cost saving.
Guy knows how to program, he knows technology, and he knew the domain from a previous job.
You and I and probably half of HN could do the job of "CEO of a mid-size tech company." We don't, not because of lack of ability, but because we don't tick those culture/personality/background checkboxes.
You're not going to leetcode-grind or PhD your way into a $10M software engineering job.
I'd take that deal in a heartbeat. I think I (and most other decent engineers) can probably point to tens (or hundreds) of millions of dollars in savings generated over the course of a single career.
As far as I can tell, it gets frittered away by empire-builders who take that $ and spin up new teams for nonsense. Kicking back a % to the engineers who harvested all the savings seems like a much better model.
At the high end of the white collar market you're paid to deliver impact and your pay is determined by the quantity of the impact and how differentiated it is (no big bonuses for the guy who presses the "print a million dollars" button).
That's not really my experience.
The reason companies hire lots of people is not because of bus factor, but because they're trying to "make 9 women give birth to a baby in 1 month".
The fact that development doesn't scale linearly is very well known to developers today, but product people barely heard of Mythical Man Month.
It astounds me that we don’t see more investment in this because it pays off quickly
Pros get burnt out being constantly called on to crank out new features under pressure and never get to leverage their deep understanding in the maintenance phase.
Non-pros have no upward mobility (career progression) and get burnt out struggling to understand and maintain unfamiliar code without any guidance.
As for pros needing training by other pros, I think it's more a question of culture. Once I was tasked with adapting a microservice to also run as a library, the challenge being we used JSON fields and now also had to store them in a mysql database that did not support them natively. I wrote an adapter that could have been released as a separate library because it was that generic, it took me 1 day of work, a 15 files long PR. But because it used a postwalk traversal on tiny json data triggered lazily upon accessing the specific field, cached in the ORM object for repeated access, it smelt funny to the CTO who proceeded to ditch my work away, spend the 3 following days adapting 350 files to try to get a highly specific solution then gave up, and decided in the end we'll have two separate versions of that system, in concurrent use, one as a lib and one as a microservice, hence multiplying by two the time it took to do changes to that highly repetitive codebase because "it was simpler that way".
You don't need to train pros, they have been training on their own since they are twelve.
I've seen several times so far that people maintaining things are actually doing a much better job, while those who were granted honor of doing a new thing because hey they are "pros". And once they did the prototype, they are _considered_ even more "pros", because hey, they wrote it. The people who rethought and rewrote the thing later were real makers, but didn't get the appreciation of "pros".
The worst thing is that "pros" status keeps working for itself, even while delivering worse job. I wouldn't have a problem if the system could fix itself over time.
Kinda related: "Confession of a so-called AI expert " https://huyenchip.com/2017/07/28/confession.html
As a producer of the mvp, a lot of times there is 'exploratory code' where you're not actually sure how it will work yet; or scale. So you bang out a few approaches and pick the best one for the constraints you have. And basically never touch it again.
In one case, I was working on the mvp and the team who would be maintaining it where the one's doing code review. The code turned out way better than either of us would have come up alone. It ended up being one of the coolest projects in the company and some internal secret-sauce for making money. For example, I remember one code review where they asked to add a configurable callback in. I was like 'why?' and they answered, "we're already being asked if it can do X, if you add a callback here, here and here, we can already start building that feature so you can focus on the core."
What do you think leads to a better result?
- I create software that I _want_ to maintain and _will_ maintain.
- I create software that I think someone else wants to maintain, I leave the project after.
At least in open source, some of the most praised and successful software was built _and_ maintained by the same people at least for a substantial amount of time.
Thinking about it, this pattern of "I, the rock star will build it, you the code monkey will maintain it." Is a red flag.
I've seen plenty of "software that I _want_ to maintain and _will_ maintain" that is complete shit from a maintainability point of view.
I've seen one-offs left by a consultant that are pretty amazing.
The main difference in both cases was that more experienced developers made better and simpler code.
Sounds like the skills aren't very differentiated. IME it's quite rare for a business to miss differentiation in business impact in an order of magnitude.
Going to your earlier question, the candidate in question has many skills but is not overly specialized. He commands that price because he can point at the right place in the stack and say "There's $10 million in savings here" when everyone else misses it or "Changing the architecture in this way lets us ship 6 months faster" and he is able to do this repeatedly.
> I'd take that deal in a heartbeat.
Same! That would be a very exciting and lucrative role. Unfortunately it's basically the opposite of what the industry does today. I think we've all seen the six-figures per month AWS bills to power infrastructure for a workload that could be comfortably hosted on a Raspberry Pi.
If the industry ever goes back to caring about efficiency and cost, sign me up.
> The CEO is in his position because he has the "right" pedigree and the "right" network, went to the "right" school, talks the "right" way, has the "right" ivy league mannerisms, goes to the "right" polo clubs.
None of this is true. The CEO is either significantly compensated because it’s her company (majority ownership) or because the board likes the direction the ship is being steered and wants to make sure she doesn’t leave to steer another ship.
> You and I and probably half of HN could do the job of "CEO of a mid-size tech company." We don't, not because of lack of ability, but because we don't tick those culture/personality/background checkboxes.
What is it you think the CEO does? Half of HN absolutely could not run a 2000 employee firm they are intimately familiar with, let alone an arbitrary one.
> You're not going to leetcode-grind or PhD
Leetcode no, phd yes. The fact that you put them side by side pretty strongly indicates you don’t know what compensation is tied to.Leetcode skills provide limited value to a company. A PhD with a track record of leading ML publications absolutely can put you up there.
Compensation is about what value they think they can get out of you and how much competition there is for you.
Don’t sit around telling yourself execs have trivial jobs. It just makes you look ignorant. Spend some time looking at what each brings to the table for that specific company. Barring nepotism or corruption, there is usually something significant there.
People say that but it makes no sense. That means the same manager would have a different "output" going from one team to another. Are you going to cut his/her salary if he leaves a big, successful team to help fix a flailing team because it has less output?
I get it: doing it this way is easy. My ideal is marginal attributable revenue or profit. How much better are you than the next person? We can haggle over the marginal part but I'm not conceding on attribution. Otherwise you're mooching off other people's work. Why are we paying you the big bucks? Show me what you did.
I think it works fine if we add on the concept that a good manager operates on a different timescale. The on-boarding timeline for a first-line EM can be months at some companies.
In your example, the manager moving over should be given a compensation package based on the expected value he's going to get out of the flailing team once he works his magic. (And, indeed, this is exactly the kind of package often given to externally-hired CEOs brought in for turnarounds.)
> Otherwise you're mooching off other people's work. [...] Show me what you did.
But that's just it. The manager's skill is improved utilization of resources. The leader's skill is improved direction of resource utilization. By definition, those skills are expressed on the canvas of the team they lead.
Or, more simply: A leader without followers isn't actually a leader at all!
And, indeed, we see that in the real world too: A successful executive in one company goes elsewhere, doesn't fit in with the new company's culture, and performs terribly. It's a tale as old as time.
Or it could just be somebody that happened to have a crucial bit of domain knowledge.
Or the original composition of the team could have included some negative-contributors.
one guy for two days = 2
240/2 = 120
Re. scaling -- probably, agree, just I never seen myself (working in distributed systems). On my personal experience it was always more-less clear, depending on what software/database/data types we intended to use. So before implementing an MVP, I could roughly answer what price/scalability it would have. But I can see it might have, especially in fields of statistics/ML/new algorithms. E.g. "try using SIMD for this new sorting algorithm" -- only tests could show if it's any better.
But in that areas, usually managers shouldn't judge by outcome anyway, but rather by how fast they make and test hypotheses.
Remind me, whose job is it to convince other execs and the board of the most appropriate talent strategy? Maybe lesson 1 here is "if you're running a company and don't trust your CTO, look for a new one"?
There's probably nothing wrong with your existing team except it being too large and democratic. One guy can bounce ideas around in his head faster than a dozen negotiating them with each other. I've seen teams like yours, and IME the issue isn't always incompetence, sometimes each of them thinks they're the rockstar. Anytime one of them doesn't get their way, they jam everything up for everyone else.
We've become too democratic as a whole. Nobody follows orders from above; everybody expects to have a say in things they don't even have a stake in. Whenever you get to hiring again, try stacking your team with 2-4 ex-military types and see how it works out for you (it works well for UPS well beyond that scale). You don't get bleeding-edge experience (and never a 10x-er), but you get tax breaks, competence, obedience, people trained to work as a cohesive unit, and you can wave the flag and call yourself a patriot (which helps with government contracts if that's your business).
Literally everyone thought this, and probably even the talentless hacks that you hired. Strange that you had to create a throwaway account to figure this one out.
It's not necessarily that he's that much better technically, but he's got the right drive, focus, and business acumen, and isn't bound by methodology or legacy.
The real problem with software development is that a lot of developers are too removed from the business to deliver something practical with value in the right places.