I accidentally saved my company half a million dollars(ludic.mataroa.blog) |
I accidentally saved my company half a million dollars(ludic.mataroa.blog) |
When I find someone very talented, I go out of my way to promote them because good engineers who want to stick around and not jump when the times are good is hard.
The sad reality is that I’m limited to how much I can bump their pay, give them time off, etc.
Open-source: https://github.com/datafuselabs/databend Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059 Cloud: https://app.databend.com/
(Peter Principle writ large.)
Only one thing left to do. Leave 'sleep(600)' calls all over the place and resign.
1. Remember who you work for. Yourself. Not your boss, not your grandpa who always wanted you to become a writer. You only have so many breaths before you get your ticket punched, so make them count.
2. Don't seek work as something you should love. Those should be your loved ones, and hobbies etc that stimulate you. Sure there's a small minority who get to do what they love, but over time love can turn to disgust.
3. When starting a job, figure out the organization's incentives. Some orgs want change, some will fight any change. Figure out what the org wants and your life will be much easier. My job is like the Maytag repairman, kind of waiting for stuff to break. I refer to it as being a digital janitor.
4. Nothing you build digitally will last. Hell, between linkrot, bitrot, and the heat death of the universe, nothing lasts. So don't expect what you've built to last, or to hold value. It'll all be re-factored/re-engineered/re-architected and become obsolete.
5. Large organizations are toxic and often sociopathic. So are small orgs. If you can find a way to start your own business, try to avoid becoming that way. Good luck, most businesses either fail or become that way. But at least you have more choice in the matter.
6. Find a mentor (sometimes referred to as a rabbi) who can provide guidance to you when times get tough. Don't choose a rabbi from your reporting chain. You will find yourself adopting a worldview that might not be aligned to your own interests (in other words, the bastards will manipulate you). This rabbi can be a friend, or just someone you vibe with. Be careful in choosing your rabbi, and take care of this relationship.
7. Illegitmi Non Carborundum
Should have just let them burn dude
I'm sorry, but this needs a privilege/gratitude check. You are guaranteed your salary, and you're welcome to take on the same level of risk your company is by starting your own. If you think it's so easy go ahead.
They didn't make $500k, but saved $500k from a mistake that the company themselves made. Why does that deserve a raise? I've seen people implement features confined to a single file that made over $1m/year but they didn't ask for a $30k raise.
I could see a one time bonus or stock gift and maybe a more modest raise next year if they stick around.
My overall point is this: If you want to make the big bucks, you can go make your own company, but be prepared to take on the mountain of risk that comes along with it. Otherwise, a tech salary with guaranteed income ain't so bad.
Pain I feel very acutely. As an employee, I saved a FAANG company billions of dollars in potential GDPR fines after they carelessly declared 'mission accomplished', weeks before the deadline, and never even got a 'thank you', much less a raise.
When you find shit like this, people will go "my bad" at the worst, and you all are happy the company has more money. Small teams and small firms have their downside, but honesty and transparency, especially when it comes to cost savings... tend not to be one of them.
Illegitmi Non Carborundum.
It's important to remember this.
To give an example, most large (multi-billion dollars) companies I worked for where amazing with happy people, whereas the small startups where filled with people backstabbing each other to make sure to be at the top of the food chain IF the company were ever to make money one day.
The moral of the story is: anything can happen anywhere.
> ...
> I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds.
Wow, so they're still over budget by 300k dollars. This is a funny story, but the company sounds incompetent.
I've seen "typical" spelled this way several times on Hacker News. Is it a British thing or something like that?
Developers - Always do this as bonus paid by contingency of saved money and a signed off scope if it works out, especially if you’re d looking into it on your own time.
Specifically if you like you can get signatures from everyone on the hierarchy on how much money or time this will save, cost or make them.
Do it enough times and the right kind of CEp/President will tell you to stop bringing the business case to them for approval and just do them if they make sense and you have your backup.
You will enter a side door, bypassing most politics and c-levels (and maybe triggering some new), reserved for people who say let me see what’s possible instead of coming back with reasons why it can’t be done.
From there, paint a picture of what if you built a team of only doers, minus talkers across the organization.
:)
Tech should never report into Finance, the group that can’t even tame spreadsheets.
Yes, business owners pay.
I can only testify that this is not my experience at all, so it might be that you're in the wrong place.
In every company I've joined, I've found issues, often large ones, managed to get assigned to them, and had fun fixing them (or at least the satisfaction of seeing them fixed). Then moved on to the next issue.
I haven't always received much recognition from the powers that be (although me and a few colleagues have once been wined and dined at one of the most expensive restaurants in the city for having saved the company 1M$+ with a few days of work each, that was nice), but I've gained respect from fellow engineers and from myself, which was nice, plus lots of things to tell during job interviews.
So, may I suggest getting out while you can and finding a better place to work?
It was a .net app that I reverse-engineered that was basically calling 10 or so encrypted stored procedures in SQL server. Because the older versions of SQL don't actually use encryption, but a type of encoding, I was able to decode them all and replace the entire app with a script that I wrote.
The whole process with testing took about a month (they thought it would take me a year).
The consultant that made it wasn't very happy when he was told they would no longer be needing his services.
Fire the majority of people leaving the A players, unmanaged, and the world will begin to work near-flawlessly in a few weeks to months.
But this line of thinking hurts feelings which are (presumably) paramount to actually solving the problems people set out to solve.
Am willing to bet it was 'KTMJ'... (Batman reference, but... strangely similar to one of the "big-4")
There was a tale maybe 10+ years ago about someone who automated their job with a script or Excel sheet or macro and didn't tell anyone about it. Having a hard time tracking it down again, anyone remember what that was?
- Last year, there was a post on Reddit, with an HN discussion as well: https://news.ycombinator.com/item?id=29994776
- In 2017, there was a question about whether someone should tell their employer that they've automated their job: https://workplace.stackexchange.com/questions/93696/is-it-un...
This is Dilbertland, at its finest.
I was baffled and urged him to be careful and assume he was wrong that it had to be wrong.
Anyways, he is still fixing it, getting some people to validate it.
I would think he will save at least hundreds of thousands a year.
But seriously, why a company that spends millions of dollars in an area would not hire an expert to try to save some money in it is beyond me.
It doesn't make sense until you realize that they operate in an environment where there is no competition that can disrupt their business model. If their costs go up, their rates go up. There is never an incentive for costs to go down.
In short, I was revamping/enhancing a property insurance policy pricing system as the lead from the business side (i.e. portfolio manager/actuary).
However, much of the development team was truly incapable of writing software in any useful way, so I inevitably dipped into C# and SQL to help diagnose issues as we (as a team) worked on the enhancements.
My business coworker and I found something that remains funny to this day: code littered with references or attempts to use the .NET TPL library to make the platform do more things "in parallel," but without actual knowledge of using such a library in practice.
Nothing worked in parallel! There were blockers all over the place! As someone else said, TPL might well have been a //TPL TO DO.
We suspected that TPL was there as one of those "look what I can do" intrusions that received praise from more senior devs/mgmt. even if it never actually did what it was supposed to do.
I can relate to this.
I think many of the people who disagree have either been serving in non-engineering roles since past couple of years, or have bought in the pitch of rosy world created by managers and so-called leaders.
I threw all of their configuration advice out the window and run all extra small at 1 minute idle. No issues for my use case and we paid 17k for the year.
I have to expect that snow will change their pricing model soon.
To put it another way, to rack up $500k spend for 30 days of constant provisioned time:
$500k/(30*24) = $694/hour
Presumably, with the 10 minute blocks, there was idle time where the spend was zero, so the instantaneous amount would be higher.
I have never used snowflake so I'm not sure how this works.
If you keep the warehouse running, there's a fair amount of caching going on behind the scenes that can make your queries run faster on the warm warehouse (how much faster depends on the query) . This article shows one example - a cold query ran in 20 seconds, after caching it ran in 1.2 seconds on a warm warehouse:
https://community.snowflake.com/s/article/Caching-in-the-Sno...
So you don't always want to set the idle shutoff time to the minimum.
Well duh...of course you'd want that. But it is not so easy is it? I'd also instantly fire half my staff and hire hard working geniuses instead. But that just isn't an option, unfortunately.
> I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened.
I know he doesn’t work the same place I did but damn, it sure feels like it.
Definitely a pattern through the places I've worked. The ones that are smart, have good intentions and go a little bit rogue tend to make the biggest impact.
Executives are too concerned with risk, and honestly, it playing out this way suits them better: if someone goes rogue and it works, they can claim the win, if it doesn't, they can blame and reprimand.
Also, since you're paying by the second, it can be cheaper to keep a warehouse running longer to avoid having to re-warm the cache with every query.
Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
The worst thing I've seen is a stack that parses out a file and loads it into a DB. So someone sends us a file via an expensive SFTP+S3 thing in AWS. That is then picked up by some scheduled task using a proprietary in house scheduler process running inside kubernetes. This proceeds to download the file to the local pod. Then it makes tens of thousands of API calls to match up data which cranks the CPU up on a huge database server. This breaks all the other jobs running. Then it writes another file out to S3, consuming 17GB of RAM in the process. Another process picks that up and then batches it and inserts it into the DB with no transactional stuff around it.
The original process this replaced was a copy into a temporary table and then a bit of transaction-wrapped SQL that took about 20 seconds to import + run. They improved that to 7 hours and reduced the success rate from 100% to about 80%
Literally instead of a table.
This stuff is everywhere. Microservices made it worse and half legitimized it.
Just two examples from my prior gig (fashion e-commerce).
#1 Our hottest dataset (db of current products) stored in DynamoDB. Core dependency for all our code. Easily fits in < 1Gb of RAM. OMG, just make a hashmap. Over a year, I managed to persuade the team to start transition from DynamoDB to Redis.
#2 Tiny (vs micro) service that munged some URLs. Blocker for an important campaign. Prior team of 4 churned for a year, was no closer to delivery. Spring, ORMs, CI/CD pipelines, the works. I spent a week unraveling the requirements (repeated facepalm). A second week banging out a trivial nodejs thing. (My team preferred nodejs, which was their prerogative.) Really trivial. I felt so bad for the biz dev people who'd been dying to get this functionality for so long.
"I dont think you are fitting in here at MegaCorp"
Where?
I don't care any more. I'm just there to tell people what's shit and then laugh when it explodes in their face.
But hey, they can run some docker and git commands and piddle around an AWS GUI, which means they are highly technical.
That's on the industry for not training and gating well. It would be nice to have glue/plumber positions so expectations are not out of line too.
They're not usually software engineers. They're tool users not tool makers.
So they'll cobble things together to accomplish the task, using only available tools and never anything custom that would do it task much more cleanly, because they understand data, not software. They're not computer scientists or programmers, they're just users. And we all know what that means.
I've spent the majority of my career building better tools for data-related tasks, then winning over my users by showing off performance and productivity gains.
Those replacement systems aren't even working, they're failing to deliver on the same simple data pipelines that we had working by the start of 2020. They're cobbled together using a million little AWS pieces and Docker and k8s... I'm glad that I left that role when I did, we were being pushed by a new-hire with a fancy Data Engineering VP title to do all sorts of asinine things. I went and looked just now and I see that he's Senior VP at a different company, he started there this summer. Onward and upward!
And I thought my unholy xmllint -xpath (bad stuff, lots of slashes) ${1} |sed -r -e s/this/that/ -e s/alsothis/alsothat/ -e /ohyeahthistoo/somethingelse/ | grep something | while read AA; do stuff then echo ${COUNTRY},${SIGN}$(perl -e "printf('%.2f', ${VAR}/1000000)"),${ENTRYDATE}; done|sort
was as bad as things get. I need to get my horror code game up. I mean, not only is the code awful, its very purpose is horrifying (XML to CSV with some transformations, bit of math, all without being able to use any external sources due to security, only what's in a baseline RHEL7 (soon 8, yay!) ).
I promise I'll rewrite it in python at some point.
If that doesn’t exist yet (it usually doesn’t), we test manually, but only core workflows.
This is why I hate recruiters, I can't even tell you how many times I've had a recruiter call me saying they are looking for service XYZ. The same concept rephrased in my resume. I have to rewrite my resume just to satisfy these people? No thanks.
Never heard that before, but that’s so on point.
Came for the engineering, stayed for the blisteringly on-point observations of corporate life
It had me stuck reading until the very end - usually that never happens!!
Meanwhile big tech thinks the way to reduce costs is to wholesale fire 1/2 their company.
Your job is the venn diagram intersection of an organized system of rewards and punishments from both inside and outside of the firm intersected with your own goals and aspirations over time. Anything not in this intersection is not a job, it's something else we like to delude ourselves into thinking is a job to preserve our egos.
More specifically, your worry about how distorted incentives lead to poor quality software is not your job unless it is your job to worry about said incentives. People keep hoping to go work for one of the magically sane company where all the incentives are correct and everyone is doing the quote-unquote "right" thing at all times. Such a company does not exist because the very concept is incoherent, every system of possible incentives will come with a different set of tradeoffs and the art of operating a firm is to pick from amongst a bunch of shitty options for the least shitty one.
People who do not accept this will be perpetually unhappy and bitter about how others obtain undeserved success because they simply did the unsporting thing of playing the game they were asked to play. You can either be one of those people or you can receive the radical acceptance of what a job is.
I've been on the receiving end of one of those projects and once you dive in without the person who pitched it, you see how hollow and nonsensical it really is. But by then its too late. The consultants who wrote it are all paid and the executives who attached their names to it as sponsors keep it limping along for years, draining everyone's will to live.
The other guy meanwhile is doing the same thing again at a bigger company having a great time.
So far I'm feeling pretty good about what I've developed over the years.
Anecdotally, optimization tasks (in this brain) are multitudes easier than innovation tasks. I spend a lot of time thinking about how to do things differently whereas optimization utilizes many lessons I've learned over and over again with well-trodden patterns. That's to say, I'm grateful for the doers :)
Bringing what used to be the privilege of upper management (wasting massive amounts of resources while getting paid handsomely) down to software developers.
It's that trickle-down effect people talked about, right?
Dare I say, any time I see a function as a service my brain immediately drifts to inspecting the cost implications of said process.
I said the above as a jest, but seriously, simplification of complex stacks has been a good consulting gig.
When developing anything:
1) you don't get to touch anyone else's code. And another department's code? Something another manager's team manages ... that amounts to treason. Never for any reason. MAYBE if they've totally abandoned it and you absolutely need it (but only during unpaid overtime)
2) you don't get to spend ANY time on anything outside of the current project or JIRA ticket. Any time at all. So really, NOT optimizing anything is faster and cheaper. Just look at all the spreadsheets made!
I’ve had enough calls with the “Senior/Technical Lead Azure Cloud Engineers” telling them exactly what they need to do that me and them really really don’t get along.
I don’t do any of that shit and even I can muddy my way through it, but these people cannot. The real kicker of it is how much these people make.
And you know how there are those people who, every time you need to work with them, they answer a teams call and then “need to get to my computer, give me 5” and their status is perpetually set to away? I don’t want to RTO at all, but dealing with this team almost makes me think I’m wrong about that.
Agree solidly on the gating aspect though.
The problem is that quality hires continue to be rare and at some level you are doing area-under-the-curve reasoning.
* Spend £5,000: prudent investment in the future
I'd bet there are a 100 different answers to your question though. This is the way we handle it.
What makes someone a staff+ is finding a path to iteratively evolving towards that end-state without breaking anything along the way and while having progress to show off at each step.
There's a service that watches the transaction log of your regional replica so that you can make long-poll HTTP requests that return when any change matching your filter is committed. (Edit: usually the HTTP result handler is used to invalidate specific memoized results in the data flow graph, letting lazy re-evaluation re-fetch the database records as needed.)
It makes a lot of sense for a financial risk system, where you end up calculating millions of slight variations on a scenario. The data flow model with aggressive memoization makes this sort of thing much cheaper.
However, I saw plenty of systems written where you'd attempt to write your request to the next key matching some regex (and retry with the next key if it already existed), where your request would contain some parameters and the database key and/or filesystem path where results should be written.
Under-experience with databases easily results in rewriting a database using message queue/bus. Under-experience with message queues/busses easily results in rewriting a message queue/bus using a database.
Database are still fairly poorly documented when it comes to administrative work.
There is an incredible amount of tutorial, books and courses on how to write sql queries and stuff... But there is almost zero content on how to properly administer a database.
I mean, from novice admin to DBA-level capabilities.
I said all this before and i'm ready to write this again: i think there's a good market space for dba-style courses.
The most unpleasant codebases I've dealt with are ones that have suffered from a lack of strong leadership, and they are almost uniquely microservice setups that pull in everything but the kitchen sink, usually because it's just trendy to use it. Monoliths can get pretty damn ugly too but at least it's contained in one single codebase.
It went like:
App -> DynamoDB -> Kafka Connect Sink Process -> RDS -> Kafka
The reason for all the middle processes were because teams couldn't agree how to structure their data and the first app would dump literal nonsense sometimes so the Kafka connect process's job was to clean it and dump any of the nonsense they pumped into it. Pretty sure there was a gnarly log aggregation layer in the middle somewhere too IIRC.
"But we have no users at all right now"
"But we might have hundreds of millions of users in future"
I want to see more lines of code, not less.
If you do not improve next month we are going to have to let you go, we just cant carry a 0.3x employee such as yourself.
It got to the point where some dude built a system that completely floundered but got him a promotion. He then re-wrote it so it sucked slightly less and got another promotion. Last I heard he was working on v3.
Never mind that someone with actual domain knowledge would have either not built the system (since it didn't _really_ need to exist) or would have built a much simpler/more reliable system to get the job done.
His boss is happy with it because it looks good for him too
The company is happy because it allows them to sell milestones to themselves and or customers.
Customers are happy because the sales-people took the right person out to a fancy dinner.
I don't feel like I could fault anyone acting their part in the play.
I would find it boring/tedious to keep rebuilding the same thing since I'm more of a solve the problem, operationalize the solution, then move on person.
This is a good comment, however a giant issue is most companies will not say this out loud, and in fact say the opposite.
So while it's fine if you're savvy enough to see through this, it's not really fair to smirk at people who don't and who honestly take what leaders say at face value.
It's like a weird western version of Japanese office culture where you need to know the wordless rules, but it's not wordless and you need to ignore the BS said while feigning agreement.
Between the “perfect” company, and the shitshows described here, there is a lot A LOT of shades…
I used to work in a company which was far from perfect, but it was great to work there. Until the CEO changed…
I just search for a company where thing are just noy insane. I’ve been there, couple of times, and my current position is kind of ok…
But not all companies are sociopath boondoggles. Some are companies where people are trying to do the right thing (make money, grow business), and management would prefer to be successful. In those environments, we still get "promotion-driven-development" because, sadly, most managers are not software engineers and can be successfully bullshitted. And it's not even always malicious: you take someone who is a great people person but a mediocre engineer (huh, that correlates a lot), and they really think that this pile of steaming shit idea is a good idea, and their genuine enthusiasm convinces management who also don't know better. This mediocre engineer really wants to do good but has no idea why their idea is fucking herpes - because they are not competent enough to understand why. Dunning Kruger time. And of course the competent engineers are all Autistic and come across as rude, disparaging assholes in neurotypical management's eyes, and everyone gets herpes except mediocre engineer who gets promoted (possibly into management).
In that situation you have to have "... a handful of good engineers and going totally rogue, we outperformed the entire department pretty effortlessly."
All of my major career jumps have involved "going rogue", and having the outcome being recognized. And one time I was basically fired for it, and the work buried by a psychopath, and the remaining team did it anyway and delivered the solution, averting a product-ending scaling cliff. YMMV.
But not all companies are like that, and in some places you can point out to management that the idea is bad, and management agrees. In my current job we had someone pitch a fabulous promotion opportunity that was not merely a total waste of time but also fundamentally missed (didn't even attempt to identify) the root cause of the problem it was trying to solve, and fortunately management agreed when we pointed it out. Yay!
So I don't think it's as hopeless as you make out, at least not everywhere. I'm having fun right now, and getting paid enough (it's never enough).
or they go start their own business to get away from this nonsense.
when i deal with clients i actually have to deliver something because i am not getting any promotions.
I'm not saying that any one individual can't find a job that is the unique perfect blend of incentives for them, indeed, the entire point of this framework is that alignment between your goals and the incentives of your job is the most powerful lever an employee can pull. But that incentive structure will necessarily make others working there deeply unhappy as its unique choice of tradeoffs is just shitty in a way invisible to you.
that's fair, because i also have the power to fire them.
loosing one client when i have a dozen others is not a big deal.
with so many bosses i can focus on the good ones and reject working with the bad ones. i can also unilaterally raise my fees, especially for bad clients.
You're triggering my PTSD. Not to brag, but on paper, I was top performer in the dept at a previous employer. Was loyal, busted my ass, tried and many times did save the company money, but because I started posing a threat to my two sociopath managers (yes, two legitimate sociopaths) and spoke up against unethical (and likely illegal) activities of theirs, they made my life hell and eventually "laid" me off.
Getting fired was actually a blessing though, so fuck them. I'm not playing scummy games and larping as an asskisser for brownie points any more and have been much better off.
My career record (US Navy) for cost savings was something over $50MM. Every time I did something I had to do PowerPoints, present to Flag Officers, etc. Hell, once I almost got hugely punished because I didn't let my boss take the credit (he had no desire to because he had zero idea what it was I even did).
Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects (I hate every single bit of terminology in that entire godforsaken sentence), where it was literally my JOB to save the DOD as much money as possible with the least disruption possible. Those were the absolute worst years of my career. That period of my career was my reward for just going rogue and fixing things that saved millions.
I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.
Never stay at a job where the culture is toxic. Even if the money is better. Nothing eats at the soul like being held down by feckless, petty careerists and control freaks.
Ideally, you should be working for a company that pays well and that actually rewards you for doing things that are good for the business. But it does seem like virtually every company has some kind of dysfunction.
When they are, just move on. Life's too short. Whatever blows you up with $2 million in savings would blow you up at $500K
Paraphrasing, "The great thing about the US Government is it's so big, if the optimal solution would save 30%, but you find a suboptimal solution that only saves 29%, nobody will notice because you're still saving tens or hundreds of millions of dollars of savings."
One of his big projects was on energy efficiently in remote military camps. The cost of fuel per gallon was around $100 delivered to parts of Afghanistan. They had tons of portable generators running at ~20-40% of capacity. If memory serves, ~70% is most efficient. By some combination of building a small or connecting a few tents to a single generator, they were able to very significantly reduce the fuel required and improve the quality of service.
This was one of the most deeply insane parts of the whole twenty-year war. Of course, wars have unlimited budgets, so trucking gasoline thousands of miles through hostile territory with occasional losses and then using it to aircondition tents was going to happen. While back inside the US teachers were having to buy school supplies with their own money, because only wars have unlimited budgets.
I'm an engineer in a FAANG, who worked directly in money flow, and have had experience in diverse areas where it could come handy.
Then I start thinking finding the right person to work with & right area to start at is probably 95% of the job, then give up.
The direct employment option might get better with the recent executive order on AI. USG pays market wages for VA doctor/s; eventually they’ll do the same for tech roles.
I'm so triggered by this phrase. The worst (to work for and performing) company I was at hired SSBBs like crazy and none of them really did anything or made a damn bit of difference, but if you took the courses you were fast-tracked for success as far as promotions went.
That said, it did give me the opportunity to get the hell out of that community and back into IT/CyberSec.
In all honesty, there are absolutely good concepts and tools in that world. The issue is that the people who gravitate to that world have no ability to provide any value themselves so don't actually understand how to USE those tools properly.
Everything is a nail when all you have is an ENTIRE TOOLBOX.
A few jobs ago I was asked to list the names of projects I was actively maintaining for the company, that list stretched out to two entire screens in Excel at default font size! I was burnt out to the point of severe depression.
Even worse is I hate the bullshit ritual of submission that is job hunting.
Let’s rephrase this in a less nihilistic way: Understand your organization’s values and culture. And beware of doing good (or bad) things in a way that goes against your organization’s values and culture.
There isn't a way to make things better in the DOD that doesn't go against the culture. Period. My literal JOB was to make things better and it was the worst time of my career. That was with Flag Officer backing and independent authority. I got a chest full of shiny candy and a pathological distaste for it all after a few years of that.
The takeaway is don't be good at things you don't want to be asked to do.
You do have the option of becoming self employed. If you do, then you might discover a very different perspective. I've been doing it for 23 years now, with two partners and around 20 odd employees. IT company, you wont have heard of us but you will know some of our customers.
I'm still sort of working now (its ~0030 here) keeping an ear to the ground - HN, r/sysadmin (int al), The Register and a few others are some of my canaries. I listen to their songs. I also have a BirdNetPi doing the same for actual birds too.
If my company cocks up it will probably be fatal. I/we don't get the luxury of blaming someone else and I certainly don't get to mumble some sort of bollocks about "Your x is very important to us, soz we failed, lol and that - here have some free credit rating checker thingie".
Why not try running a small company and living on your own wits with no mother to land on - you are mother!
Even then, my kind of self work is just as unstable as my industry. I may in fact be good at it and still fail.
>(I hate every single bit of terminology in that entire godforsaken sentence.
Checks out. :).
A super convenient way to do cool stuff like that in the DoD is to do it for the most senior Flag you can get interested in it directly. Projects at SECDEF or SECNAV offices work...differently than outside, as there isn't really an 'up' left for most of the folks involved (in my experience - most are extremely focused on getting the job done and/or the geo-strategic problems).
Anything that was directly interfacing with that office was great; as soon as I was detailed to do something downline from that office it got painful despite reporting to that Flag.
I wasn't in a position to go higher without spending more time in the service, and that was a non-starter for me.
A part of me finds I might know you. Another part states: this story is way too common.
... I'm mentally tripping over that part. Is that normally expected? ... By recursion, it seems that POTUS should get all the kudos, all the time.
Some of that credit may or may not spill over to you, but the general thought process is that they are responsible for what happens in the command so they get the credit. Good ones avoid that and push the credit to where it belongs, but then they don’t promote. This leads to an incentive to take credit from those under you in order to get promoted.
Preach. 100%.
Likewise with chip software tooling; despite it being standard to outsource tooling to large EDA vendors, we got a lot of mileage out using our own custom tools, generally created or maintained by one person, e.g., while I was there, most simulator cycles were run on a custom simulator that was maintained by one person, which saved millions a year in simulator costs (standard pricing for a simulator at the time was a few thousand dollars per license per year and we had a farm of about a thousand simulation machines). You might think that, if a single person can create or maintain a tool that's worth millions of dollars a year to the company, our competitors would do the same thing, just like you might think that if you can ship faster and at a lower cost by hiring a person who knows how to crack a wafer open, our competitors would do that, but they mostly didn't.
He talks about a "cocktail party version of the EMH". The credentialed econ version of "nothing works" is https://en.wikipedia.org/wiki/The_Market_for_Lemons , which discusses the role of information and its asymmetry in markets. The EMH fails precisely because total knowledge is impossible and adverse selection is real.
There are some companies that represent the opposite of what OP describes. Once you work with one of them, I've found it impossible to settle for one of the bad places.
- managers asked how it was possible we saved that much without help from them
- asked to prepare slides
- asked many times on how it happened
- had to roll it out slowly to make it look like they did it over time incrementally vs one small toggle
- asked for a raise due to impact and did not happen.
Sir, for your sake, apply to a FAANG or something, you'll be at least taken care of better.
Also please implement Twitter card metadata in your blog so it looks better on twitter :)
Large organisations are so woefully inefficient that I'm surprised they're able to compete at all, but they have a ton of money and economy of scale and all that, and along the way there's more than enough money to waste millions on stupid nonsense and inefficiency and nobody really cares.
Over the next 15 minutes, I kept bumping the priority of the top job in the queue up, an hour later everyone had their work done, and went home. I had the room to myself. Rogue sysadmin for the win. ;-)
I poked around and realized that there was a system that we weren't using anymore that was copying files to the bucket I reached out to the stakeholders and they turned it off and we deleted the files.
The higher-ups didn't seem to really care, My boss's boss told me to reach out to another team that should've caught this and that was about it.
That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading.
I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.
The problem was to calculate bunch of stats from web servers logs (e.g. 10 most popular pages). The original solution was loading it all into Oracle database running on multiple servers since logs were huge. And then running bunch of SQL queries. Rinse and repeat daily.
What really scared me is that I couldn't identify any of the issues raised in my own organization, even though we often run into similar, smaller-magnitude problems caused by a blindness to obvious mistakes. It makes me fear I, too, am blind to massive bleeding wounds. Here's hoping they actually don't exist.
The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was.
The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.
Yep.
If you don't make them realize it was a Hard Problem™ that was only solved by their smart hiring, funding, and task-deciding, you might shatter their whole world view.
Yes, management would have killed me, but I would have been absolved at the Pearly Gates.
Sure you shouldn’t have to do that for every single change. But when something significant comes along it’s wise to document it and socialize it.
If OP had done this they may not have been asked to make PowerPoints or dread conversations or hide anything. They could just point at the public doc that explains it all in technical non political language that also educates other teams how they can improve too. This is leadership and mentorship opportunity.
* "Can I just remove or slowly deprecate this?" I.e this job generates data no one uses, or alerts people just bin-bucket or has been already replaced by faster/ better. * "Can I cache this?" * "Can I run this at a lower-priority/off-peak/less available?" * "Can I reduce the frequency or move processing to deltas?"
This is just the easy stuff you can usually do in a few lines of change without even getting into basic optimization rewrites. Most internal things at $large-co are built because someone thought they might be useful, they might get promoted and using that hypothesis and moved on, but few things are actually continuously validated as still generating value > their cost.
This morning on bluesky I saw a story of a very hasty onprem-to-cloud migration that was facilitated, and accelerated, by the rising floodwaters drowning the premises.
this machine kills imposter syndrome
It is incredibly reassuring that it is not simply my organization that is hamstrung by the pretense of complexity, when really someone just made it complicated to make it seem important.
Then job postings would have to be, "we're looking for competence and adequacy. We want to pay you a normal amount and get our money's worth."
I showed a demo how easy it is to read private ssh keys to the head of infrastructure, and after some months people could connect to network only using custom credentials (ldap) which was good, but also asked us to install "spyware" that among other things checked the firewall. I never installed the "spyware" but nobody pushed me. I didn't think I somehow prevented a disaster or did some heroic deed because everyone in the company was professional and nobody would exploit this. But of course I didn't tell about this to anyone except the infra because such information should not be disclosed until is fixed. And once is fixed why disclose it?
I really miss the Mac checkbox to enable the firewall. On linux I use nftables which is really powerful, but with so many possibilities it is easy to miss something during configuration.
I observed a lot of senior engineers don't have sufficient network knowledge. A lot of people on linux don't use the firewall which is really bad if you work on shared wifi.
Also when running docker images, if you map a port when using docker run (ex. docker run -p 80:80), docker will automatically add firewall rules and bypass the enabled firewall, exposing that port publicly.
I used to think startups would be one of the few places that actually gave a shit about being lean and efficient - but turns out that's only true if they're bootstrapped.
Success and super efficiency was rewarded with additional work, including hinting I should help other departments (apparently a joke).
I received no extra remuneration despite asking, yet my company continues to hire new staff weekly.
I've learnt my lesson, just like this author.
I love this so much
A colleague of mine and me got the opportunity a while back to basically work as wanted to in one side project. So we just worked truly agile, without any of the "Scrum" bs around it. Everyone involved was blown away how successful it was in terms of implementation. Two companies tried it before in the span of a year, and they didn't get it working at all. We worked just some evenings and weekends outside of our full-time job over the span of six weeks, and we got 90% done, with everything essential included to be able to start working with it. We then implemented everything else the customer wanted within the next weeks by following the same principles. There were no problems at all (apart from the customer basically almost running out of money because they spent most of on the previous companies which couldn't deliver).
At my day job, the managers usually argue against anything we want to do, question every decision from our side, want to have regular big meetings with 16+ people to talk about everything at length until no one wants to do it anymore, and then micro-manage everyone to death. We tried to at least keep these guys out of the daily by pointing to the Scrum guide where it says that only people working on tickets should participate, so one of these guys just created a bs task for himself, then talked every day about it for a lengthy amount of time without providing any real value at all.
I never got those silly shows before, which play in offices where everyone just acts silly and they never really work. But now I know why they are so popular. They just provide comic relief for people who really have to work in those environments and it is really as bad as in the television shows, if not worse.
And you can cut this in half by changing some defaults in instance lifetime?
I’m starting to understand why Snowflake’s market cap is something like $50B. This sounds like a nice money-printing business if you can convince enterprises to use it.
The immense cost was coming from someone writing a query that translates to "I need one row of data" and then we get billed like $10-20 in idling compute. With multiple computers and several full-time SQL modellers, it adds up very, very quickly.
As the guy that just has to keep Snowflake running smoothly and isn't paying for it, it's a really nice product. I would still prefer something else on principle because it isn't open source, but eh, I guess it reduces my stress at work.
"hi everyone, I wanted to walk you how we got to a half-a-million dollar savings, basically I spent a day looking at how terrible the original infrastructure was deployed and removed a code test feature that was causing the problems. This was just complete oversight from every aspect of the development, management, testing, and everything. Overall this code is as bad as it can possibly get, and we just launched it. And basically I was told not to say any of this because it makes everyone look bad, so I was to roll this out gradually to make it seem like managers were doing some sort of work."
Then drop the mic and walk off stage.
Honestly the amount of give-a-fucks I would have lost would have been a lot. And this is coming from someone who's done this for almost 2 decades and cares about his job because bills to pay, kids to feed.
About once a year or so one of the stand-out engineers that had the weight of the world on their shoulders would get burnt out and frustrated enough to do exactly what you suggest above.
Literally everyone would just look around awkwardly, leave the meeting and never talk about it again. All of middle management already know all of this, the only way they keep their jobs is by never talking about it, and just ignoring anyone that does. The VPs and President only know what those below them feed them.
But you would get handsomely paid.
Didn’t Xitter recently remove the display of all external site metadata except the image?
See, for example: https://twitter.com/simonw/status/1717768637799706922
“We’ll need the percentage complete as well and a summary of the savings expected. It needs to be sent out every Friday at noon.”
Noooooooooo
Wait, it wasnt the managers that designed the overly complex solution! this is all on the engineers
It's easy to understand: large orgs are competing against other large orgs that are similarly inefficient, because efficiency in a large org is a really, really, really hard human problem that we simply haven't solved.
And remember they're still providing a valuable service/product. For all their inefficiency, they're still more efficient than not existing at all. This is why they would exist if there weren't competition.
And you might ask, what about competition from small orgs? Well, a small org has less efficiencies of scale, but if they are efficient otherwise, they can sometimes compete effectively with the large org. But then they grow into a large org and will wind up with all of the large org inefficiencies.
It's just how things are. It's not that nobody cares -- to the contrary, company owners care hugely. It's that literally nobody has any idea how to solve it.
We used to break up companies that became Too Large. Either they became so large that it was impossible to compete against them (as Large Company can demand way better pricing on volume than Small Startup can or has way better access to financial instruments due to better ratings, or like many automotive companies even run their own goddamn bank), or because they could use a wildly profitable business to price-dump competitors out of existence or because they became so large that they felt free to extort their customers to the tune the people would complain about it in the media/their congresspeople too much, or because they became so large their sheer size represented too much economic risk ("too big to fail").
We should begin doing that again. Something as big as Google/Alphabet, Apple, Amazon, Meta/Facebook, Walmart, the Big Four consultancy shops, virtually all major banks - there's absolutely no reason these should be allowed to exist at their current size. Or, if these companies still wish to exist at their size / their existence as one platform, they at least have to be regulated to mitigate the threat originating from that scale.
Competition is usually eliminated by one or more "moats": huge capital upfront investment, patents, market lockin agreements (e.g. the way Windows is sold to OEMs), vertical integration (e.g. Apple), compliance (it's genuinely hard to start a bank or a hospital), buying out competition, buying out the staff of the competition (why FAANG have so many highly paid staff doing work that gets cancelled) or even less legitimate means.
Large companies tend to end up with similar failure modes to authoritarian state-controlled companies. It's interesting how well China has been able to ride the line between control and growth on this one.
They typically don't. When the government doesn't enforce it's own rules, these companies just buy up competition. They strangle the market and they get more inefficient all at the same time.
Competition is a fairy tale we tell MBAs so they'll start new businesses so it looks like there's competition. They never had a chance.
Inflation is actually very simple: More money is added to the system than value produced. That is, money is printed faster than value is created through labor. Stop and think for a second. How do we have trillions of USD? Banks create money. And they're doing it faster than ever before.
Now you can absolutely have price gouging at the same time, but the two are independent of each other, even though combined the affects are worse for the price gouged.
It's the reason why most central banks really want some minimal level of inflation going on at all times.
The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG.
I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly.
EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.
Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
If you had just let the crisis occur, everyone would get a chance to spring into action like heroes, handshakes and champagne all around on a job well done.
Or find and company that values your talent and knowledge.
There is a high likelihood that many other people in the org were lightyears ahead of the author. It's even more likely that an engineer or manager in their group was "Banking" the inefficiencies to use during a cost-cutting period - and the author ruined that chance... which will inevitably cause massive suffering and possibly poor performance reviews when there is nothing to trim.
It really sounds like they stayed in their lane and did their assigned job, just like every other person on their team.
I have worked with someone who was like this. Perhaps unlike the author, they were not in fact the savior in a room full of idiots.
This person was no doubt clever and had interesting experiences to learn from. But they were also dysfunctionally arrogant and utterly resistant to ideas that were not their own, to the point where it repeatedly and painfully got in the way of their own progress and other people's progress around them. At the end of the day they got done what they needed to get done, but they left a big mess behind that took weeks to even stabilize (as in, "prod is broken, prioritize above all other work") before the rest of the team could get back to their regular work.
May such people forever find themselves in solo contributor roles, so as to spare others the pain of being their close collaborators.
In our collaborative environment, it's valuable to maintain a level of respect for diverse perspectives, even if they initially appear confrontational. Instead of dismissing the author's stance outright, we might benefit from conducting a thorough assessment of our current procedures and identifying areas for potential improvement.
Engaging in constructive dialogue allows us to pinpoint inefficiencies and work collectively towards solutions that benefit our organisation. Ultimately, it's our ability to learn from each other and adapt that will contribute to our long-term success.
I'm guessing it's dormant because the level of discourse on HN has devolved to a point where nearly every comment on HN would now qualify.
I wrote a tool that saved about two hours a case, in total this saved about £500k. I got a free case off beer.
The only "personal" reward I get from that is: whenever I feel guilty for not having done much in a given day, I remind myself that by this action alone, I've saved my company several times what I would ever cost them.
Helps with self-esteem, but I don't think my company see it that way.
Perm is more "we pay you so fix this", consultant is more the reverse "we need this fixed so we'll pay you".
I always find that dynamic hilarious because in a general sense permanent employees have more value than contractors/consultants as perms usually have a much longer tenure at a company (years and years, vs 6 months to 1 year).
However one thing I noticed after moving to the UK is that the culture is completely different here - everywhere I've worked here there are contractors who stay for years and years like a perm would. It's not necessarily a bad thing, but it does make me reconsider why anybody would ever be a permanent employee beyond a bit more job security (ie long term contractors definitely have to trust that they'll be renewed, even if they usually are it's always possible for the business to decide otherwise).
In fact the team was pretty upset that they'd budgeted that money for infra already and it'd have been better spent instead of waiting till next year to re-budget it.
It wasn't necessary though because we had a committed spend target to reach and we just had to figure out how to legitimately spend the money somewhere else. :(
I've been on the receiving end of middle management because I've been able to fix things in the business that have always been broken but never got fixed until I worked on them.
Management will claim it's their work, will give you just lip service, will not use your name higher up the hierarchy and will actively down play their own mistakes whilst blaming the rest of the department or developers (e.g. like building a project with wrong requirements they actually provided, then missing the deadlines because stakeholders demand changes).
Where that has happened I quit and then they just go back to the status quo.
A company is like a chain, as strong as its weakest link. There is not much gain to have some links much stronger than the others.
The famous "I've learnt my lesson" which means you really reached the same low level of incompetence as everyone but secretly thinking you are more competent than all the others.
The funny thing is that among all those incompetent peoples / idiots mentionned there are probably smart people who just learnt their lesson years/months and adjusted to the weakest links in the chain.
I'm here because there's some path-dependence in careers. I started at a mediocre company due to not having a permanent work visa, and have been clawing my way up. I should write something else on this, but I've also realized clawing my way up was unnecessary - it turns out that while I've had some skill atrophy from working at these places, good engineers recognize someone that isn't going to cause a spreadsheet dumpster fire, so I should have just jumped to one of the top rungs on the ladder years ago.
I agree though, there is no perfect place; wherever you go, there you are. Still, if you're unhappy, you're not going to be less happy by changing to a better job.
It comes from RTFM, but instead of the Manual it's the Article.
I have moved from admob (google ads) to robotics to facebook, and worked with different orgs, and have seen differences.
90% of what I do is ask dumb questions as if I'm completely clueless.
Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves.
One of the things I'm hoping I could keep alive in my child
When I'm talking about how large companies have inefficient processes, it doesn't really matter if the company has 10,000 employees or 100,000.
I'm constrasting with companies that have 50 or 500 employees.
My argument is that there should not be companies this large at all. Everything above 100-500 people tends to breed a layer of pointless middle management layers and a host of supporting bullshit jobs, and in many cases the lack of internal competition leads to ossification.
So, yes, of course one may argue that without this one couldn't build something as complex as an airplane, a GPU or whatever... I'd say, keeping a cluster of teams as distinct companies, maybe with some sort of "service company" that deals with shit like payroll, travel expenses and whatnot, is actually more efficient as it offers a chance for people to get (literally) invested into their work and have the profits end up in their wages as well instead of everything being siphoned off in increasingly opaque ways. And it enforces proper working processes (e.g. documentation about interfaces), and so actually reduces the chance of stuff going wrong.
Much like people who underperform in technical fields often pivot to Management, you see the same in the pivot to Process Improvement. That creates the issue of the lowest performers at the actual job being the ones most inclined to pivot into that position.
There are always good ones (I'm biased, but think I was). The issue is that they are a relative rarity due to those (and I'm sure other) factors.
This can't be serious...Not sure if jealousy or just regular hubris.
That said, it’s anecdotal at best; I don’t pretend to have a statistical sample. I also noted that there are exceptions.
I spent a significant amount of time in management and Process Improvement, so I’m speaking from experience of the inside of those within my section of the DOD.
Of course this kind of inflation should be fought by competition, but you have to have companies actually willing to compete on price.
So the government is pumping money, but is not getting the desired signal, so it pumps more money.
So there's another reason to replace welfare with UBI. Seems every discussion leads to that these days.
Figuring out what said own thing is is proving a bigger challenge than anticipated, but I'm enjoying it.
* To add to this, the crazy people were slowly pushed out after we were acquired by a slightly larger company.
MAAA! THEY'RE DOIN STUPID STUFF AGAIN AND THEY WON'T LET ME USE MY COMPUTER!
In reality, this is certainly possible. It just would have been a career killer. My CO was the one who sent that presentation up with my name front and center. He had to defuse the situation.
Also, and this is absolutely relevant, I was not a Commissioned Officer. I retired as a Chief Petty Officer, though I had a degree (Nuclear Engineering) and multiple "graduate level" certifications (PMP, LSSBB, CISSP, etc., etc.). There is only so far competence and capability can take you without rank in the US military. I chose to exit rather than move over and promote, which was the right decision for me.
I apologize for lacking the context in my previous comments; I often forget people don't understand the intricacies of the arcane ways of the military unless they subjected themselves to it.
I'll drop you a line via your profile contact info and maybe we can have a chat!
E.g. why didn't you turn off that temp m3.xlarge instance. $X gets docked from your pay.
And sadly, I'm far from overpaid (you can probably guess my industry in 1-2 guesses now. Not like I keep it secret on this profile to begin with). I really do want to do my own work without being binded to corporate whiplash, but I still have a mortgage and car to pay off. And I need to rebuild my 6 months savings after the current year of layoff.
I'm going to start laying down the foundations on the side next year, but I see at least 5 more years before I can even consider jumping off.
There's always career direction change. Lots of folks are eager to hire gamedevs, because they work hard and think different. Not sure if that's something you want, but perpetual burnout is never worth it.
Searching "USDS or 18F" helped google understand what I was looking for
It happens. From DoD civilian world, 6 months in with a new org (I was the fng)…
tl;dr — Made some software that moved a project from imminent mission failure to mission success, and got all of my command chain in trouble in the process.
1. There was an 8-figure, 2-year R&D contract for some mission critical software that had functions that my org needed. We were updated regularly that everything was moving along swimmingly. At the end of the contract, only about 10% of the functions actually worked. This is totally ok from their side since R&D contracts are “best effort”, so no functioning deliverables are actually required. The part of the software that was mission critical for my org had 0% working. My org was fucked.
2. We need this software in 3 months or less or we can’t fulfill obligations that we had made years prior.
3. Manager gives me free rein to make something that can solve the problem.
4. I have a prototype in one week. It’s not as automated as the 8-figure solution was supposed to be, but we had bodies to help with that (and also do sanity checks at various stages).
5. Weeks two and three were refining based on feedback from stakeholders.
6. Week 4 was training about 60 people to use the software.
7. Software was iteratively refined over 6 months or so based on user feedback during active use, but no major changes were needed — just minor QoL stuff.
8. Software had the bonus side effect of getting two embattled groups to start talking to each other and resolving process issues, which saved unbelievable amounts of time and lowered stress levels immensely.
9. End result was that we were able to meet our obligations. Mission success! Software worked with amazing consistency (very little down time, almost all of it out of our control), and resiliency (changes were easy and fast to make without breaking the system).
So… what did I get for this? A spot bonus? A coin? A handshake from the commandant?
Nope… I got a reserved “thank you” from my supervisor.
The aftermath tells why.
When my supervisor changed the status update up her chain of command that “mission critical software missing, mission failure imminent”, to “mission software exists, outlook favorable” in the course of one month, people started asking questions?
“You had someone in your department who could do this already?” (No, I had just started, and not for a programmer job.)
“You asked to spend a big chunk of the 8-figure R&D contract on your mission critical needs, and that work could be done in a month!?! And you didn’t know this?!?!?” (Supervisor was not a programmer, so it was all basically magic.)
So supervisor was sort of criticized for “wasting time and resources” and “not knowing her people”, even though she could not have reasonably known that I existed (the R&D contract started two years before I worked at the org) and could solve the problem.
That’s bad, right?
Well, it got worse.
The head civilian of our org had been making updates to DC about a potential mission failure due to the 8-figure contract software not working. When that got updated to “not a problem” after it was evident that my system worked, he got called to task in Congress, with congressmen asking him the same sorts of questions that he asked my supervisor (“You had in house talent that could do this? In a month? And we spent how much time and money on a contract to get the same functionality?”). There were no good answers.
Needless to say, once I heard about the fallout, I was looking for the next thing. I felt like I had done my civic duty as a public servant. It was clear to me that further involvement would probably have similar outcomes (efficiencies made at the cost of negative career risk for my command chain).
Also there was a point when Blackwater boasted that they could deploy a light brigade (~3k troops) anywhere in the world, people in the Pentagon figured it out and very quickly killed that org. Compare with Prigozhin and his adventures, just for reference he had more than 10k soldiers under his command, on heavy equipment (arty, tanks, ifvs), permission to illegally recruit from prisons, he even had air defence like Pantsirs, the motherfuckers even tried to get nuclear backpacks (man-portable nuclear bombs) from a storage site near Voronezh.
I think those entities are being tolerated and even welcome, as long as they don't exceed some potentially threatening size.
I can't remove it either.
I'm open for more questions.
> I'm open for more questions
I laughed much harder than I should of at this.
And who's going to fire you for this?
The people telling you to roll this out slowly are doing so mostly to protect themselves from having their incompetence exposed and to appear useful. Protecting them will help them steal your credit and will get them promoted.
> the top of the company should love because a) you saved them a ton of money
The top of the company is probably already rich. Being richer is great, but one of the few things rich people generally won't burn to run the money-making engine hotter is their own sense of prestige and entitlement.
> b) you identified a pile of incompetence in their company.
Yes, and you told that fact to people who were already responsible for identifying that, which means you just told them that they are incompetent too. And that transitively works its way all the way up the org chart.
You would make the whole chain of command lose face and do so in front of the rest of the chain of command. It would be career suicide.
Every manager would rather silently waste money than be made to look like a fool. Because the money comes out of the business's bank account, not theirs, but looking stupid affects their personal reputation.
more likely, the top of the company is just as incompetent but has lucked / gamed their way into promotions anyways.
So this engineer throwing his managers under the bus may be good for the company in theory, but to the senior management this is a red flag. they don't want engineers who do this - they want engineers who give credit to their managers. if they promoted this engineer and fired everyone else, he'd come for them next.
Of course if everybody quietly agrees that this company is a suitable vehicle for their incompetence, then it's a bad idea to address this. And some companies definitely are.
Let's say 5 or 10 years later you're applying to a job where one of these upper-level people now work. How do you want them to remember you? The know-it-all who wasn't a team player and kind of an asshole? Or the engineer who gets things done and has demonstrably shown to land impact and value, an engineer the exec would consider lucky to have?
Some of you will say you wouldn't want to work for one of these executives again. But people change, incentives change, the environment changes. Have you ever made a technical decision you later regretted?
And maybe you don't work for them. Maybe you're applying to a different company where someone knows these previous upper-level management folks and they ask about you. How do you want that recommendation to come across? "That engineer was an asshole.", "That engineer was amazing, I wish we could have kept them. We made a big mistake by not trying to keep them.".
I've definitely missed out on work sometimes due to having a reputation for being about as subtle as a brick to the face with no lemon, but I've also -got- certain pieces of work as a direct result of being criticised and somebody who heard the criticism thinking "if he annoyed that person that way, he's probably serious about doing the right thing."
I would, however, suggest that probably I would've done better overall if I'd toned it down a bit.
The salaries adjust by location, pick your location from https://www.opm.gov/policy-data-oversight/pay-leave/salaries...
This is the SF area table: https://www.opm.gov/policy-data-oversight/pay-leave/salaries...
(no.)
edit: incidentally, this other thread [2] on the front page today has lots of comments confirming. ctrl-f for USDS.
[1] https://www.opm.gov/policy-data-oversight/pay-leave/salaries... (PDF)
I mean, if you're upset about government spending on tech, you better sit down before someone tells you what goes on in the large corporate sector...
There was an old thedailywtf post about how a company thought they'd incentivize fidning and fixing bugs. Suddenly every engineer had a QA buddy, and they'd make like 50 spelling errors, which QA will find, and enigneering will quickly resolve. They took down the bounty within a week.
The term cobra effect was coined by economist Horst Siebert based on an anecdotal occurrence in India during British rule. The British government, concerned about the number of venomous cobras in Delhi, offered a bounty for every dead cobra. Initially, this was a successful strategy; large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped. When cobra breeders set their now-worthless snakes free, the wild cobra population further increased.
(There may be some question as to whether these events actually occurred or not, but there are similar examples of documented pest-control campaigns (and others) on the Wikipedia page[0] where similar things happened).
[0] https://en.wikipedia.org/wiki/Perverse_incentive#The_origina...
Yeah I've been looking into the cobra effect in the current youtube adblocking thingy... It certainly got me to use primarily adnausium which fixes the problem since ads are served, just... you know... maliciously clicked on.
That might have been this Dilbert comic:
I felt bad for leaving so soon, but good for not having cost them a dime.
when I have made large improvements like the article/blog describes, I am pulled off of that and put on something far worse, immediately, except without the autonomy. "why can't you succeed here?"
Also pitched a plan to the pentagon about privatizing their presence in Afghanistan and extracting their natural resources to pay for it. https://www.militarytimes.com/news/your-military/2018/09/05/... (didn't vet this source)
That... didn't happen.
Blackwater went through several reorgs, rebranding, and acquisitions, initially starting with at least superficially deemphasizing security in favor of other functions because of “business risk” just after the Nisour Massacre, but (as Constellis Holdings) is still doing the things it did before, notably as part of the Saudi intervention in Yemen.
But largely you're right, the story is more complicated. I originally heard it in a non-English interview with some Colonel (OF-5, the rank just below General of brigade).
Personally I am very much against the idea of a democratic nation allowing a private company have a combat strength of an infantry brigade. Definitely too much. (And of course such structures, if they exist, must be penetrated by intelligence, though this is not something you can legislate or even admit).
Never been laid off during a recession or had your pay frozen and bonuses cancelled during a hard time? Employees risk a lot more than most stock holders by working for a company. On average, stock holders are way more diversified.
Uhm no?
You apparently never been a business owner. Employees get their wages, and even can legally enforce them. If a business go down, owners eat the losses and envy their employees.
I've been on both sides. Being a business owner is much riskier.
Is the risk high enough to justify the ever-growing disparity between owners/C levels/investors/etc and the employees that get the work done?
What with all of the bail-outs through history, running an especially large company seems pretty much riskless. And hell, if you look at the history of technology, say games and game consoles (because I like retro games) the number of times a hugely successful product/project that netted 100s of millions of dollars was "not allowed" by the CEO etc but was hidden until it was too late (see Xbox etc) is super high. In addition to the number of decisions made by higher ups where the business swallowed a loss (particularly easy in larger businesses) is also high.
Imagine if Bill Gates and Steve Balmer hadn't been convinced/swayed to make the Xbox. How much profit has MS made from that? A fucking shitload, and have the guys that pushed it, or for that matter anybody in a similar situation (of which there are many) ever seen any of that success? No.
And we can't say "well the CEXs have the final say because they take on all the risk" they do, technically, but in reality when C levels screw up oftentimes it's just taken as a loss and things move on.
Pay frozen == no pay raises
hmm, they can perhaps reduce the risk by not working for a company. They can just be stock holders or launch their own company, that way whatever may happen they will never get fired.
How can anyone seriously type this? If you fuck up bigly enough, you will 100%—without fault—get sacked. Again, not even talking about long tails (bad economic conditions, layoffs, etc.).
This is under totally normal situations: if you lose the company money, you will be fired. As a bonus, you also lose unvested options or equity. These kinds of posts are exactly why engineers have garbage bonuses compared to finance even though they probably generate an order of magnitude more value.
which is to be expected - making a big mistake might not be something that can be forgiven and overlooked (depending on the magnitude of the mistake).
But you will not lose capital as an employee, since you did not put in capital to lose. Your time would still have been paid, up to the day you are fired.
Therefore, you obviously have no incentive to take on a risk that can result in a mistake (but which the reward you take no part in). You just do your assigned job, and whether it saves the company money or not, as long as you can cover your ass, you're golden.
Unless the company incentivize you to save money - for example, via a bonus through hitting a target or achieving some goal that was set.
The conversation is a lot more complicated because there's an opportunity cost, you lose time (your time is finite, company time is infinite), you lose reputation, and so on. Besides, your argument is a bit weak as it's not like hedge fund managers put up the cash themselves, either.
My point is only that value-generators should be rewarded as such, and it's a bit weird that engineers are totally cool with not getting a piece of the pie.
It depends on how much you mess up. Mess up large enough as an employee and you can end up sued by your former employer. Losing a lawsuit is losing capital. A probably not comprehensive list of reasons an employer can sue an employee, not all of which are because of negligence or malfeasance: https://www.mylawteam.com/employment/can-an-employer-sue-an-...
Depending on the state you can also have your pay docked (if that's not a capital loss, at least for transportation costs, then I don't know what is): https://www.avvo.com/legal-library/employment-law/paycheck-d...
Nah, if they had enough capital to live off of before then, they are far more likely than us to fail upward into a management job.
No there isn't. You haven't put up any capital when you join a job, and you aren't (generally) required to invest any as you go along. Your time is not capital.
Software engineers are some of the best-paid labor in the world with great benefits and workplace conditions. They often receive equity as a compensation, even when the salary is still vastly above many other lines of work. They are absolutely getting a piece of the pie, and in much greater proportions than almost any other economic activity.
You may be discounting the value of capital, management, sales, and other roles in a successful software-related business.
The remuneration that labor and employees receive is never going to be in line with the value that they generate, precisely because the former group doesn't take any risk. They don't invest any personal capital and they aren't liable for anything. They can walk away any time, sometimes voluntarily, sometimes not. In return, they work fixed hours and get paid on a routine basis. The owners receive only what remains above and beyond all that, which could be great profits, just breaking even, or even losses.
> there's an opportunity cost, you lose time (your time is finite, company time is infinite)
Everyone everywhere loses time, because time passes whether or not you choose to do anything with it. Employees aren't unique among economic entities that they face opportunity costs.
> it's not like hedge fund managers put up the cash themselves, either
This is actually a good example to dive into. Hedge funds are typically paid "2/20", meaning 2% of assets under management every year whether or not there are any gains, and 20% of any gains above some benchmark. It's similar to, say, a commission-based sales role that gets paid a certain fixed salary and a percentage of sales they make. Whether or not 2/20 is "fair" is solely up to those who buy their services, since there is a competitive market of providers of fund management (the "employee") and providers of capital (the "employer").
And in some situations, the "employers" do in fact lose a lot of money, while the "employees" walk away; the limited partners of Melvin Capital, for example, lost many billions of dollars, all while Melvin Capital itself continued to charge the 2% management fee.
And within hedge funds itself, there are again employees who receive a stable salary and maybe some performance-related bonuses on top of that, versus the principals and owners who have personal capital invested. When LTCM blew up, for example, it's estimated that its owners lost $1.9B[0].
[0]: https://en.wikipedia.org/wiki/Long-Term_Capital_Management
The people at the top get an even better deal. They get given stock options, so they get the upside but not the downside. They can also walk away, but they'll get a big payout if they walk away involuntarily. They work fewer hours whether you're counting butt-in-seat time or making-efforts-about-work time (some people, bizarrely, compare the CEO's making-efforts-about-work time to the employees' butt-in-seat time and conclude that the CEO "works more").
> And in some situations, the "employers" do in fact lose a lot of money, while the "employees" walk away; the limited partners of Melvin Capital, for example, lost many billions of dollars, all while Melvin Capital itself continued to charge the 2% management fee.
You're flipping the categories. Being the "investor" can be a bad position, sure. Being the manager, the decision-maker, is where you can't lose. Concluding that that somehow makes employees better off than owners is ass-backwards.
https://en.wikipedia.org/wiki/Texas_two-step_bankruptcy
https://www.businessinsider.com/corizon-health-bankruptcy-ye...
> If successful, Corizon's Two-Step would avoid a much wider range of liabilities than previous companies who've used it — not just injury lawsuits, like J&J, but the routine debts to vendors that companies rack up every day. If the company succeeds, it provides a "roadmap for eliminating virtually any unsecured liability owed by any corporate entity, regardless of whether that entity is solvent," Ian Cross, a Michigan civil-rights attorney who represents multiple prisoners who have sued Corizon, wrote in a procedural objection in April.
or do a leveraged buyout in which: https://www.investopedia.com/articles/markets/111015/10-most...
> The goal of leveraged buyouts is to make a large acquisition without committing much capital investment.
Or they'd sell to other investors, who are using a leveraged buyout, in order to get a better selling price than they would have otherwise. Meanwhile the new investors would take enough in income to cover the amount they put down, plus some, and then the company would eventually fail because it had too much debt.
