As a manager you have to have really good EQ to be able to read this sort of stuff implicitly from your employees because it's usually not coming out explicitly.
I'm not (that) conflict averse - I'm perfectly happy to suggest, when asked, that we could deliver better results faster if we could spend some time fixing some non-customer facing problems, had better tools, and had a testing environment that was closer to the production environment. I have no trouble saying those things, but I've never seen any action taken on any of them.
Both "This is just how the company is, it won't change" and "We survived thus far, we'll get through" are the unofficial mantras of the company I work for. Even though many in the middle management see the problems, these mantras are repeated whenever one speaks up against silly processes. And boy, do we have some silly processes.
It's not helped by the fact that the second is actually a mantra often quoted as the official motto of the neighboring metropolis, as if resistance to betterment is something to be proud of ("et hät noch immer jot jejange", for those in the know of both German and the dialect).
Though I suppose it might have also been the motto of the dinosaurs before the big impact.
There is "learned helplessness" in a way also in management, because they in turn seem to have learned that raising any issues with the higher management only leads to being branded as, if not a source of discord or trouble, then at the least as someone who requires a higher manager to deal with something, as opposed to those not requiring a higher manager to deal with something. The later clearly being preferred by the upper management.
I (not that long ago) worked at a company that kept all their data in MySQL. Sounds fine. The bad part was all their user-facing stuff was written in Microsoft Access.
This didn't make any sense and was the main thing that prevented them from moving to a web-based or API-based interface. They had written some code here and there that would cobble a few things together but it wasn't the right way to do it.
During a discussion, I asked why there were 2 MySQL servers. They held different databases, but it would have made more sense to have them both on one server. I assumed that at some point in the company history, the data got too big to fit on one server, so they split it up, and that ended up being true. And ever since they had some misguided sense of safety having two different databases, even though if you were missing the other, all functionality would break.
At the current time, the databases weren't large enough or accessed enough to require being on separate hardware. It seemed simple enough to just import one into the other and then get on with modernizing the rest of the code.
"We would have to re-write the entire codebase since it's based on having two servers"
Access lets you basically do joins etc on two separate databases. So this was used heavily. Whenever they tried to duplicate this functionality in PHP or whatever they would try, it turned into a nightmare.
I would imagine if they had spent another $10k on hardware back when they first had this problem they would have saved a lot of future troubles. Imagine running legacy versions of Visual Basic, developers needing Windows XP VM's, end users needing two copies of Microsoft Access installed...
At the end of the day, if you're a developer, you don't get to call the shots in terms of what issues get fixed when.
That's up to the lead dev/product owner/BA.
You can flag issues and propose solutions, sure.
If you think the company isn't worth the trouble then quit and get another job, there are plenty of tech opportunities out there, one's that are no doubt more dev focused.
This time around, an engineer produced a fantastic design. But this gatekeeper engineer had something else on his mind. The gatekeeper did not even communicate his thought process but continued asking ridiculous questions.
Later we found that the gatekeeper engineer wanted to change the entire design to his own, even if his own design had major flaws. Wasted a whole quarter just because the gatekeeper didn't seem to want to "concede". He would much rather have engineers make bad designs than accepting his own missteps. The author of the design fought vehemently at the expense of their own time and energy. To what end?
Other engineers on the team have forever stopped producing design docs. Learned helplessness.
I have nothing to gain by getting caught in the crossfire.
I would just quit after biding my time.
Adding process is a delicate balance and most large companies err on too much of it.
Every checklist you add, every new approval, every sheet that needs some rows filled out, every brief - all of them add friction.
New projects are inherently speculative, and most things don't move the company's bottom line. The power law that says 10% of work get 90% of results is true. Friction means whoever is doing the work is just a little less likely to propose a bold test, to push on a tweak they notice, because they know it will become an ordeal.
Obviously you can't be 100% wild west and some process is needed. But you have to be very very careful when adding it.
Even with a ton of nice safe ossified process to make the owners feel safe, if you just make it a policy to give them a win once in a while, they'll live off of that and not even be all that unhappy.
The 99% friction is felt less like a waste and more like a high bar.
I think in a lot of places, they just don't allow even that calculated IV drip of dignity and purpose.
Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.
I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.
The main driver of learned helplessness is those who refuse to listen to feedback or act on feedback.
The tech lead started out by complaining about how he has to do an untested unplanned release today because another team made some urgent changes. He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
Both of us have been working at this company for about 3 years and we both have over a decade of experience in software development.
When he finished complaining, I started asking questions and making suggestions about how we can improve things. - Push back on the team that needs these urgent changes. Let them learn to do the release. - Deny the release since they didn't communicate earlier. - Improve the release process.
Everything I suggested was just flatly denied as impossible. - The other team doesn't know how to do the release. - He wants to be a "team player" so he can't deny the release. - Project managers will never allocate time to improve the release process.
I feel strange because I've seen this same thing for my whole career and I still try fight for what's right when others appear to moan and carry on.
However, my experience tells me that bringing this stuff to my manager is even worse. My manager doesn't know anything about the code, my project, or the release project. He may assume it's complaining for the sake of complaining. It has been used as ammunition in reviews against me.
Learned helplessness sucks and I wish I could do more. I don't think either of the suggestions in the article are feasible for many ICs. Teams are ambivalent to making improvements, and retrospectives carry very little weight. Managers are above the fray and won't be held responsible for by people below them.
My manager doesn't know anything about the code, my project, or the release project.
IOW, your team has no leadership and no management. Your team lead may be called that, but sounds like he doesn't lead at all. Your manager may be called that, but he doesn't manage either. This happens a lot when people are given responsibility without any authority (probably the case for you "team lead"), or authority over things they don't understand (probably the case for your "manager").
The bad news is that you're part of a disorganized mob. The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want. Of course, even if you succeed, it's possible no one will care about your accomplishments. And you will probably make enemies. It's possible you'll be rewarded but given your description, I view that as unlikely.
As a side note, suggesting to the team lead that he lead and do X, Y or Z may appear to him as if you want him to take on risks with little upside. He's probably thought about doing those things anyway but just decided it's not worth the trouble. It's not all that surprising that he doesn't want to rock the boat.
Perhaps. I thought so once in exactly this situation, and it led to an incredible amount of frustration, and burnout. And others were hurt the same way. That leadership vacuum was very real, but nobody wanted it to be filled by anyone already in the company, it turned out.
Eventually we got a very competent new team lead from outside and he was accepted by everyone immediately. For one of the teams anyway; the other one is still a complete mess. I just make sure I'm on the right one.
If you don't want to become a leader, what do you do in a disorganized mob situation? Leave?
> The other team doesn't know how to do the release.
> He wants to be a "team player" so he can't deny the release.
> Project managers will never allocate time to improve the release process.
and your good reasons not to push internally for a better system?
> My manager doesn't know anything about the code, my project, or the release project.
> He may assume it's complaining for the sake of complaining.
> It has been used as ammunition in reviews against me.
In fact the team leader is a "hero" who extinguished a fire caused by other department.
There is a self-reinforcing aspect to learned helplessness in teams, as you've pointed out.
I learned pretty early on that I have political capital, and very little of it. Instead, I have to have a manager that's pretty buck wild when I tell them I need them to be, and I have to deliver them some wins that will grease their wheels too. Good managers are hard to come by though, and if they get noticed they're promoted and gone in a couple of years.
Personally, I think the transactional nature of software positions is what largely holds us back. If I get promoted, then there'll be a new lead, and this cycle will start all over again.
Everything I suggested was just flatly
denied as impossible
[...]
I feel strange because I've seen this same thing
for my whole career and I still try fight for
what's right when others appear to moan and carry on.
I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole. Management will never let individuals waver from their short term goals.
The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
(Although, the issues you ran into are kind of an org issue in addition to a developer experience issue)
I came to this conclusion after watching the place I work at for 2 and a half years fail to implement any of the grand ambitions management had in their heads. We wanted code review, pull requests, build pipelines, automated releases, updated OS's, logging, disaster recovery, you name it. No one had the experience, time, or energy to implement it, so it falls on one or two juniors (me) to quickly learn and implement while learning. No one is hired with experience in any of these topics, only fresh-grads that will replicate the garbage you're trying to get away from are hired, because they will on paper be producing more features for less financial cost.
I'm a manager. Please. PLEASE don't believe this. In-the-trenches folks ARE powerful. Really. I promise. I can't implement change if YOU don't do it. I don't know what sucks if YOU don't tell me.
I NEED you to help me make things better.
Social tissue is a strange medium and most of the time it's a friction generating engine. People complain, time passes on, nothing happens, repeat.
You can change things but you must do it. You must put your reputation at risk, expense time that you wont be compensated for, go through conflicts you'd rather avoid, make lifelong enemies wether you succeed or not. Then, you changed the rigid structure once more and the next guy will either try to put it back where it was and face less resistance or fold it one more time and face even more.
You cannot teach a rigid structure agility just like you cannot teach iron to be silk, and the structure is rigid for a reason: size, cost, regulation, cultural beliefs, rot due to time, past mistakes (in my experience the largest factor: a mistake 10 years ago can justify 3 days a week of useless processes today) etc. You can change the structure but it's CEO-level work to change its rigidity.
Ironically, the only way to minimize the "game playing" is by mastering them. There will always be people out there whose only real skill is manipulating the people around them, and who will deploy that skill at full force.
I'm not saying you should manipulate the people around you... but the twin powers of, "understanding whats going on around you" and "documenting everything", go a long way.
I wrote my first custom business program 40 years ago. Since then, I've only been on two teams that didn't exhibit this attitude.
At this point, I really don't want to continue programming for a living, precisely because so many people adamantly refuse to fix problems with painfully obvious (and quick and cheap) solutions.
I've given up on trying to get people to change their behavior. If I do continue programming, my approach will be to try to find one of those rare teams that actually changes and improves.
I don't introduce new information at meetings unless I am ready to back it up, which usually takes an hour of preparation for every hour of meeting. If time is of the essence, it's often better to settle things in hallway conversations or via web chat.
This.
While I currently have a manager that mostly understands such complaints, I have learned that even then, it amounts to nothing. Because he can't change material things without invoking higher-ups, and they won't be brought on board. For one thing because that would require a lower level manager to create tasks for a higher manager. And that's not how management works in many companies: A lower level manager didn't get to be a lower level manager by creating "work" for higher-ups. He got to be a lower level manager by absorbing the complaints from the ground force, assuring them that their voice is heard while at the same time shielding the higher-ups.
Maybe I've just been in sh*tty IT companies all my life, but I'm seeing this, and similar pattern for nearly 30 years now, and that's why I'm now searching for a job that is as far away from the potential to create overbearing complexity as possible. Even to be point where salary be damned.
Quit. This is a sign of a broken and unfixable workplace.
What... does he know?
Fine, he doesn't get into the code itself. But the project and its dependencies? Yikes.
At one point, one particularly hard-pressed user wanted to make really sweeping changes to the library and had limited time. I stalled and stalled to try and review all their changes and convince myself they were going to work, and they got more and more agitated, and my TL (who was himself frustrated by how much of my time this was consuming) told them "you can merge whatever you want if you take over maintaining it". So that's what happened, and everybody left happy.
> It is your life. You own it. You run it. You create it.
> Many developers we talk to are frustrated. Their concerns are varied. [...] And the answer we give is always the same.
> "Why can't you change it?"
> But, for some reason, developers seem to resist change. They hunker down, and hope things will get better.
> [...]
> So here's the most important tip in the book.
> Tip 3: You Have Agency
> Does your environment suck? Is your job boring? Try to fix it. But don't try forever. As Martin Fowler says, "You can change your organization, or you can change your organization."
When I read the situation you describe, the best thing would be to acknowledge where the tech lead is emotionally, and sympathize, because it sounds like they simply wanted to sound off about it, not actually change it at all. It's always easy to say no to why something won't work, and it sounds like a classic drama triangle situation, with someone coming in as a persecutor, you stepped in as a potential rescuer, then you became a victim because obviously your suggestions won't work.
You, however, don't need to accept the situation, and you can try to change it if you wish. Too many people (including myself, hence why I always re-read those few lines) fall into the trap of thinking that someone else should make this or that change, make a decision, or that you require approval or authority. This is far less true than people believe.
As someone who has stepped into a software manager role for the first time from being a software engineer for most of my career, my view is that I'm not always there to solve the problem, but to give space for people to solve the problems themselves. It helps if I know what's going on, or understand the issues, but I trust my team, and those I work with, and would rather get them together with the right people to discuss it through, and work out a way forward.
I appreciate I don't know the situation at your organization, the battles you fought, the things that have been said. Change is never easy, and there's always resistance, but you do have agency, as does your tech lead. Never forget it. Good luck.
If they’re using your valid concerns as issues on performance reviews, then the company values making low quality software, not their employees. Get out of the sausage factory and get into a place that values their employees and product. You’ll be much happier.
If the team is disfunctional or the manager bad consider changing work. Not only due to the dysfunction but because a good team and manager is part of ones' mentoring, and it hampers ones' growth. At least now you gain some reflexes that will assist you in the future, if you chose to move on.
Also these deployment tasks are quite repetitive, it's a big strain, I can understand your pain. I have managed to automate myself out of this, at my current job: most deployment tasks involve the following steps: Commit the change, open a pull request, wait for the CI build to complete and download the build log, extract an image id from the CI build log, put it in some other repository and commit the change. It is possible to automate the first three steps with the github api (if your shop is using github), I have an example here: https://github.com/MoserMichael/githubapitools . The rest of it is easily scriptable.
I'd suggest to try to see the situation from their perspective, which you maybe already did, but sometimes this helps understanding a perceived lack of action.
Dont suggest, make enemies: send emails to both teams saying now it must be like that, lay the plan, pose a meeting to go through the plan and get yelled at. Once you get yelled at by the complainypants because what you tell them to do is too hard, now you can go to your manager and explain him they complain for the sake of complaining. The tech lead will get a earful and either fight you to his grave or admit defeat and let you do your reform grumbling waiting for you to fail.
I do that in a giant bank that cannot move without 10 approvers to everything, and the more these helpless teams fight me the more management put me in their midst to reform them ... and Im just a silly dev nobody and I can tell you the wonders one can do by just bulldozing. There are things to do if you fail eventually, which is to teach the higher ups to enjoy trying and failing, but that requires a bit of weaseling charisma :D
You can try but once there’s resistance is probably a better idea to find something else. Let management figure out why people keep quitting.
I don’t like giving this recommendation out but the other way can make for a career limiting move.
I feel you, it's especially frustrating when your product has such fragile and bespoke testing that things break on a regular basis. Then, us devs have to spend valuable time just troubleshooting.
They broke it they have to fix it. You need the light shining on that team, not strive to fix it yourself.
This is called being a toxic employee, I've come to think of that as a good thing but it's definitely not a career move.
Good story though, can definitely relate
A lot of times learned helplessness is true helplessness with learned resignation.
It's also caused by misaligned incentives. At most BigCo, people don't get promoted for fixing technical debt. They get promoted for launching shiny new features and products.
If they had clear, unimpeachable metrics, then you could increase your status by fixing them. Since they don't, few people engage with them. Worse, engaging with some of these problems can lose you status, and so tackling them becomes a form of self-sacrifice.
Big new features are, it’s infinitely easier to build process improvements into new projects vs fighting for improvements in existing areas.
And sometimes the technical debt cloud disappears after fully understanding the business rules and special cases which were the root cause for the code in question. This often gets overlooked and may (edit) lead to refactoring/rewrite approaches which ultimately fail. As everyone knows, not a rare occurrence.
A group of monkeys in a room. In the center of the room is a tall pole with a bunch of bananas suspended from the top.
Every time a monkey tries to reach the bananas it is hit with a torrent of cold water from an overhead shower. Eventually, the monkeys learn that something bad will happen if they climb up the pole so they just sit and don’t even try again.
If a new monkey is added to replace an old monkey, it will of course try to go for the bananas. But the original monkeys will grab and drag it down before the punishment is even triggered. After a while, the monkey gets the message and it stops trying.
Repeat the action of adding a new monkey in place of an old one enough times and you will reach a point where none of the monkeys know why they shouldn't get the bananas but they still won't try.
The story concludes by stating that even if we remove the shower at this point it won't really make a difference. The learned behavior is already deeply ingrained in the culture of the group.
Funny enough, I've seen this type of issue in many human teams/projects. And usually the inertia is far too strong for just a couple of engineering "monkeys" to radically change things if they don't get support from a few management "monkeys".
While oversimplified, it can serve as a good example on how an initially rational practice could easily became dogma even when the initial stimulus is no longer present.
Part of my day job in the past 2 years has been to consult a client in refactoring/rewriting an old platform. The client's old team is still in place and even if we've added some new people just to try and change the mindset, the inertia of the spaghetti codebase coupled with the client's short and mid term financial goals as well as the "traditions" of the old team has stopped us from enacting any groundbreaking changes.
I guess the moral is that there are many forces at play in this kind of setup and stories like these can help in framing some of them in a fun way.
Just because the fires are different every day at Startup Inc. doesn't mean we're not being conditioned to endure suffering.
Case in point, a previous BigCo on call roster that I was required to partake in, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing 5-7 of the nights in the week.
I tried to improve this process and was mildly sucessful even if that was just to get these incidents recorded.
Fast forward to my next job at Startup Inc. and their on call roster, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing less but still often enough and with higher urgency.
The source of the two problems was different most of the time, one being a lack of training and effectively giving children knives.
The other being moving fast and breaking things resulting in things breaking after it was declared that version x.x went out without any issues just like the CEO wanted.
> Hold managers accountable: one of the key responsibilities of leaders is to create positive change for their teams. Once you notice a situation where you think everyone is in learned helplessness mode, make sure to notify your manager and follow-up until the problem is addressed.
"Hey boss, I noticed that Bob and Joe are really helpless at their jobs and that you just kinda watch it happen."
Yeah what could possibly go wrong with that?
I'm not saying the thinking is wrong, but a little idealistic. Not everyone works at BigCo where it is impossible to get fired and you only ever get transferred. Nevermind most of the people in this position are going to have extremely short tenture.
My company is small and we have very small teams (1-4 people). One person recently hired has been working at successively larger companies over the years (I’ve known him for a long time). Sometimes he’ll do something like this: stop working on a task he’s been assigned, say “test data needed”, and just totally bow out until someone else makes it for him.
As I said, this is a small team. He knows how to make his own test data, and sometimes there is even a UI dedicated to making the kind of data he needs. But he has learned from working at larger companies that he can sometimes just pass his work onto someone else (perhaps even an entire other team dedicated to the thing he doesn’t want to do) instead of stepping up to the plate to do such a trivial thing on his own.
To be clear, this is just an example (which has actually happened) and other similar “not my job” thinking/actions have shown up repeatedly. Many years ago we worked together on a small team and he was not like this at that time.
The way it worked is that the person would describe their situation and then everyone else would ask questions.
The format was:
1. understand the facts, and understand if they are actually true
2. explore options, even crazy ones
3. pick an action to take and commit to it
For the questioners, the most important rule was you can't give advice. The person who is asking for help needs own it. They provide all the answers and decide on their own action. That action could be anything from talking to your manager, to setting boundaries, to looking for another job.
It worked amazingly well, many people who started off feeling stuck realized they had the ability to find a better path. Even just making a conscious decision to do nothing and being okay with the consequences of that is empowering and insightful.
I used to be completely helpless, learned from childhood. I finally realized that I'm not a child anymore and I have so many more options as an adult.
Happy to chat with you if you're feeling stuck, feel free to email me (see my profile).
Shameless plug: I stopped this to work on a related area - empowering people through team-building. The first session is free, you could do it in place of your next virtual happy hour! Check it out at https://risingteam.com.
I've gotten used to just accepting that every job I have will involve a code base that uses up way too much memory to run in a debugger and doesn't have (or allow for) unit tests.
But when I'm called upon to work entirely without tests, or with inadequate backups, or otherwise in ways that create a scenario where every keystroke I make holds imminent potential for disaster, I quit. I can do trapeze acrobatics if you let me use a net. If you insist that I must not use a net, I'm not willing to live with that constant stress.
I did that early on. Then again, because the next place was the same. Then again, because the next place was the same. 10 places later, I finally realized they're all the same.
Most of the projects/companies/teams I've worked on didn't have tests, adequate backups or had scenarios where errant keystrokes blow up production but it's nothing that has been world-stopping and these jobs rarely involved anything were downtime equated to real harm (outside of the economics of the companies themselves, but usually nobody necessarily cared as long as things got fixed).
Companies that subscribe to YOLO-level development usually already _know_ that and with it, comes a certain understanding that shit will break, it's just minimizing the blast radius by learning/knowing the code bases and leaning on those with more experience.
I've been here for 2 years now, and while there's still things to be improved in both the code base and the way we do things, we've been making steady improvement over all that time, all while managing to deliver on new features. Even at times when the progress might slow, just knowing that there is some improvements happening, and more will happen when resources allow, creates a whole different mindset from previous places where I've just given up and left.
I have one particularly memorable situation where I tried to coach someone out of this. Sometimes the New Guy is the only one who can change something at BigCo, and the moment you convince them that fighting the bureaucracy is too hard, you've lost one of the few assets that person has.
As the processes get more complex, the time between onboarding and being able to tackle projects as an IC grows and grows. That outsider perspective may be one of the few things they're good for. That outsider perspective may be the only thing that ever reduces that gap.
I can't help but feel like people caught in that uncanny valley are damaged by being there, and one of the only ways I often have to improve morale is to point out how they're doing things - important things - that the Curse of Knowledge keeps me or my peers from doing well.
I started sending people to ask the person who has been vetoing the change about why we can’t have nice things. Sometimes it works.
* A powerful tool against Pattern 2 (Complexity-related Learned Helplessness) is just cataloguing and quantifying things that cause wasted effort, i.e. a waste snake. Make a channel for it and encourage people to add items like "spent 4 hours reinstalling docker after corporate pushed an update" or "18 people X 2 hrs when the vpn was down" or whatever. Some of them may not be solvable, but quantifying the impact is the first step towards prioritizing a solution (or realizing that the solution isn't worth the effort).
* IME the solution to learned helplessness, somewhat buried at the end, is team autonomy. Self-managing or autonomous teams are the exact opposite of learned helplessness. It's nice to imagine your boss solving your problems with heroics ("Hey guys, I joined the on-call rotation this week and discovered it's terrible so I'm getting rid of it, hooray!") but waiting for that to happen sounds like more helplessness. A more realistic solution is that you solve this the same way you solve your software problems: figure out who the stakeholders are, figure out what problem they have, and devise a better solution.
A lot of the time, when one is an individual contributor in an engineering team, that is not the case. Rather it is the case that you really are powerless.
That's an important distinction to make. Don't assume that your struggle is with internal psychological forces, when your struggle really is with your teammates and the team's power structure.
The key is to find the proper amount...
As a dev, it's my job to make someone else's job easier. Whose job is it to make my job easier?
Moreover there are people who want their team to be like that, because they enjoy not having their decisions second guessed by any of the team members. These people in turn suck up all the autonomy that the team is given and produce the NIHest examples of architectural astronautics possible.
I've been in several projects where this was the case. The usual scenario starts off with me barging in and crying "cease this madness!". Replies range from indifferent through apologetic to hostile.
Eventually, after shoehorning too many refactors into my assigned tasks I either get fired or quit.
I now actively avoid such places because I'm simply not cut for such work.
Jeff Bezos on Learned Helplessness (2005): https://www.youtube-nocookie.com/embed/WhnDvvNS8zQ
Paul Graham on Schlep Blindness (2012): http://paulgraham.com/schlep.html
I fear my current workplace that was so cool years ago is showing these signs.
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-d...
But about your point I think it can depend greatly. Personally I switched jobs many times but have so far kept in the same field and my knowledge has compounded and I have never dealt behind. In fact sometimes I feel the opposite, some coworkers who have less diverse experience have less to offer.
This point resonated greatly with my own observations, both of myself and of other colleagues with each development team I've been part of.
I think another challenge is fostering a culture where everybody genuinely cares about the output that's delivered. Not everybody will. Some people are comfortable moving tickets from triage through to a resolved state day in, day out without caring whether this process or the output can be improved.
The examples mentioned are painfully real and accurate but it's possible to swing too far in the other direction too.
Too much autonomy, feedback collection, etc. can also bog down the entire team. Not all feedback is good or well-reasoned. It's easy to get stuck in a spiral of short-sighted reactive changes and the second-guessing of every past decision.
-
It takes a lot of focused effort to both empower individuals and educate them (without indoctrinating them into learned helplessness?).
The only lead I have here is to document decisions in ADR-style[1] docs so newcomers can see what else was considered and not chosen.
I'd love to hear any other success stories or tips.
--
You can’t change an org from the outside, but it’s also difficult to change it from the inside.
Eh eh eh, this feels like reading the Phoenix Project all over again: "are you me!?"
My first job out of university was a job at a BigCo, right in the middle of a Death March. Nobody at the beginning of the project remained. Most people who "finished it" were new comers like me or "team leads"(tm).
The three years I was there my teammates and I tried to push numerous changes and much needed improvements. Every time we had push back, especially by the team leads who literally shouted everyone down; I now have a profound hate for people with loud soprano voice by I digress.
I tried my best to raise up problems, find their roots and solutions. I mean, I couldn't be subtle that I wasn't satisfied with what I have seen.
And yet when I quit, my colleagues all could point out reasons of why I quit and yet none expected that I would leave.
As the article pointed out, you should quit early and make sure your reasons are known; your old colleagues will owe you and this is pretty much the only way they can get the message.
Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you.
For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity.
When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis.
At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated.
To me it's just a headache.
When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?"
When a problem can be solved by responding with "please follow the instructions in the error message you sent me".
When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
Have you ever worked on infrastructure? This comment is wrong on soo many levels.
it's basically "beautiful mind" syndrome, and it's steeped deeply in ego.
now if on the other hand that teacher is coming from a place where it was hard for him, he probably has a sense of empathy and is willing to teach.
But when ops is glitchy, I have to blow half my day faking interest in docker? I mean best case, I learn how ops works, and then what. I become the one they tap when ops is swamped, then before I know it, I'm doing ops.
And I quit, because as much as I love the ops team, I hate the gig. I'd hate to be a designer, or a BA. It's not where I find joy.
So when I hear people say.. Just follow these steps to trouble shoot, I wonder if it would be cool to tell that to our users..
Infrastructure is like application dev in that it’s a bundle of histories of feature work over time and with many contributors, and many customers with competing wants and not enough time to satisfy everyone.
Granted a primary goal should be to have simple APIs/workflows for app devs to use, but complicated insides is as natural as any advanced system.
We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up.
The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with.
So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration.
I just don't understand how you get to the point where that's business as usual.
Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.
Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help.
It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
I've also seen new hires burn an entire week because they were afraid to ask a question.
But I think the balance has definitely shifted towards asking too much help...
Study a bit, WRITE my discoveries down
Craft a message with issues and background clear to readers (that includes myself when I forgot about this)
Send message to selected recipients, continue studying depends on free time available and issue importance
If they don't read the error messages from their tools, they are not engineers: they are idiots.
You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)
How else am I supposed to know Bob and Joe are doing bad work? Should I count their LOC/day? Pore over their code and rely on my own (10 years out of date) opinion of it?
In my humble opinion, there is one (1) good way to measure developer productivity, and that is to ask their teammates, and if you won't tell me your teammate is awful, you can't really complain that I don't act on it. Of course it's possible that you tell me and I don't do anything, but at least then it's my fault and not yours.
It describes very well the phenomenon but gives advice that I can only see ever working at BigCo. Where your team is huge and if you screw up there's another one down the hall.
The truth is most places that exhibit this level of helplessness often include a small team of 5 or 6, each with plenty of tenure. With no other lateral teams to move to.
Now you want the new guy to walk up to the boss and tell him that his drinking buddies for the past decade are the reason things are falling apart and he's an ineffective manager for not noticing it. You can play with the wording but that's the message.
In the real world we have political factors like seniority, favoritism, nepotism, tenure, and nevermind external factors.
In the real world the boss doesn't actually care if the team is working well or not. He only cares that the perception of the team is positive. If the new guy threatens the perception, he's gotta go. Nobody cares if it gets fixed. If nobody knows it's broken there's no need to fix it.
Let's have less articles like this one where we basically fire ourselves ostensibly for virtue signaling and more articles about how to socially engineer your way to the fucking top. Because let's face it, that's what it actually takes.
If you work in a small company, pretty much any boss will be happy to hear feedback like "Hey, it looks like there's an impediment to smooth work, and the team has gotten used to it, but it's costing us". Bonus points if you propose a solution.(If your manager isn't happy to hear that, it's time to leave, because small companies closing their eyes to this have a tendency to go down in flames)
But you'll definitely need to learn to take the blame out of your communication. Learned helplessness is a technical term, it doesn't mean "Bob and Joe are helpless". And your manager not noticing a problem is hardly "you just kinda watch it happen".
Frame it as a possible improvement to what the team can do - because that's what it is. Yes, you might get turned down. And if you repeatedly get turned down, you have a choice to make.
> Imagine you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen Kanban work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.
> Your first reaction might be to confront this head on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience so they respect your judgement, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.
Imagine a 5 year old code base that doesn't even have technical debt: it's a manifestation of the technical debt itself. It's hundreds of thousands of lines of code, mostly with "emergent behavior". You can't release without spending days on clicking through the RC system manually to check what's broken. Also, your product is extremely complex because apparently the barrier to entry in your field of business is very high.
And you have no users.
Then suddenly users arrive.
What it takes is 1) management buy-in (they should be desperate enough) 2) engineers who are not willing to make compromises.
Also, as usual in crisis management, people who successfully manage crises are generally ousted after the crisis is over because the personality needed to solve crises is not compatible with peacetime modus operandi.
Too much technical debt, but the boss doesn't want to refactor? Refactor any code you were touching anyway.
Build process too complicated and brittle? Write build scripts, add error checking, and make other necessary changes to fix the problem.
My past bosses never asked "Where did you find the time to do that?" even if they even noticed that anything happened at all.
At one company I worked, we had no build server. The boss didn't see the need for it so he never approved it, but he also didn't reject it once we set it up.
Any organizational change that requires other to change their behavior or their opinions? It is very difficult to change. The only success I've had is by mentioning an idea multiple times and eventually my boss started thinking it was his idea.
We have a sister organization which is larger and with their own CTO. I’ve tried to boil the ocean and inspire change across both companies, it was hard, for now I'm just focusing on trying to inoculate the new hires against accepting any sub-optimal bureaucracy. Though I see that need to find some way to train them to be so open and confident that they can start to challenge processes when they need to.
I have a lot of hope in the new hires though. Culture is hard to change but definitely doable. It’s all about who you hire, fire, praise and incentivize.
I don't know how common this is in other places, but that organization could only change through external, not internal, influences.
I might be misremembering, but I can't find it in the HN search history.
Like an explanation given for how companies will let key employees quit, then excitedly pay more for new hires. New employees (or contractors) are all marketing about how great they are. Existing employees are all too obviously mere mortals with plenty of flaws. Of course getting rid of the ordinary employee and getting Superman looks like a good idea.
Manager spends time listening to employees complain: losing proposition, boring, time consuming, looks like doing nothing. Manager brings in amazing consultant who helps: manager looks good, saves effort, has a claim to need a bigger team and budget in future because evidence shows they just needed some extra help.
And reputation, enemies.. all of these are absurd notions to me. It mostly shows immaturity in our notion of work and society. And I'm not asking for people to find solution to state finance, but even for changing the simplest thing, people will resort to primitive behavior. It's child like behavior mostly.. and the bad side of childhood, not the "let's make something fun for everybody" more like "boss doesnt like me so i wont move a finger". In those structures everybody becomes everybody's enemy, and it's a super sad sight.
Most Tech Directors or Tech Managers at my massive media company were previously front end or full stack devs who were good at coordinating (asking questions and never internalizing the answers).
Since they talk to so many people and have great rapports, they can easily manage people and conversations but they cannot manage a technical system because they think they can negotiate with it.
Usually the shit they need to know is 2 bash scripts and what build is in what env. Simple enough if you are so well educated with a degree and half a brain.
If the person hiring isn't technical, they likely aren't qualifying competancy, nor are they likely capable of building an Engineering culture.
I am the expert. I know things. I choose to change my viewpoint to taking pride in making the best possible explanation that serves the users' need. When they ask me "what do I do?" I no longer snicker or sigh that they don't know or that they are too lazy to find out. Instead I copy-paste the answer they need. It takes me seconds, perhaps it saves them thinking. Thinking seems hard to some people. I choose to believe I save the company some time when I think for them.
Yes, they do not learn when they are spoon-fed the answers. Yes, they could have figured it out. So what if they didn't? They are producing whatever they are supposed to produce.
When I get too tired of the same questions, I change jobs. I am in a C-level position now, and somehow the questions from other C-level managers are similar. They have no idea how they should deal with things from my business area, and that's okey. I'm here to help. I help. When I get too tired of the same questions, I will change jobs.
devops sets up something convoluted that they for some reason like.
Developers are confused by the thing. It causes them endless problems.
Documentation is sparse, or out of date, or just plain wrong. The instructions to fix problems omit key requirements because the person who wrote them just assumed the reader knows about X and Y when the reader often has no clue. You follow the instructions and the problem is not solved, and maybe to make matters worse, you run into new problems.
The way I see it is: if developers enjoyed dealing with infrastructure, devops jobs would not exist, it would just be something that developers do as part of their job.
As a devops engineer, you should think of your fellow software engineers as your users, not as your "project members".
Your job is to make it as painless and problem-free for them as possible. If problems arise constantly, that's not the user's fault; that's your fault.
This is the truth, as far I'm concerned. We say "technical debt" instead of "bad code" because the former sounds more acceptable in the business environment. But it's still the same meaning, code we don't like, and code we don't.
Not only that, but it's been around long enough that bad management types throw the term around. That's a sure enough sign that whatever usefulness it had has been far outlived.
All of these comments should be said in group setting to try and get the crowd to pickup similar views. Just make sure no one thinks of what they are doing in a positive light.
If you have to stay, it helps to try and make your team a nice place to be. I remember a manager that would organize team lunches and would give Fridays off (unofficially) making the team an oasis in the middle of a corporate desert. But I can’t imagine it was easy for him to deal with a dysfunctional leadership and coworkers…
So either you become your island and power on from your base, or you become part of the dysfunction as a leader. Leaders who succeed in this are rare enough they write books about it, most of it fiction..
You may think I sound naive; I think you sound burned out. In my experience, it's pretty common for someone to say, "hey boss, Joe is really phoning it in these past few months, it's dragging the team down and you need to do something about it" and for the manager to either help Joe improve or fire Joe. And given how cynical your attitude is, I think you'd be very surprised at how often Joe actually does improve. Everyone will have the worst year of their career at some point, and it usually isn't the last one.
Because it’s easier than socially engineering your way up a competent organization and the paychecks are similar.
https://www.academia.edu/36290372/Monkey https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
I also try to distill good vibes and motivation, bit by bit. Listening, suggesting, showing.
I've asked a few places but I've been looking for the social dynamics of work groups.
I'd love to poke in people's head to be sure what are the motives behind their behavior (the main one I assume being a balance of respect/equality and money).
No, don't bring the company to a halt with a tantrum. Just insist on getting one thing fixed a month and nine months from now everyone will be a lot happier.
Being able to say "no" is absolutely critical.
Of course, you do incur a risk and you may fail or hurt your career in that org up to the point of termination even if you're "right". Moreover your follow-up after saying "no" will be judged harshly and silently for every little snag, real or not.
But that's part of the deal, right? It's not easy, if it were easy you'd have dream-teams all over the place rather than an industry where occupational dysfunction is normal.
> Just insist...
Some companies work like this, but many don't. Such utterances could be charting a course to the door for you.
Learned helplessness, indeed.
Similarly to your org not having a dedicated DX team being the reason, dedicated teams will happily dissapear into a void of non-delivery as their purist vision, unconnected with reality, will simply never apply to your stack, and thus you'll never use it.
End of the day, different things work and fail. The key is almost always the quality of people, their motivation and how many road blocks you put in front of them.
dedicated teams will happily dissapear
into a void of non-delivery as their purist vision
1. Rotate team members onto and off of the DX team, so they don't lose touch with reality.
and/or
2. Have the rest of developers vote (or otherwise have direct input on) on the DX team's priorities and products. The other developers are the stakeholders here.
> because they will on paper be producing more features for less financial cost.
So make a better business case. Inform them that after investing Y hours on refactoring X, future development is expected to happen M% faster with N% less bugs. New features can be expected to be developed, tested, and shipped to production in S% less time.
But be realistic. Don't make promises that you won't actually meet when the time comes.
Inform them that after investing Y hours on
refactoring X, future development is expected
to happen M% faster with N% less bugs.
(I chose 10% because we had 30 developers at the time and generally had teams of 3 developers. So, one dedicated DX team...)
Do you have any articles, case studies, etc of how to express this to management that is generally oblivious to the day-to-day, in-the-trenches, experiences and pain points of developers?
What you're saying sounds great to me, who is a developer, but how would you come up with halfway realistic numbers for Y and X?
The best thing I could think to do is have the team minutely log all lost minutes productivity caused by delays and inefficiencies that the proposed fixes could address.
But even that's tricky. For example, at one of my previous positions, the build pipeline could take hours and could randomly fail.
The only way to maintain any semblance of productivity would be to juggle 3-4 tickets at once. This was incredibly challenging, and productivity took a massive hit due to the frustration and context switching.
However, even if I had logged those experiences down to the minute, there weren't any literal downtime minutes. It was more a matter of juggling five tickets at once that should have taken one hour each, but instead they took five hours each because I was context switching faster than the Linux kernel itself.
New features can be expected to be developed,
tested, and shipped to production in S% less time.
Scrum and its much-maligned "story points" are a possible solution here, as far as management is concerned, assuming both developers and management have bought into that concept and are using it correctly.
> Do you have any articles, case studies, etc of how to express this to management that is
> generally oblivious to the day-to-day, in-the-trenches, experiences and pain points of developers?
> What you're saying sounds great to me, who is a developer, but how would you come up with halfway realistic numbers for Y and X?
> The best thing I could think to do is have the team minutely log all lost minutes productivity
> caused by delays and inefficiencies that the proposed fixes could address.
Not really. From an application perspective you can generally get away with two really big servers (one of which is the backup), and a load balancer.
There may be more involved in the surrounding infra, but that’s stuff the application developer will never have to touch.
I have worked in many aspects of web development. I know for sure that everything in the process is way too complicated for what it does. Unnecessarily so.
There's no reason to think that infrastructure is the one place where all the complexity is there for good reason.
Have I worked on infrastructure? Yes and no. Depends on what you mean.
I have worked on infrastructure in the sense that I have setup the infrastructure for websites that can handle thousands of concurrent requests, and the infrasturcure I setup is stupidly simple, everyone who sees thinks there must be something missing. But there isn't anything missing. It handles the job much better than the complicated mess that I see in all other companies.
Now, the thing I haven't done is actually setup the infrastructure for a truely distributed system that is horizontally scaled. I do all my scaling vertically because computers are so fast and storage capacity is so large that one server can handle thousands of concurrent users without nearly sweating.
Part of the reason everyone else has a super complicated infrastructure is because they chose bad software technologies to develop their applications, such as, python.
Yea, if your application backend is in python or ruby, you have no choice but to create a complicated infrastructure. Because those languages are so slow, you just cannot scale vertically. That's not an option. You get an explosion of complexity because you also need additional servers to handle stuff that would normally just be part of the application code.
For example, when I program in Go, and I need to cache some data, I just create a map, or something, to cache data in memory. But if you have Ruby, you can't just do that! Since you have 200 application instances running on aws, you want to use something like redis to act like a shared distributed cache. So now your infrastructure has to include information about how the application servers need to communicate with the caching servers.
All of this is unnecessary complication that can go away if you stop what you are doing and just analyze the problem from first principles.
Your massive convoluted infrastrucutre is only there to compensate for the bad decision you made early in the process, which was to use a slow interpreted language to write the backend of your system.
Most of the time the real solution to scaling problem is just writing better (and simpler) code.
I've seen at more than one company, how this complciated infrastructure does nothing to make the application robust because it grinds to a halt when there are about 1000 concurrent users.
Let's take an example, say Reddit. You want to support iOS, android and web. All platforms have the same core data but maybe have different APIs for optimizing queries, e.g. you may want to use server side rendering for web.
So here's some of the microservices that you will use server side. I will label which I consider to be infra:
LoginService (infra)
Database server or library (to create consistent views/transactions on top of your raw database, infra)
Core business logic server
Web frontend server
Android/iOS frontend servers (probably not needed)
Image/audio/video processing (infra)
ML server (infra)
I don't see how any of the infra services here are trivial.
I wholly reject "microservices", the philosophy and ideas behind it, the tooling and the culture surrounding it.
It's not only too complicated. It's worse than useless. It's extra work that produces negative value.
Seriously, though: I firmly believe that most managers do think and act like you do.
(I've been in lead and/or management-lite roles myself; I don't view management as some weird or evil "other")
But, I think it only works if that sort of change is valued all the way up the org chart and folks at each level are empowered, incentivized, and motivated to address the pain points of the folks below.
If the org chart is more than N levels deep, then I think this becomes practically impossible. Managers feel the pain of their direct reports deeply and acutely and absolutely want to make things better: after all, it is in their best interests as well. But a manager won't feel or even understand the challenges of folks N levels below; it's probably not even practical to expect this.
(I'll let others argue over the exact value of N above)
That's why I don't think the situation can be resolved without some sort of dedicated resources allocated to developer experience. The most obvious answer (for engineering orgs big enough to support it) would be dedicated developer experience teams. For teams that lack the headcount to dedicate engineers to it full time, an alternate solution could be dedicated chunks of time -- "hackathon Fridays", etc.
I‘m working in a strongly hierarchical org,
whit very high N number
The persons in charge, empowered to make
decisions are so far from reality
I think it's inevitable, honestly. I think it's fundamentally impossible for management to really understand or respond to in-the-trenches stuff. That's like tasking the mayor of a large city with sweeping the streets or physically fighting crime themselves, in person. They literally cannot spend enough hours in the shoes of the folks under them to understand that stuff.
And, that's fine. Management just needs to have the humility to understand that, and has to empower people/groups that do have their boots on the ground to fix those problems.
It's not unique to this industry, although I do think the newness and explosive growth of our industry may make it particularly prone to this kind of thing.
If you however, do something about it, that counts for something. Not just asking others to do your job for you, without even asking them.
Yes, there is 1% who can't follow that advice for various reasons of desperation. Not enough to continue arguing this thread.
My dilemma should I leave or not? Is it better or worse in other place? My helplessness is, I reached limits where I know I can‘t improve the situation anymore. But I see a general pattern and so I am pessimistic to find a better place…
I suspect you're approaching things too logically. In my experience most people do something then rationalize why they've done that later, if at all. And that's without any perceptive, or memory problems or anything like that, and assumes good intent-- which a lot of people don't have.
I also suspect some people's brains work so differently from my subjective experience that there's effectively no way to understand them.
A very hard lesson for me was "don't try to use logic to understand irrational people."
I now tend to think of people in terms of, "can I predict a response or action by them?" By observation, you can kind of build a set of rules and start hypothesizing about them and update your model when you get more evidence. But you still need to be mindful that different conclusions can be reached with different or conflicting information.
It worked so well on an ex of mine that she thought I was spying on her somehow, because I often had reasoned out things I shouldn't have known. The most terrible example-- that she was cheating on me. I came to this conclusion from a few bits of evidence-- she wasn't really the curious type. Most of the film, music, etc she knew about were from me. And then one day, she started talking in detail about a film I knew she hadn't seen, and she used a word I had never heard her use before.
This lead me to think, "She's having some sort of social interactions that I'm unaware of, and they're probably watching movies together, and if its being kept secret, its probably something bad."
And I was correct.
I'd rather try to connect emotionally and see who's responding. I grew up the way you think and I'm tired of it. If someone doesn't like me (or doesn't like me anymore), no problem, then I'll find other people, all I want is honesty.
That said, I've encountered the post rationalizing lying kind just too many times.. :)
Oh yeah, we had unit tests all right, fragility didn't prevent that since you couldn't commit otherwise.
So there was no safety net. Just a constant dread of customers actually using the feature. So "fast-and-fragile development" resulted in a worst-case scenario as far as engineers were concerned.
The author says : "Their reports won’t tell the manager that they are having a terrible on-call experience". This was exactly what happened to me. And I quit one day, this was a big factor, two weeks out of a month (in my rotation) I'd be on the hook for supporting this morass of problems.
Moving safer takes up a lot of, imo, unnecessary time with dubious payoffs. I'd personally rather live on the side of YOLO and ship a lot then hem/haw over getting certain % unit test coverage (and the maintenance costs that come with that in the face of uncertain requirements).
Well then you're talking about a completely different scenario than I am.
To maybe put it into examples:
World stopping: Traffic collision avoidance system (TCAS) has a bug that lets planes crash into each other.
Don't care: Not enough vegetarian meals made it onto the plane because of a bug in the meal reservation software.
In between: Thousands upon thousands of people get stranded in some airport for 3 days because of a bug in the route planning software.
(I'm also against this "YOLO because it's fun" attitude, though.)
most of healthcare software (that i have worked with) is really, really bad.
The fact that it mostly has no tests, or if it has tests they mainly test just the happy route.
They usually have lots of documentation, but it's not comprehensive and mostly just describes the happy paths, so when thing break you are out of luck. Also can't google anything since this are all closed source enterprise type software, so you are at the mercy of their support, that often also sucks.
And if you have to integrate with it, you often can't get the documentation, or tools to do it unless you are working for one of their VARs. And the VAR's are often even worse.
If you pick random 10 high school kids, get them into some code academy for a 6 months, and then have them develop an application, the ones that finish will probably be better than majority of health care software out there (for simple reason that they won't know enough to really screw up as bad.).
The only saving grace is, that its often written in Java so you can find workarounds in runtime (or inject stuff) if you really need to.