Managed services killed DevOps(techcrunch.com) |
Managed services killed DevOps(techcrunch.com) |
Overall, disappointing article.
> Good luck to whoever will manage an AWS infrastructure
> on his spare time between developing one feature and
> another.
(edit: Think about $MEGA_RETAIL_CORP. Their main-line of business is not software development, but retailing. Despite being a large company with deep pockets, they have difficulty recruiting the best and the brightest in the DevOps world to come and work with them. When $TURNKEY_DEVOPS_CORP approaches them and offers their devops as a service product, $META_RETAIL_CORP eagerly signs up.)
I really like the goog model of your SREs spending 50/50 dev time and ops time, because it forces a shift in how you are deploying things and the ops people have a big say in whatever the devs are doing (not the case in many companies.)
It is not impossible to do any of these things, but you cant get around the basic complexity problem by saying "oh this is dead now, just pay someone else to do it." because the fact is you are giving over your keys to the kingdom.
For instance, People have been saying for years that "The DBA is dead." but I work with many companies that building new database driven products, thinking they can handle concurrency and growth, and then when the real complexity starts occurring because their abstraction is a bit leaky shit hits the fan.
An argument against specialization only works when you have completely replaced that industry with something entirely better (we dont have a lot of blacksmiths anymore), and that has definitely not happened for devops.
i would prefer having a guy or two dedicated to it, but we make it work.
Given that I've done this a while, I have a bunch of patterns and reusable things, that use to deliver something quite quickly. Once it's up and running, your job is to patch. Obviously developers like to be constantly running the new shiney thing, but if you put a foot down on that, stick to a low number of well known tools, and don't chase fads, you'd have a lot of time to work on features.
However, it also depends on where you are and what you're outsourcing. If you want me to start from nothing and decide we're using a whole bunch of tech I don't have patterns for already, it's going to be a while before you see me come up for air and do any features.
It also depends on what you're hosting though. Much of what we're selling is just a fancy CRUD app, they're easy to run.
We built it on Google App Engine. If you don't like GAE there are other PaaS providers that offer something similar. Developer life on a PaaS consists of building features day in and day out. It's not mucking with Docker, Chef, Ansible, Salt, Puppet, BOSH, apt, yum, or any other low-level systems management tools. This stuff doesn't even waste space in my brain[1]. Instead of learning this stuff, my team has been been learning about the business needs of our customers and building them out in code.
I plan never to hire a devops team. Running servers is like writing assembly language; appropriate only for companies with very specialized needs.
[1] Not quite true - I used to work on the BOSH team. It's a great tool for deploying CloudFoundry. But you shouldn't use BOSH, you should use CF, hosted by someone else.
Testing was more closer to writing-code - so that got absorbed into the dev side faster. Disappearing QA was a you-blink-and-you-miss revolution.
DevOps is a little farther. Several developers are not able to wrap their heads around the deployment systems, the implications of hardware and network etc. But they will. Eventually. Because they are smart people.
I think the message of the article is that people who solely have DevOps expertise, should look to evolve to developing code as well. That way we have smaller, well-knit teams with each member having an additional speciality other than writing code.
>Developers are responsible for full test automation and making sure their code works.
Is laughable outside of [insert new sexy startup here], if only due to the difference in mindset.
Granted I'm totally biased as a QA Automation, but the analogy actually removed quite a bit of credence that this article might otherwise have held with me.
"Yeah, I'm in team X and I do the Devops part"
So we basically have an "ops" person sitting with the devs, and call the team a "devops" team. The devs have still no idea how to deploy their code. The ops have still no idea how the code looks like. But hey everyone feels better.
This is also true for many job offers in the form of "looking for a devops engineer". No, what they are really looking is for an ops person that has to sit in a dev team and do the part no one likes to do. s/He will be lucky if s/he can actually spend 10% of his time producing application code.
So? What's the problem with this? DevOps generally means someone who uses developer like approaches (e.g. scripting, automation, developing one-off solutions, etc) to solve operational issues. It is a role, and someone has to do it, and I would argue that there are people who love to do this role.
But if someone sought a job in DevOps thinking they'd be producing application code, they only fooled themselves. If someone sees some sort of status issue and forever tries to fight against their role, well that's just detrimental to everyone.
I always thought of the Devops Engineer as a person who cares about their full stack, both application development and the operation of it, and understands how one affects the other. Something a step further than just infra-coding.
The rub seems to be that quality managed services are widely available and affordable enough to obviate the need for in-house DevOps.
I have yet to find developers who are throwing themselves at carrying a pager.
DevOps isn't going away. Its just less relevant in smaller teams. Larger teams are always going to have an ops team.
Should be fun for the Fortune 500's. Part of the justification of moving to the cloud was laying off the sysadmins and other people that did "DevOps" before it was called that.
Once it's centralized, the human instinct to "ensure survival of the group" means the DevOps team will get bloated in headcount, budget, tools, etc.
I assume that will be followed by some new wave of features called the "AI Cloud" or something that manages itself, allowing you to now lay off the new centralized DevOps team :)
The reality is the field is less welcoming of specialized developers. We all have to develop, test, and deploy our code. Call yourself whatever you want but make sure you comfortable doing it all.
http://www.amazon.com/Site-Reliability-Engineering-Productio...
It would be more precise to say that the era of the code illiterate sysadmin or QA engineer is coming to an end. So is the era of the developer who didn't understand the basics of operating systems and networking.
I don't know when "DevOps" became so watered down -- I guess it's the same old story as with Lean/Agile: the real win is in changing the business process, and has nothing to do with tooling and consultants: changing processes is actually hard, so no-one does it. But when you've paid out the nose for "educating" your "team" to do agile/devops, you start to claim that your waterfall was built by Toyata. Because you paid for the privilege.
Furthermore, leveraging QA in an Agile process is critical to continuously improving our software quality.
To me, this article is more about the architectural/business decision of outsourcing as much as possible to 3rd-party services instead of reinventing the wheel in-house. How many Chef repos have the same stupid passenger cookbook vendored in? How many times have you had to fuck with that cookbook because it broke or something got updated? My websites for http://psychedeli.ca, http://brother.ly, and http://www.waxpoeticrecords.com are hosted on Heroku and, apart from the occasional Redis issue, they've been chugging along quite smoothly. I've had little to no issues with my development and continuous deployment process, which give my apps the same safeguards that I get with work (even more) with no one needed to "watch over" the project...other than me of course.
[1] http://www.ryan-williams.net/hacker-news-hiring-trends/2016/...
You need someone to liaise with the Cloud company, set everything up, test it, organize changes, and update the system regularly with new functionality. Even if a Dev had 10 years experience managing infrastructure, they still shouldn't be doing it because they should be developing products. If only there were someone whose job it was to manage Operational changes, and liaise with the Developers... Hmm.
And by the way, not everything is moving to the Cloud. We've had managed services in one form or another since the early 90's, and the result is always the same: the service provider is blamed for things working shittily and the company brings services back in-house.
And his continual refrain that "QA is dead" - does this guy understand product development lifecycle? Whether you're designing a brand new product or just bringing out a new version, you have to have QA. Can you imagine a medical device or medical web service being developed without QA, or QC?
Thanks for the troll, TechCrunch.
Second, this is a really clickbaity article.
You don't have a DevOps guy, or a DevOps team. You have a team that does both development and operations. The less work they have to do (like the reduction of effort with managed services) the better, but it's still work we need to do.
If you call out DevOps as a specific role, they you're not doing DevOps. Just like those practicing "Agile" are likely lacking in actual agility.
The servers and databases are all managed by my one boss, but I'm basically now responsible for every aspect of my application's lifecycle. While I'm a 1-man team now, I know down the line we're looking to expand and I'm trying to set things up right for when that day comes - proper version control, and an automated way to push updates to my Django app.
I was looking at Docker and the likes, but we're unfortunately running Microsoft SQL Servers and Windows Servers which apparently aren't going to see much support for virtualization/containers til the SQL Server 2016 version comes out. My company is an engineering firm, not tech, so paying for an AWS bucket or me making a case to get the fuck away from Microsoft backend stuff (the headaches I've had with MS SQL already...), etc etc probably isn't high on their to-do list.
Maybe due to the sort of company I work for this article just doesn't apply to me anyway, but it seems like the author assumes any company rolling out software or a service is doing so at the cutting edge and as its primary focus.
I'm not sure my struggle to find a sane way to automate our deployment and updates would be helped too much by literally any of the cloud based services anyway. Even if they gave me a green light to go all-out on whatever service I wanted to use, at the end of the day I'd also have to manage all of those services and their connections to the app as well. I'm trading off some work for another kind, and a lot of these cloud solutions are hardly painless.
> HTML used as the screenshot is as horrible as article.
+1
I guess our company didn't get the memo?
Then Parse shut down.
I think the lesson here is, you should never trust your core product to a third party. Also, when you use cookie-cutter services, you really limit the kind of things you can do, and vendor lock-in really, really limits your options in the long run.
First, it isn't actually true: Third-party managed services (say, exotic databases) are a mere network hop away. If you really had to you could call out to your own self-managed services in EC2 or whatnot. Even on Google App Engine, which is one of the most restrictive systems, we use a multitude of services running elsewhere in "the cloud". This is occasionally a mild annoyance, but it doesn't significantly alter the value equation.
Second, it's irrelevant: Migrating our application off of GAE would be tough, sure. You can call this "lock-in" if you want, but it ranks somewhere down a distant thousandth after a long list of concerns about my product and how well it is meeting the needs of my customers. Worrying about lock-in is a premature optimization.
I imagine that when Flight Navigators started disappearing from flight crews in the 1970s with the advent of GPS and other related electronic navigation systems, there were those saying "I'd never leave the ground without a flight navigator - what if the GPS stops working?!?" Should we keep an old school Flight Navigator on standby "just in case"?
I remember talking to an retired mechanic about 20 years ago, who told me he never did a road trip without taking all his tools with him so he'd be ready to fix anything on the side of the road if need be. I wonder if he is still doing that....I doubt it.
As technology gets better and better, the "Number Needed to Harm"[1] needs to be considered.
[1]https://en.wikipedia.org/wiki/Number_needed_to_harm
edit: missing word
"The first software companies had thousands of developers. The second wave of software companies had hundreds of developers. The next wave of software companies will have 1 developer."
Can't find the source, might not be the exact quote either, sorry.
This has happened with Parse, and this will happen again.
Does DevOps or SRE's not exist in this realm?
Fyi, the title is using the rhetoric template of:
<SOMETHING> is dead. Long live <SOMETHING>. [1]
It's as if someone wrote "Cars are dead. Long live cars." to signal the change from humans manually driving the cars to AI robots doing it. If you read the "cars" literally, it looks nonsensical because there are obviously cars existing both before and after the shift to automation.
[1] examples:
http://www.pcoitmurphy.com/_books_are_dead__long_live_books_...
https://www.tnooz.com/article/hotels-Airbnb-rented/
http://www.wired.com/insights/2014/08/moocs-are-dead-long-li...
It was used when the king died in the sense of "the [old] king is dead, long live the [new] king".
So in that light it means "[the old way of] devops is dead, long live [the new way of] devops".
I recently created a private cloud infrastructure on Digital Ocean. It would have been easier and faster (and more polished) if I'd gone with AWS, but then I would know how to use AWS, and would not know how to do these things myself.
Why learn about HAProxy, keepalived, tinc, etc when you've got AWS to handle it all for you?
Answer: Because relying on a service to abstract the inner workings for you is a path to the dark side.
I also like to think of using a cloud service as "hiring" the entire staff working on that product + benefiting from all of the accumulated knowledge that's been baked in. That makes a cloud service a huge bargain to me.
Deadlines, I guess.
Then there were systems programmers and application programmers. Systems programmers wrote operating systems and utilities for them. App programmers wrote apps. There was a lot of crossover.
Then there were operators, systems programmers and application programmers. Operator was a junior position who did physical things (mount tapes, plug in cables) and ran commands to do things on the systems. They usually moved up to being...
Systems administrators, who did some programming in service to the systems, but not not much. The more senior a sysadmin was, the more time they spent programming and the less time they spent doing physical things... unless they wanted to do that.
Sysadmins started to specialize. People who configured switches and routers and talked to telephone companies became "network engineers". People who spent time working on firewalls and security policies and thinking about that became "security engineers". Junior people who read scripts to end users became the helpdesk. And so forth.
Then we noticed that a bunch of people were doing things manually when they should be automated. This was especially bad in places where there were no senior sysadmins or systems programmers. But we did have the internet, and senior sysadmins got together and started writing tools to make their lives easier: infrastructure automation.
You probably know the story from there, but I'll wrap up with one more important point: you know how when writing a business application you need to have a subject matter expert who actually knows what they are doing? Operations is exactly the same way. All the automation in the world won't help you if you don't have someone around who knows what they are doing. Some people can outsource this to "the cloud", but not everyone.
I think you see devs make this comment cause they think they are doing "ops work" instead seeing it as consuming the ops team's work. It isn't about everyone doing equal code commits. It is about speeding up things across the whole business. Every ops person you hire is money you don't have for a dev. It is more efficient to enable the devs than it is to hire ops people to do a small portion for their job.
Same goes for QA. The groups that have really got the message are including QA teams that handle much of the testing via automation. A good QA team writes code. There is plenty of development going around. It just might not be on product code itself.
There are more and more opportunities for QA Automation with the ability to write those procedures, which I suspect are the ones replacing the manual QA specialists you reference.
Or you could say that ops now has the responsibility to scan all containers for old binary code/bad behaviour, and throw the container back over the fence if they find an issue.
I do agree that containers are a good start on having such a "contract" -- I'm not sure it's enough.
The solution that worked well for me was using tinc to set up a VPN so all my servers, no matter what datacenter they're sitting in, can communicate with each other using private IP addresses (tinc has added benefit of encrypting, which DO's private network does not).
If you're needing help, I'm currently looking for more contract work :) Check my profile for contact information.
This is a case of the wrong tool/service for the job. You need advanced networking and spent all this time duct taping it together on DO when you could have a real VPC on aws with security groups, subnetting etc. Not to mention locked in to DO's limited instance types.
Don't get me wrong, DO fills a niche for quick one-off type applications but it is absolutely no substitute for a real IAAS like aws, gcp, azure... Investing time in trying to turn it in to one is a waste and way worse than perceived "lock-in".
You also could have just as easily rolled your own setup the same way on AWS, so i'm not sure what argument is other than cost. (btw t2.nano instances are $4.75 a month) http://www.ec2instances.info/?cost=monthly
But if you have to rebuild your whole company, because everything you use runs on GAE, 1 year is almost guaranteed death for your company.
For a point of reference, when Google rolled out the High Replication Datastore, they kept around the Master/Slave datastore for years and even built a pushbutton tool to automatically migrate apps. GAE has been around since 2008; this is not an experiment like Wave.
I recently migrated from a dedicated DC to AWS because of business reasons (ie. CAPEX vs OPEX).
Before, my PostgreSQL DBs were always perfectly in sync because they were on a dedicated, private network. Now, I often get large replication delays because the network in EC2 isn't of the same quality I had before, so I've had to spend a lot of dev time working around the delay issue with creative caching solutions.
My point is, there's trade-offs and mostly my calculations show that they're not really worth it, if you have knowledgable Ops people. Which, once you grow to a certain size, you should.
For instance, I went with running PostgreSQL instead of RDS because I have much more control over what's going on in the DB than I would with RDS.
But your point that the managed services don't require ops might be true in the short term, but what happens when something breaks? Or you experience a level of growth that expands past the capabilities of the services you've organized? There's hard practical limits to all the of SaaS offerings - mostly latency and throughput - that doesn't really come up in these discussions.
The analogy about it being like assembly programming might be more apt than I originally thought, given how not doing it yourself generally results in slower and poorly optimized solutions, but then "good enough" is often the mantra of the day at a certain scale.
It's basically the ultimate technical debt.
The situation I'm in is our apps currently do low enough traffic to exist in a PaaS, but they weren't built that way from the ground up and live on bare metal currently. Migrating to say, AWS, would using ElasticBeanstalk be burdening myself with technical debt I don't currently have vs using something like CloudFormation (Terraform) and CodeDeploy?
There's plenty of things that won't, for a long time, so you're fine to save the time since you won't need to re-do it later.
But if you can really see the product expanding to a large scale, you need to think a year or more into the future when planning your infrastructure.
All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore.
EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.
See also: https://github.com/blog/1261-github-availability-this-week
> The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.
https://engineering.pinterest.com/blog/open-sourcing-pintere...
To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.
As management tools mature and automatic, systemic optimizations are developed for SaaS systems, I wonder if the analogy will continue and areas where you need to manage it yourself to eke out performance and flexibility will get fewer and farther between over time.
Given a non leaky abstraction, ignorance of whats under the hood is just fine.
It is unthinkable right now, but what will happen with Amazon/AWS goes bankrupt and no longer exists? All these AWS Certified Engineers who are not actually devops will suddenly be lost.
On the flipside, if you learn everything from the ground up and then jump on AWS or whatever service is available, you will be in a much better position when the day comes to move away and to something else.
Abstractions in something like programming languages is fine because the language won't simply disappear one day - you'll evolve as a programmer and will be stronger for it.
Or they could just read this document and move to GCP.
https://cloud.google.com/docs/google-cloud-platform-for-aws-...
But I really believe someone will figure this problem out, and at that point Dev Ops will be done. There is just to much money to be made.
Minus vendor lock-in. Once you have your infrastructure dependent on ELB, S3, Redshift and all the other AWS niceties, you are effectively locked into Amazon. Locked, in the sense that migrating out has a relevant knowledge cost (deprecation of knowledge of AWS, need for gain in alternative knowledge).
It's not a dumb choice, but it's also not clear for either side. Running your own iron is much much cheaper. For my case (email service provider), our bill estimate on AWS would be about 300k€/year, while running our own servers costs 84k€/year (split at 48k€ depreciation and 36k€ operational costs).
Something to note about US orgs: New depreciation rules allow you to depreciate $500K/year in hardware, in that year. That means that gear is yours, forever, whereas that $500K/year in opex to AWS is recurring forever.
I'm stealing this. I love it!
Difference between Ansible and package manager feels like difference between procedural programming and OOP.
Difference between Puppet and package manager feels like fight between two competing object oriented models.
In my cases, I define few variables, then I will install repository definition and main package:
export NODE=2
dnf install -y http://..../staging-release.noarch.rpm
dnf install -y node-backend
It works fine on bare metal, vagrant, docker, aws, etc.
Know the fundamentals, and you'll be fine. Everything else is rhinestone marketing.
What will you use? Bash? Python? Perl? Why not something like Puppet that keeps track of changes it needs to apply and has many useful features on top of it?
What happens if you write your installation scripts to use Ubuntu, but the order comes from high up to move to CentOS/RHEL? Did you write your scripts to be distro-agnostic? Because Puppet's `package` is.
Ops tells me that they are using configuration TOOLs. I tells them that I am DevOps, so I use SERVICEs and SCRIPTs (clients) instead. They work automagically, without need for me to reconfigure servers manually.
Am I from the past?
Maybe your "services" and "scripts" are really sophisticated. If true, then perhaps you've implemented your own configuration management tool. If so, maybe you could share it and other people would find it useful too.
The reason, why I don't use configuration tools is because I do no configuration in production. One of main rules for me is to perform all installation and configuration steps in controlled development environment, then test them in staging and only then deploy verified result to production.
Imagine that you are fully installed and configured everything at your new production host but in your development environment, then it powered off, deployed to production environment and powered on again and it works. It is result I want to achieve: deploy only ready to use software to production, in sealed form, so no installation, no tweaking, no anything. Like smartphone on contract. It can be achieved with docker, but docker is relatively new invention and my solution plays well with docker too.
So, no, my scripts and services are not a configuration management tool.
In general, plan your changes ahead of time, test and automate them at staging servers, prove correctness of changes, and then deploy to production. Never do anything by hands (or by configuration tools) at production servers.