Going into a "cloud native" stance and continuing to micromanage containers, VMs, databases, message buses, reverse proxies, etc. seems absolutely ridiculous to me. We're now using exactly 2 major cloud components per region: A Hyperscale SQL database, and a FaaS runner. Both on serverless & consumption-based plans. There are zero VMs or containers in our new architecture. We certainly use things like DNS, AAD, VNets, etc., but it is mostly incidentally created by way of the primary offerings, and we only ever have to create it 3 times and its done forever and ever - Dev cloud, Prod cloud, DR cloud. And yes - we are "mono cloud", because any notion of all of Azure/AWS/GCP going down globally and not also dragging the rest of the internet with it is fantasy to me (and our customers).

When you literally have one database to worry about for the entire universe, you stop thinking in terms of automation and start thinking in terms of strategic nuclear exchange. Granted, one big thing to screw up is a big liability, but only if you don't take extra precautions around process/procedure/backup/communication/etc.

The benefit of doing more with less also makes conversations around disaster recovery and compliance so much easier. Our DR strategy is async log replication of our 1 database. I really like the abstraction of putting 100% of the business into one place it magically showing up on the other side of the flood event.

How about this for a litmus test: If your proposed solution architecture is so complicated that you would be driven to IAC abstractions to manage it, perhaps we need to re-evaluate the expectations of the business relative to the technology.

- We need to get rid of YAML. Not only because it's a horrible file format but also because it lacks proper variables, proper type safety, proper imports, proper anything. To this day, usage & declaration search in YAML-defined infrastructure still often amounts to a repo-wide string search. Why are we putting up with this?

- The purely declarative approach to infrastructure feels wrong. For instance, if you've ever had to work on Gitlab pipelines, chances are that already on day 1 you started banging your head against the wall because you realized that what you wanted to implement is not possible currently – at least not without jumping through a ton of hoops –, and there's already an open ticket from 2020 in Gitlab's issue tracker. I used to think, how could the Gitlab devs possibly forget to think of that one really obvious use case?! But I've come to realize that it's not really their fault: If you create any declarative language, you as the language creator will have to define what all those declarations are supposed to mean and what the machine is supposed to do when it encounters them. Behind every declaration lies a piece of imperative code. Unfortunately, this means you'll need to think of all potential use cases of your language and your declarations, including combinations and permutations thereof. (There's a reason why it's taken so long for CSS to solve even the most basic use cases.) Meanwhile, imperative languages simply let the user decide what they want. They are much more flexible and powerful. I realize I'm not saying anything new here but it often feels like as if DevOps people have forgotten about the benefits of high-level programming languages. Now this is not to say we should start defining all our infrastructure in Java but let's at least allow for a little bit of imperativeness and expressiveness!

CDK, CDKTF and Pulumi all use general purpose programming languages, so reusing parameter objects in the way that is described is trivial - indeed it is so close to second nature that I would not even think to write it down. Indeed, it's not uncommon to share functions that make such parameter objects via libraries in the package ecosystem of your choice.

I agree that IaC needs a rethink, but that is more to do with the fact that declarative systems simply cannot model the facts on the ground without being substantially more complex.

Sure, I can get behind this. Yesterday I was trying to figure out how to give a name to EC2 instances generated by AWS-managed autoscaler group that’s created by a node group resource. Simple, right? should just be able to add a Name = $tag field to the node group somewhere to apply to the generated ec2’s?

well, not quite. What you actually need is a separate autoscaling_group_tag resource.

Well, that resource needs a reference to an autoscaling group arn. but I dont manage an autoscaling group, my node group does, so in the end I have to figure out how to reference it like:

aws_node_group.node_group.resources.0.autoscaling_groups.0.arn

well, not quite, you may need a try block around that, and maybe some lifecycle rules to get around weird race conditions.

so yea. I’m not complaining about HCL or terraform. I find it much better than the alternatives. but lots of times my reaction to stuff like this is “there’s no way it has to be like this.”

the structure and syntax for AWS is entirely different from Azure is entirely different from GCP.

Instead of abstracting to CSS, I would argue modeling what Bytecode did in java for multi-operating system, we should do for infrastructure of code.

That way, you could easily replicate in different environments, free yourself from vendor lock, and have readability/re-usability all in one.

This is what I want from infrastructure as code and I have yet to see it.

OAM has a model of components (things like containerized workloads, databases, queues), traits (scaling behavior, ingress) and in the latest draft, policies that apply across the entire application (high availability, security policy).

It's all a little disjointed and seems to have lost steam. KubeVela is powering along, but it's the only implementation, and IMO it's highly opinionated about how you do deploys and works well for Alibaba and perhaps not for others. But it has some interesting ideas.

I feel IaC really peaked around Puppet 3 and Chef 1. IaC should be simple enough that people use it, and trivial to write providers for. People tend to glue much too large libraries to their IaC platforms and end up with a maintenance mess which is what kills it in the long run. However both the above projects went corporate and grew legs and arms and a billion other features that everybody won't use more than a subset of. Most people migrated to Ansible which kept more of the open source project culture and was simpler in design.

Now people seems to use a little of this, a little of that. Some Ansible, some Terraform, some other stuff. They don't know what they're missing when the entire stack is built ground up from templated components defined in a common declarative language. Some people seem to really like Nix, which I haven't used professionally, but from what I've seen it seems to inherit the same type of design. There was an experimental project called cfg which worked in real time using hooks such as inotify which was promising, if there was a Kubernetes distribution made like that it would be really easy to manage components that didn't belong to a host.

But, hey, when looking at the origins of OOP and its main uses (back then: Simulations and UI): Maybe this is exactly what one needs to describe and setup infrastructure and there have been various projects going in that direction.

Make message passing truly async, throw in Garbage Collection, make it dynamic (i.e. creating a new instance of an object leads to some sort of deployment) and voila: Your traversable, introspect-able object graph is now a representation of your infrastructure.

When I need to provision anything I have a powershell script that interacts with Azure CLI.

My script sets up a new resource group for every service we create, logging, key vault, webapp/functions, and if needed some kind of data storage or queuing.

In my powershell script I can via a variable indicate which environment I want to spin up: dev, staging, prod.

I have one yaml file which is for my build and a build trigger which points to the above powershell script with the given environment.

All environments: dev, staging or prod are setup manually with manual user assignments for deployment access etc.

It's really lightweight but I also believe it's lightweight because we run a small services setup where each service takes care of its own provisioning.

Terraform and Yaml are so verbose but that's not the most problematic. You can't execute those files from your local machine.

I use Terraform HCL 40 hours a week, but it is severely lacking in lang design and type system and IDE/LSP experience.

> Centrally updatable: Sometimes best practice or corporate policy changes over time. You can update what LowCost or SecurityPolicy means later on, in one place, and that change will reapply to all resources that used it.

It sounds great but it's not. This is essentially the Fragile Base Class problem. You may _think_ that updating one of these traits in a single place will be safe and do what you want, but it may be disastrous for whoever is using it. And you're not going to find out until you deploy it.

Dark is a good example of something that sidesteps this stuff by more fine grained integration of infrastructure and app code.

It’s an interesting idea. My initial reaction was “you can take my HCL from my cold dead hands” but I can’t seriously argue that Terraform is perfect and that I enjoy writing so much boilerplate.

@firesteelrain said, "you can do that through abstraction. You "include" your Terraform Azure Provider or Terraform AWS Provider. At the end of the day, your module needs to know what it’s interacting with but not the higher level of abstraction. We have done it at my work to make it cloud agnostic just in case we need to go to another CSP"

Single ops eng in a 3 person startup here. Ops eng is only one of my hats right now :) I found crossplane to be a solid tool for managing cloud inf. My assertion is that "the only multi-cloud is k8s" and crossplane's solution is "everything is a CRD". They have an extensive abstraction hierarchy over the base providers (GCP, TF, Azure, AWS, etc) so it's feasible to do what firesteelrain did. My client requirements span from- you must deploy into our tenant (could be any provider) to host this for us.

I can setup my particular pile of yaml and say - "deploy a k8s cluster, loadbalancers, ingress, deployments, service accounts (both provider and k8s), managed certs, backend configs, workload identity mgmt, IAP" in one shot. I use kustomize to stitch any new, isolated environment together. So far, it's been a help to have a single API style (k8s, yaml) to interact with and declaratively define everything. ArgoCD manages my deployments and provides great visibility to active yaml state and event logs.

I have not fully tested this across providers yet, but that's what crossplane promises with composite resource definitions, claims and compositions. I'm curious if any other crossplane users have feedback on what to expect when I go to abstract the next cloud provider.

cyberax's note on state management is what led me away from TF. You still have to manage state somewhere, and crossplane's idea was- k8s is already really good at knowing what exists and what should exist. Let k8s do it. I thought that was clever enough to go with it and I haven't been dissapointed so far.

The model extends the k8s ecosystem, and allows you to keep going even into things like db schema mgmt. Check out Atlas k8s operator for schema migrations- testing that next...

I also like that I can start very simple, everything about my app defined in one repo- then as systems scale I can easily pull out things like "networking" or "data pipeline" and have them operating in their own deployment repo. Everything has a common pattern for IAC. Witchcraft.

My personal take is that at some point you are better of just using a full programming langugage like TypeScript. We created TySON https://github.com/jetpack-io/tyson to experiment with that idea.

One thing that's sorely needed, especially for beginners, is something like a schema. Something that would provide editors with typical language features, but especially autocomplete. Maybe protobuf would work but I've also heard about some language called Cue that may be worth exploring as well.

I also feel that declarative is the wrong approach. Building infrastructure is inherently imperative and making the build process apparent in the code would go a long way towards readability. I'd love to be able to read through the terraform modules like I'm reading a story about how the system gets built.

1. A low-level, open-ended language for describing infrastructure; it should have absolutely no facilities for abstraction, should be human-legible and machine-readable (so based on JSON, probably), and should be applicable to everything from configuring physical hosts and switches up to containers.

2. For each kind of infrastructure, a tool which can apply that language to the infrastructure; one for AWS, one for physical hosts, one for Kubernetes i suppose, etc.

3. Tools and libraries for producing documents in that language from more expressive, concise sources; could be a YAML-to-language compiler, could be a classic Ruby DSL, could be a Python API, could be this guy's CSS idea, could be a GPT prompt, whatever.

Mostly, i want options for the last part to include libraries in sensible programming languages. Then i can just write real code, with full abstractive power, and the possibility of unit tests etc, to define my infrastructure, run it, and feed the output to the applier tool. No more enterprise YAML engineering. No more trying to shoehorn abstraction into Jinja2 templates. Just normal code.

Because the code produces the language, rather than operating on resource directly, writing a new library / DSL / whatever, based on a cool new model which will solve everyone's problems, becomes very easy. You don't have to build a whole IaC tool from scratch.

It also means you have an obvious and simple checkpoint to apply diffing, linting, security checks, etc. Not on the input code, but on the resulting document.

And it means you have one place you can always look to determine the ground truth of what is going on.

It seems like this will just amplify mistakes when a lowly dev tries to increase the available RAM of their resource and instead doubles the entire RAM allotment of a resource type for the entire enterprise.

I think HTML + CSS is an example of a declarative system where you can start out not knowing very much about it, just drop in Bootstrap or Tailwind, and start getting great results by using prebuilt CSS classes from someone else.

This is what is missing in most modern infrastructure as code. Sure you can start out with prebuilt IaC templates from someone else, but these templates are basically like getting handed a big chunk of HTML that has inline styles on it. It might render great in the browser and look great, but its hard to read, its hard to understand why it works, and you'll have trouble adding on your changes to it without breaking things.

What I'd like to do is decouple the semantic aspects of infrastructure from the specific configuration aspect, similar to how HTML + CSS lets developers write their semantic markup with semantic CSS class names, and then have a CSS framework provide the exact styles that make it look pretty.

Infrastructure as code needs a similar standard library of semantic configuration mix-ins that you can apply to your infrastructure as layered mutations to produce the final result. There are many tools out there approaching this challenge right now from different ways, and I think the future of infrastructure as code is going to look quite different from what most people are doing today, more like HTML + CSS, or imperative code, than flat YAML and static structures that must list out all their own properties.

The layering of distinctly-defined concerns contained in separate files which collectively project a merged specification to an IaC tool is a good idea, I think.

I've been using CDK since early beta, and have actively contributed to the project. But most people that I'm seeing using it today are just wrapping up new higher abstractions with a simpler but limited API. I think that is an okay start, but I want to encourage people to think more about the infrastructure as being made up of traits/classes/adjectives that are mixed together to form the final product. The same way we have class inheritance in object oriented programming, or CSS classes in HTML.

Eventually the dream is to be able to provide a library of standard infrastructure as code mix-ins that can be applied to your cloud architecture. For example imagine if you could apply a generic "Graviton" trait to a CloudFormation or Terraform or Pulumi or CDK stack and it would automatically configure the appropriate properties on your EC2 instances, and your RDS database, and your Fargate tasks, and all your other compute. With CDK's built-in container and image builds it could even run your Dockerfile based build inside of the matching architecture as well, all based on a single trait that you add to your stack.

There are a wide variety of these types of "traits" that you might be able to build and add.

If you sit down with the terraform specifications for an AWS instance, a GCP instance, and an Azure instance, and start trying to write that harmonization, you will rapidly discover why for yourself. Even just trying to specify a network setup and putting an instance on the public internet is impossible to harmonize, without making something so lowest-common-denominator it is almost useless, let alone anything complicated.

This!

We're essentially writing locked-in vendorscript. What we want is an actual infrastructure language. One that lets us write once and deploy anywhere (nods to Java).

That would also allow us to standardize the way additional tooling (monitoring, logging, etc.) hooks into everything. It would allow us to easily deploy to new environment types as they become available (I keep hearing about the wonders of WASM). It would allow standardized ways of doing ops testing.

Say I have resource A with property X=1 I define in IAC. Someone comes along and modifies X=2 outside of state. With your way, the IAC tool would see that change and think it was naturally part of the desired state, whereas stored state will catch the drift. And before anyone says “well dont modify outside of IAC” I say 1) that’s often impractical and 2) sometimes automation can modify resources outside of IAC beyond your control.

Also, dynamically creating state creates all sorts of concurrency issues, which is another nice thing about stored state, you can put a lock on it.

Have you ever actually used terraform? You execute it from your own local computer, or from CI/CD. It runs in a compute resource you own, not the cloud provider.

You can execute terraform from local machine just as easily as a powershell script. I dare say you could even make it work a shebang if you wanted (though I’ve never tried that).

Sounds painful, as you have to make it declarative yourself, while Terraform (which runs perfectly fine on my local machine) is already declarative.

https://learn.microsoft.com/en-us/azure/azure-resource-manag...

I have found it quite joyful to use with Azure.