I really disagree with the pros listed on over-engineering, specifically "future-proofing" and "reusability". I doubt you can accurately predict the future and whatever assumptions you make will likely be wrong in some way. Then you are stuck having to solve the problem that you created by trying to predict. As for reusabilty, it's similar. Start with solving what you have to, then abstract as you see it fit. Again, don't try to predict. Be thoughtful and really understand what is actually happening in your system. Don't try to follow some pattern you read online because it seems like a good fit.

Realistically you should engineer for the problem you have or can reasonably expect you are going to have pretty soon. You can solve future problems in the future. I'm also not saying to write horrible unmaintainable code, but don't try to abstract away complexity you don't actually have yet. Abstractions and where to separate things should become apparent as you build the system, but it's really hard to know them until you are actually using it and see it come together.

One challenge with the term over-engineering is that it implies that the over-engineered solution would be generally superior to the under-engineered one were it not for the extra cost. The article rightly points out that this is not true, but it's something that really isn't discussed as much as it should be.

A good example of this would be avg's teardown of the Juicero, in which IIRC he described it as under-engineered despite it's expensive ultra durable components. The rationale being that rather than build a design suitable for the purpose of squeezing juice out of bags they built a machine that was specced for a much more demanding task, thus driving up costs and wasting materials. The implication being if they'd spent more time or care engineering it they wouldn't have poorly engineered over-specced components.

Perhaps a preferred term should be "well engineered" or "poorly engineered". A well engineered thing is something that is well suited in a number of different dimensions, including product capability, business needs, cost (and its impact to end users in terms of price), etc. That sometimes means ugly code, it sometimes means technical debt, but it always implies elegance at a higher level than just the code or components, but an elegance that encompasses a wholistic understanding of the context in which that code exists.

In the software world some examples of poor engineering might be using kubernetes for a small internal app that could run well on a single VM or container. Or, in a different context NOT using kubernetes for the exact same app, but in an organization where k8s is standardized, thus creating more inconsistency and driving up organizational complexity in order to reduce local complexity.

The two terms imply "engineering" varies along only one dimension. I personally don't find these terms useful or constructive for anything apart from "talking smack" about engineering decisions outside of your control, influence, or understanding.

The issue isn't over or under engineering by these definitions. A good solution needs to satisfy both all requirements, some of which may be clear from the onset and some of which need to be discovered. The issue is how one handles unknown requirements. So often people either guess what unkown requirements will be (which is how this piece defines over-engineering), or ignore them (which is how this piece defines under-engineering). You should be doing neither. Instead you should be decoupling your known from your unknown requirements so that you are agnostic to what solutions need to be implemented down the road. You don't need things that can handle problems you don't have, but you need to be able to easily rip out and replace parts of your solution as they become inadequate. You don't need to handle every edge case, you need to design things to fail safely by default. You don't need to hold off on making decisions until you have information, you need to make decisions you'll be happy with no matter what information you receive later. A robust solution can still be quite lean.

I always live my life by: You can always make something more complicated, but once its complicated (and more likely deeply re-used and embedded in your system), you are going to have a hard time making it simple again.

I practice something I sometimes call negative space, which sometimes ends up about as vague as it sounds.

My gold standard for well written tests is if the engineer who broke the test can fix their regression without looking at the test implementation, you have achieved nirvana.

My gold standard for well factored code is if people add a feature exactly where you would have added it. But that can be arrived at through socialization or by leaving spots where a feature would need to go if you actually need it.

You don’t need to build in conditionals for speculative features. You can just think about how you would start. What’s the first refactor? Can I arrange the code so that’s not a pain in the ass?

Bertrand Meyer felt that actions and decisions should not be mixed. For one they make testing a pain. They also increase the lines of code in impure functions, which reduces scalability of your system. A common effect of new features is adding more complexity to the decision process, so it’s easier to add 3 lines to an 8 line function than 3 lines to a 40 line function.

There's a middleground that many developers have trouble finding :

You can design your code so that it'll be easier to evolve into the most likely path. However, you don't actually implement the future cases.

Example: it doesn't take a lot more effort to create a configuration struct instead of hardcoding a value. However, you don't want to implement handling of any other values that the one you planned on using. You can easily throw a "value not supported" error if the configuration has anything else.

However, this will greatly help any newcomer on the codebase to understand what possibilities your component potentially offers and how it can evolve.

How about some actual examples? This seems like a fluff piece article written in 10 minutes without much real content.

Scope context matters in hitting the Goldilocks spot in the engineering.

But given the choice of over- vs. under-engineering, overshooting a modest amount absorbs the inevitable scope creep more readily.

Newbie devs think under-engineering is a problem. Experienced devs know over-engineering is a problem.

Scalability and Reusability appear to be contradicted by Reduced Agility in the pros and cons of, “over engineering.”

I appreciate the attempt to carve a positive definition of over-engineering. However I think most people will disagree that there are any pros to it as their definitions tend to be quite negative.

While I tend to agree that anticipating future needs is best avoided, there are situations where it can be done successfully. It’s good to recognize that these situations are rare before considering it. When building a system for a project where you have literally built the same thing before more than once and the team you’re working on lacks the experience to understand or appreciate the complexity of the problem, this is one area where you can get away with what appears like, “anticipating future needs,” but to you is, “solving the problem that will come up before it’s a problem.” There are times where you can avoid learning a lesson the hard way (again).

Update However one must also be mindful to avoid chasing ghosts. It can be detrimental to progress to anticipate problems you’ve encountered before as, “the same as,” ones you’re facing now or to imagine they are there. Always good to pause and get a rubber duck session going to bounce your ideas off of before heading into the weeds.

He's right.

When the author says over-engineered stuff is future-proff and all of that, he's implying _successful_ over-engineered stuff. Same for under-engineering.

Of course most over-engineered projects fail. Most under-engineered too. That's why we learned all those things people are parroting in this thread (we parrot because it's useful).

However, _some_ over-engineered projects will succeed. Those will most likely present a nice combination of future-proofing, scalability and reusability at the core of their success. Some under-engineer projects will iteratively limp their way into excellence as well.

He's not asking you to think of scalability and such, on the contrary, his recipe for following the middle way is quite reasonable (although a bit generic, like these kind of things always are).

It's easy to over engineer, using more resources and time than is required to solve a problem. The real challenge of engineering is to fit the cheapest fastest practicalest solution to your problem. Most solutions seem to start out over engineered. At the start of a new project you are looking for any solution that works. Once something works, it requires more time to improve on the prototype.

It doesn't seem like Glassonos is describing over engineering as I understand it. Possibly he's describing scope creep? In general, people should actively discourage extra scope from creeping in to their projects. But I need some examples to illustrate what I'm talking about, as does the OP.

It depends on use case.

If what you are building will be public, then: "over-engineer the concept, under-engineer the implementation". When you over engineer the concept, you start thinking about what might come next, you start seeing different applications on top of your solution. But deliver only what's needed now.

if what you are building is fully internal: "under-engineer, move as fast as you can so you have an idea how well to under-engineer next rewrite of the system"

Write code for the problem you have in front of you today. Stop.

However: If you have senior devs knowledgeable in the domain who can realistically show how the system has changed in the past, write code to accommodate those types of changes because it is far more likely that the system will change again similarly than change some other way. If someone pipes up with some "what if" hypothetical, shut them down.

I'd add the biggest risk of over-engineering: Building the wrong thing. Very often, the needs we anticipate end up not being realized.

My rule of thumb, design for what you know is a requirement, account for what has an 80% chance of being a requirement (as in, already in the backlog), and ignore the rest. That said, GraphQL is a great way to hedge your bets and make your future as well as current self do less work.

Does this appear to anyone else as the exact style, length and depth of a ChatGPT text?

Earlier I wrote about the "Simulator Effect" aka "apophenia", and "Reverse Over Engineering":

https://news.ycombinator.com/item?id=22062590

DonHopkins on Jan 16, 2020 | parent | context | favorite | on: Reverse engineering course

Will Wright defined the "Simulator Effect" as how game players imagine a simulation is vastly more detailed, deep, rich, and complex than it actually is: a magical misunderstanding that you shouldn’t talk them out of. He designs games to run on two computers at once: the electronic one on the player’s desk, running his shallow tame simulation, and the biological one in the player’s head, running their deep wild imagination.

"Reverse Over-Engineering" is a desirable outcome of the Simulator Effect: what game players (and game developers trying to clone the game) do when they use their imagination to extrapolate how a game works, and totally overestimate how much work and modeling the simulator is actually doing, because they filled in the gaps with their imagination and preconceptions and assumptions, instead of realizing how many simplifications and shortcuts and illusions it actually used.

https://www.masterclass.com/classes/will-wright-teaches-game...

>There's a name for what Wright calls "the simulator effect" in the video: apophenia. There's a good GDC video on YouTube where Tynan Sylvester (the creator of RimWorld) talks about using this effect in game design.

https://en.wikipedia.org/wiki/Apophenia

>Apophenia (/æpoʊˈfiːniə/) is the tendency to mistakenly perceive connections and meaning between unrelated things. The term (German: Apophänie) was coined by psychiatrist Klaus Conrad in his 1958 publication on the beginning stages of schizophrenia. He defined it as "unmotivated seeing of connections [accompanied by] a specific feeling of abnormal meaningfulness". He described the early stages of delusional thought as self-referential, over-interpretations of actual sensory perceptions, as opposed to hallucinations.

RimWorld: Contrarian, Ridiculous, and Impossible Game Design Methods

https://www.youtube.com/watch?v=VdqhHKjepiE

5 game design tips from Sims creator Will Wright

https://www.youtube.com/watch?v=scS3f_YSYO0

>Tip 5: On world building. As you know by now, Will's approach to creating games is all about building a coherent and compelling player experience. His games are comprised of layered systems that engage players creatively, and lead to personalized, some times unexpected outcomes. In these types of games, players will often assume that the underlying system is smarter than it actually is. This happens because there's a strong mental model in place, guiding the game design, and enhancing the player's ability to imagine a coherent context that explains all the myriad details and dynamics happening within that game experience.

>Now let's apply this to your project: What mental model are you building, and what story are you causing to unfold between your player's ears? And how does the feature set in your game or product support that story? Once you start approaching your product design that way, you'll be set up to get your customers to buy into the microworld that you're building, and start to imagine that it's richer and more detailed than it actually is.

Also:

Will Wright on Designing User Interfaces to Simulation Games (1996) (2023 Video Update)

https://donhopkins.medium.com/designing-user-interfaces-to-s...

I notice there are definitely very different styles of thinking (obviously) but it really becomes clear when I read articles like this.

I would say that nothing in this article is interesting or new. IMO, it is not helpful in literally, any single way. (That's not me trying to be rude, I believe this is more-so a function of how people perceive information, and there are certain types of information that click vs not... ie, the wrong kind of fuel).

If I had to pick the right balance on any of the projects or companies I start... it always begins with pacing around my house in a deep sort of trance... pacing around endlessly for hours, standing in the shower while I mumble over and over, trying to visualize the entire "problem space." Of course with complex problems you can't do that, so in a sense the visualization I have will undoubtedly "fade" and I must rebuild it again in my mind, again, and again, and again. Each time trying to iteratively hold more in my mind simultaneously, all the different interwoven layers, without it crumbling.

I also use a huge amount of notebooks to draw different diagrams of the problem, usually messy to start but then becoming more refined as I start to slice and dice the different dimensionality of the problem in different ways. How to hierarchize (is this a word?) the different facets, how to split the functionality up cleanly and elegantly.

For me, anyway, I always focus on just 1 word: Elegant, as I feel like that's a great "sweet spot" for me. I try to find elegant solutions to hard problems. I like this word a lot because there exists solutions which cleanly slice the dimensionality of information, either by jiggling the information in slightly different ways (which has a net positive effect on the end user), and the cost/benefit is such that it improves the business side as well. Always a sort of cost/benefit, cause/effect energy going on. Shifting these building blocks, (which also equates to the actual code/features) until there is a nice fit. A good example might be org structure for a company that has all kinds of reselling/affiliate/white labeling options, and how accounts and financials/books are set, where they live, at what level, etc.

They will undoubtedly become more complicated later, as more layers of features and such get piled on, but I just intuitively piece through those scenarios not by reading articles like this and being like "GEE I GUESS I BETTER INCREASE AGILITY!" but by just feeling out the energy of the problem in my mind, poking at it, following it around like a hard to catch elusive rabbit... slowly hiding behind bushes and trees trying to sneak up on it. This is the idea of chasing the elegant solution which almost always (for me) is very painful and hard to find, with a great amount of stress and energy (unable to sleep or eat until I find the solution). It will haunt every waking second of my mind until I have it, and when I do it hits me like a freight train and I exclaim (!FUCK!) and I quick write it down, and the process repeats, usually from 7-30 days depending on how hard the problem is, how hard the app is, how hard the idea is.

From this way, I've nailed the balance of over/under-engineering nearly perfectly in each of my later projects, a few of which have become absolutely immense

I very much agree with this. Over-engineering and under-engineering are poorly named as they are not on opposite ends of the spectrum. They are actually both badly-engineered and actually both lead to many of the same issues (which I believe this article gets wrong). Of the listed cons both overlap on all of the following:

- Fragile code

- Technical debt

- Reduced Agility

- Understanding Complexity

Over-engineering can be abused an excuse for poorly-engineered solutions and cutting corners. The future being hard to predict is often used as justification, but this swings both ways, you also often won't know what code is going to get built upon. Frequently an obscure one-off piece of code can become more useful than expected, with functionality tacked on over time, until the point where an entire product is resting on really shaky foundations.

Build a culture of quality engineering. Build the minimal solution but build it well. Have a strong (and flexible) product vision as a guiding light but always take small steps towards it. Optimize towards understandability and replaceability.

"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system." -John Gall (Systemantics)

While not the intended audience, Systemantics is one of the most educating books on software architecture in existence.

This is possibly the best description of my development process I’ve ever seen.

I’d add “staring into the distance” alongside the mumbles (yes, when pacing, that often means I run into corners of walls, couches, etc).

I think of it as iterative development in my head, and it’s often focused on finding the fastest path to proving that an architecture or implementation is wrong; when I find it hard to prove why something won’t work, then I begin to explore in depth the idea and have a pretty big running start on the code, which flows a lot more quickly when I’ve done this than when I just sit down and start typing.