Ask HN: What is the most complex software that you have built/worked on? For example: a distributed system with multiple databases and a lot of data, some high frequency trading system with ultra low latency, etc. |
Ask HN: What is the most complex software that you have built/worked on? For example: a distributed system with multiple databases and a lot of data, some high frequency trading system with ultra low latency, etc. |
I took a programming course many years ago. Not because I needed to learn programming but to get job interviews. The instructor asked us to write some C code to perform some small task. I completed the coding in about 5 minutes and it was just two lines (my prior experience was 6502 assembly not C). Others in the class were writing screenfuls of code. I imagine that the same situation occurs on large projects where mediocre programmers write more code than required for the task. Projects get bloated and more complicated as a result.
Another anecdote. I wrote a small operating system for a piece of hardware known as "touch memory" made by a company called Dallas Semiconductor. Intriguing little device, I must say. This was 20 years ago. My boss met up with the owner of a company that was attempting to build what I had already built in just two months. His 4 programmers could not accomplish much of anything over the course of an entire year. I guess that means that I was 24 times more effective. The company went bankrupt.
The most complex project I worked on? I dunno. Maybe converting PDF into braille. Would you say that is complex? I wrote all of the code from scratch without using any third-party tools. That's another area where projects go wrong. They rely too much on closed-source middleware that their own programmers don't understand. It's a recipe for disaster.
It runs on Node.js and the entire code base is just one file (under 100k). No third-party includes.
Took me three years to get it to the point where it is now (I spent most of that time wondering what to do next).
Is that complex? Dunno. I'm trying to finish the documentation but, damn, there is so much to cover! It's a beast. That's really why I'm here. Gonna announce something soon.
I worked for a company where IT owned customer onboarding for historical reasons. Each customer had their own DB in the same server so that each site would be isolated. So onboarding was basically cloning a “gold DB” and templating some values.
The script that cloned the DB was over 50k lines of Batch files (Windows .bat). There were so many includes, GOTOs, random black holes, passwords in plain text, reliance on special IP whitelisting in the AD server… it had been grown organically by someone who didn’t know how to program and it was a major mess.
It took me three weeks to figure out what was happening, and another to make sure the details were correct.
I replaced the whole thing with <10 lines of an actual scripting language. Proved it worked. But the management at the time didn’t want to change the process…
My Facebook rant at the time got me invited to speak to an upcoming class at my Alma Mater about software quality.
The call is removed, but the function remains. Fields in a form stay, unused and unpopulated, but the validation is disabled and left behind. Things like this.
No version control meant it was hard to go back, and people are afraid to break things.
That was some shit.
Mostly it was copy/pasted data (strings) with slight modifications eating up a lot of lines. A few big chunks were in a code path that could not execute, as well.
It was just really bad code.
Every time I see something like this it makes me want to scream. This is not how you develop software, setting all implementations in stone. Ugh
(1) - Actually, what I'm calling actual security features, plus jails and zones and Linux compat layers and performance counters and debug systems and obscure compile options and features that haven't been used by a soul since 2004.
Only 5000 lines of code but the complexity is insane. The engine deal with a a subset of math equations in quantum field theory. But the deal is that I have to make it compile several optimized math code in several situations.
I would be nice to treat those math ops objectively but it will be too slow. Inmutate will also be slow. So inplace modification is everywhere in the code.
Also the problem is multidimensional. Mathematicaly, I am working with tensors, matrices, scalars. Different treatment for each of them in different compile strategy. Programmatically, different situations sometimes need me to modify only one line of code, sometimes need to modify a block of code. Sometimes a block need to be coherent with the module it sits in, sometimes the module depends on states of those blocks.
Everything is entangled and there is no obvious way to do it right. Applying clean code strategy like "do not repeat" or "keep operation on same level" or "do not make side effect" all suck. Compiling the largest part will then take 1 hr, but now I happy with my messy code base that can compile in 1 min. Because of this, debugging is so much easier and I can just verify arbitrary line of resulting code on the fly.
I have been rewritten the whole thing 6 times in 1 year to try which programming decision is better also extending functionalities. Still a mess.
1. A custom in-memory graph storage/database which is threadsafe and designed for fast multithreading purposes. It also comes with a custom query builder/language which can be transportet via json so viable for every language. It can either be used by directly importing it as dependency in your go project or build as server with a http api package i build and used via api.
2. An architecture/framework which enables you to create completly self supervising software that basicly only persist of you defining a set of "abilities" which you do in form of go plugins. A plugin defines the parameter structure it needs to get executed. The architecture uses the in (1) described graph backend to store all the data. If data gets mapped into the storage the architecture will effective check if the new data can supply any of the registered plugins, and if so execute it. Since the architecture runs multithread by default all jobs will be parallelised into worker/runner threads. Results from those will be automaticly checked by a scheduler to see if it can supply new jobs with the new knowledge (and optional added already existing knowledge). Also data returned from a plugin will automaticly be mapped into the existing storage. Since all the runner do scheduling based on new data you have no central supervising but rather thread-distributed self supervising.
This way you basicly just define certain abilities as plugins and than insert the starting data. The architecture/framework will take on from there.
This was a strongly simplified explaination for what it does and i coded alot of different stuff in the last 15-20 years tho this project by far was the most complex in terms of dynamic data mapping / scheduling / etc. So complex in terms of logic rather then in size.
Something like 80+% of the logic was in SQL stored procedures across different tenants. Nobody wanted to apply static analysis or test automation against it so everything came down to memorization of the code base and the resulting implementations to user interactions in the corresponding web based application.
My experience with commercial software versus open source software has repeatedly proven the complexity of business requirements is completely unrelated to the challenges of scaling and maintaining a software product. It always comes down an appreciation of automation, organizational skills, and writing. I often see people try to cheat this with tools and abstractions only to later drown in debt and blame everything else.
Hasn't made a cent yet, but it has been a LOT of fun to build from scratch and tinker with over the years.
The hardest but most exciting part was using the PostgreSQL and SQL to process, clean and enhance data and write extensions and bring it all together.
We opensourced alpha version yesterday, more documentation to come.
-video-based traffic counting
First was the Airfields Database. The year was 1989. I was just hired with a fresh BSCS degree, and working at what was then the Defense Communications Agency, in the basement of the Pentagon. My full TS/SCI clearance investigation was still ongoing, so I couldn't do any of that work. I only had an interim TS clearance. So that meant they had to find work for me to do in the several months I still had to wait for the clearance investigation to be finished. They decided that the Airfields Database was in the most need of help, and so that's where I went.
The Airfields database, I was told, had a record of every single air base, air port, air field, road, or strip of grass where an aircraft had been known to take off or land there, even if it was just for an emergency. It had location information, runway length, runway direction, runway width, weight limitations, apron parking space, and almost a hundred other data elements that it tracked for each location. The software ran on the classified WWMCCS military mainframe systems, and was written in COBOL 66. It was our job to bring it "up to date" with COBOL 77.
There were two main code paths through the system. One where the operator selected the different fields that were relevant and what the constraints were, and one where all fields were considered relevant and you had to provide the constraints. I had the bright idea that we should eliminate the longer code path and only have one set of routines we had to modify, as we could simulate that longer path by pretending that the user had selected all fields in the other code path.
Well, it worked. We chopped out about 40% of the code from the system. During the process, I also took the only subroutine that was actually classified (it did great circle distance calculations between any two points), and I externalized that so it could be kept in a separate file. That meant unless you were actually working on the great circle calculations, that code printout could be kept locked in the safe, while the now unclassified code that composed most of the actual Airfields Database could now be left out on the desk of the programmers without having to lock it up every night.
The two full-time developers working on the Airfields Database were very happy at this outcome. Only thing is, when run on the Support system where we did our development, the printouts it created came out perfect. But when run on the production system, where all the real reports were ran, it generated an extra page break between each page of output. And if you modified the code to run correctly on the production system, it didn't work right on the Support system. We never did figure out that problem, at least not before I got reassigned to a different branch a couple of years later.
The second big system that I'm happy to say I worked on was the OpenStack cluster that AT&T used. I was part of a small team, and the work we were doing was inside of Chef for deploying their code to OpenStack, but there was a lot of duplication between the various modules, and lots of duplicated modules. We ended up cutting their code base to one third the original size, and with our all of our self tests we built in for the CI/CD system, we made sure that all the code still worked and ran exactly as the original code did. The weird thing was that we never got any feedback from them as to whether or not they liked any of our changes, or if we had actually broken anything without realizing it. That was a total one-way street.
The third big system I'm happy to say I worked on was the GPS OCX Next Generation Ground Control System for Raytheon. This had to control all of the GPS satellites currently in orbit at the time, plus all of the next-generation satellites that Lockheed was building. Again, this was Chef code, and we worked closely with the Raytheon developers to teach them everything we could about how to write Chef code, so that they could take what we taught them and apply that to all of their other code going forward.
After six months, we did a test that simulated bringing up the entire datacenter (we were just spinning up VMs instead of running on real hardware), starting from everything being racked and stacked, and the all of our code took all of their core infrastructure code (also including all the OSes and databases, etc...) and deployed it and configured it for operation. They started off with that process taking three months to do manually before we got there, and our first full test run was successful and completed in three hours. We were ecstatic.
Today, I work at AWS. Even just the tiny subsystem that I work on is bigger than anything else I've ever worked on before. It's a really challenging job, but also very rewarding. I honestly feel like I'm working with the best team of people I've ever worked with before.
What's that you say? How hard could it be? I can spin up a CRUD app where you can create items and add them to menus in an afternoon, and yes, you probably can do that but...
Are you going to make it so you can share options between items (Drink Size, pizza toppings, coffee flavor shots, etc)? Or are you going to manage those all independently on each item? That would be a pain when you go to add or remove 1 flavor from your 6 drinks that can have flavor shots added. Also think about reporting and/or stock tracking features. So instead you have "options" be another top-level entity that you can attach to items, problem solved!
But wait! Is "pepperoni" really $0.25 on all sizes? Or is it $0.25 on the small, $0.50 on the medium, etc? Ok so we need to have some way to adjust the price of the choices you select (pepperoni, sausage, etc) based on another option's selected choice (small/medium/large). Oh, and because you use different amounts of the toppings you are going to want to adjust the stock/recipe as well based on the selected choice. Let's also imagine the data structure for "gluten free is only available on the medium" and "stuffed crust is only available on the large", so we also need a way to disable other option's choices based on a previously selected choice. How about quantities on choices? 2x pepperoni? 2x espresso shots? What about 1/2 caramel shot? Are some options exclusive? Are some required? Do some have a max choice selection count? Do some have a minimum? Do some choices cost extra to add? Do you get 2 free selections then you pay after that?
Ok, ok, a little complex but we can handle this. What about discounts/coupons/etc? You'll need a way to apply a discount to an item or and order, ability to limit how many discounts can be used per item/order (different restaurants have different needs), ability to discount a flat amount or a %, limit which items a discount can be applied to ("$2 of any pizza"), limit which option-choices a discount can be applied to ("$2 off any medium pizza").
Ok, remember that menu we created? Do we really only have 1 menu? Or do we have a POS menu, an online menu, and maybe a doordash/Uber eats menu? Oh, and have fun converting your internal menu structure/concepts to the various food app's menu specifications. Do you want each location of your restaurant to have its own independent menu or do you want 1 top-level menu? Can you hide/remove items from your global menu on a per-store basis? Can you add items on a per-store basis (local specialty)? Do all your stores need to have a solid internet connection or function? (Spoiler, this is not a guarantee). Are you going to have local hardware/software so the store can function "offline"? Are you syncing menus down to these stores? Can your local hardware run the full stack without the cloud?
Are you accommodating for different types of restaurants? Drive thru? Delivery? In-store? Curbside? Table service? How about "little" things like "we want 2x Large coffee" or "we don't want quantities on items, they should all be separate"? Oh and those discounts? Might you want to also limit them based on the service type? This discount is for in-store only, this one is delivery only?
Do you have a loyalty program? Is it "points" or "stamps" based? Are there signup rewards? Are there visits rewards? How is your drip (or otherwise) marketing work? Are you making sure to only target people who aren't coming in every day/week/month already?
This and 1000 more things I never considered when thinking about POS software.
You missed the part about what the end result is supposed to be. What is the purpose? Can I use it to run a website?
A webcrawler. What does a webcrawler do? It expects a domain (data) and crawls for more data - analyzes it and enriches your collected data. You may end up with writing multiple plugins like.
- resolveIpFromDomain (takes Domain returns Domain->IP)
- detectWebserver (takes ip uses for example nmap to scan ports+banner) returns ip->port->software->(banner,state)
- detectVhost (takes ip->port->(software[webserver],state[open]) || domain->ip->(software[webserver],state[open]) and returns ip->port->(software[webserver])->[]vhost[]->page[/] ) -> loadPage (takes page loads it with curl and return page->content)
- extractLinks(takes page->content return page->content->[]link)
- loadLink (takes vhost->page->link returns vhost->[]page )
- extractMedia (takes page->content return page->content->[]media)
- analyzeMedia (takes page->content return page->content->media->[]attribute)
..... So what you do is you provide a domain, which will trigger resolveIpFromDomain. This will map the data back to the datahive and based on the Ip in new data trigger detectWebserver. This will return found webservers which triggers the requirement of detectVhost. At this point you probably see how its going.
Due to how the architecture works it will always maximum parallelize the work, it will always map the data into one big structure without you having to care about it, it will only execute things that are necesary/usefull.
So the more your software should branch/parallelize the more gain you get.
Tho as i mentioned in my original post ill release the first alpha so there is still things that can be extended and improved. And right now im spending time in writing the docs which will probably take me some more weeks in orders to make them good enaugh for people to understand how to use it by themself.
I mostly will release it because i think its a great showcase of how you can do optimized data driven processing while havin an architecture that cares about the most painfull things like data mapping / parallelization / etc. I dont expect it to be the next "big thing" or even beeing used by alot of people, but if it inspires people or someone maybe write a even better version based on the idea i would be happy already .)
So to come back to your original question - can it host a website? Probably - but not really meant to do it and a nginx would serve u better.
Circa 1984 (ish) I was in Australia and we didn't go looking for such wizardry .. it was easy enough to derive Great "Circle" paths about an oblate spheroid from first principals and, if required, weight the problem by wind strength, etc.
I was coding, at the time, for airbourne geophysical surveys at continent scale, so we were also pulling in IGRF models, normalising for diurnal magnetic flux, calibrating radiometric systems for air colum mass under craft | cosmic breakdown gammas from above, and all that jazz.
Was there anymore to your classified great circle code snippet than the correct geometric answer for a "not an actual sphere" .. or was the secret sauce just the exact series terms chosen and the order of computation in order to minimise error and reproduce consistent answers?
( I'm not asking for the revelation of the actual classified details, just curious about shape of the classified hole from the outside )
Besides, even a well-designed library can be fatal for security if it is not handled correctly.