Exploring the software that flies SpaceX rockets and starships(stackoverflow.blog) |
Exploring the software that flies SpaceX rockets and starships(stackoverflow.blog) |
> Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.”
Wow, sounds an awful lot like a typical event loop from a game engine.
Of course the main difference from a game engine would be the reliability requirements. The level of verification effort that goes into flight control software can't be comparable with the effort that goes into a game engine (assuming it's greater than zero :))
It's how digital PLC process control loop works. Programmable logic controller (PLC) https://en.wikipedia.org/wiki/Programmable_logic_controller
I coulda sworn I recall John Carmack making a similar comparison when he was working on Armadillo Aerospace, but implying that rockets were actually a bit simpler.
Apparently I misrecalled...
https://twitter.com/id_aa_carmack/status/557223985977765890?...
It's simple to reason about and analyse. You can make assertions about worst case execution time and memory useage and you know for a fact what the order of execution is. The concurrancy on large systems like this often comes in the form of a lot of small computers talking to each other over some network/bus rather than having many systems all running on the same hardware.
Building concurrant systems with predictable real-time characteristics is hard. When you have a bunch of things that really need to happen every Nth of a second in order to fly straight, a simple approach like this is definitely preferrable. In situations like this predictability tends to be more important than raw performance (assuming you can reliably hit minimums).
That doesn't mean people don't use multi-threading in domains like this, but it's one of those things you want to keep as simple as you can and avoid where possible.
Nice it see things kept this simple.
Not only does flight control software typically never use garbage collection, it is also preferable to never allocate memory during the runtime -- all memory ever needed is allocated at program startup, and memory is not acquired or released until landing or some other major end event. Because memory issues are often the most significant causes of crashes, which you don't want to translate into, well.. a different kind of crash.
The creator of C++, Bjarne, wrote a nice paper on flight control software if you are interested in more.
EDIT: One of the coolest projects I saw in recent time was CoPilot, which is a Haskell dialect that compiles down to C control loops which are statically guaranteed to run in constant time and memory. There is also arduino-copilot for if you want to play with ultra-hard real time software but can't afford an entire rocket.
But it is worth noting that these control loops can often have some sort of memory, so you normally need to test over multiple cycles - you usually have a test vector to load in, and watch for the response vector.
Trivial example would be if you had a PID controller implemented into your main loop. Your main loop would be storing the integral and "previous" error term.
You could literally replay entire missions.
The similarity with game event loops is IMO superficial. A game typically processes events as fast as it can, and the time between any two iterations of the loop is completely unpredictable. One frame could render in 20ms and the next in 25ms, which is completely fine. For a car or spaceship though, there are hard real-time requirements. Like the article describes, some Dragon tasks run every 100ms, others run every 20ms. A lot of effort has definitely gone into making sure the tasks can complete quickly enough, with some margin, and there are definitely systems that monitor these deadlines and treat even one timing miss as a major failure.
No it is not fine, if you want to have a real smooth gameplay. So also in game loops, you can adjust for that, by checking timediff.
But sure, the difference is that, if you miss a frame in a game loop - no real rockets crash, so there is probably (and hopefully) not as much effort put into that, like they do on SpaceX.
In dynamic systems, such as a rocket, robot, etc., reality is fuzzy. Your loop is mostly about "what is my best guess at my present state" and "what is my desired next state". You make changes to try and get from A to B but you may not exactly achieve B. You may have not even been in state A. The error propagates back into your next guess of state, and this repeats forever.
Sensors like GPS give you an absolute position but they're imprecise. Inertial navigation is extremely accurate in telling you the change from a sample to the next, but as your position is the second integral of your acceleration, any error compounds quickly (admittedly the quality of the INS available to people willing to spend 10s of millions of dollars on a vehicle FAR exceeds what you'd be able to put in a car). Some rockets have even used ground based radar measurements uplinked to them to further improve position estimates. They probably still do this, I just don't have any hard data on current launchers.
If you have multiple outputs you can also have a scheduling/prioritization system for the subtasks.
And yes, use interrupts or multiple timers to detect only changed parts without having to compare current input to previous input.
It's basically the same problem as updating a browser DOM in response to application state changes.
Control theory algorithms can be developed using tools such as Matlab, Matlab-Simulink, Modelica or Scilab-Scicos (Open-Source) from Inria. Besides C or C++, another language used in embedded systems like that is Ada, which is much safer and reliable than both C and C++.
Or Arduino's loop() method, where all the code just runs in a loop "forever."
https://github.com/arduino/ArduinoCore-avr/blob/master/cores...
There are some details that may change, for example, we use mostly C, specialized software and a bit of ADA, no C++ in embedded systems. But the input-output periodic loops, isolation and making sure things continue working in degraded condition principles are the same.
Nothing special about SpaceX here, for good reasons. In fact, some of it might be mandatory if you want to fly something that can cause serious damage if you it fails, be it a rocket or a plane.
It sounds like they created their own internal API configuration scripts to be highly dynamic and configurable. Similar to many gaming or operating systems config files. Sounds be a highly productive way to test and deploy software changes. Not only for people who are not C++ proficient but also to allow Engineers and Scientist to focus on their own design work and not have to worry about software development.
Flight software for rockets at SpaceX is structured around the concept of a control cycle. “You read all of your inputs: sensors that we read in through an ADC, packets from the network, data from an IMU, updates from a star tracker or guidance sensor, commands from the ground,” explains Gerding. “You do some processing of those to determine your state, like where you are in the world or the status of the life support system. That determines your outputs – you write those, wait until the next tick of the clock, and then do the whole thing over again.”
[0] - https://en.wikipedia.org/wiki/Inertial_measurement_unit
Does anyone have much insight into the longevity of the ISS from now? I can see it's approved for operation until 2024, so just 3 years, but could potentially continue to operate after that.
If the ISS does get decommissioned, how many years does that process take, and once it's gone, what purpose does Crew Dragon serve?
Not trying to be negative, hopefully by 2028 or even 2024 we will have concrete operations underway for continued space station development that could use Crew Dragon, but it does seem bold calling it a new era, when it's so precariously reliant on the ISS existing.
It could probably be extend even further but it would require incensing maintenance and expensive re-qualification.
NASA already has a plan however. They have a contract with Axiom Space to extend the ISS with new modules. After they have about 4 new modules they should eventually decouple after ISS and be a free floating station.
NASA already has a program in planning for a commercial space station. The same way they did Commercial Crew and the moon lander. They have already asked companies to come up with plans for a station. SierraNevada really wants that contract and SpaceX will probably bid something, probably others. These would be free floating privately run stations
So NASA basically hopes that sometime between 2024 and 2028 there will be two new stations and then its politically easier to drop ISS.
Crew Dragon can still do free flying missions (as they will do later this year) but SpaceX hopes to replace it with Starship.
why cant you use higher clocked speeds ? like even 500 mhz, etc ? is there something special about 10 and 50hz ?
On a hard real-time system like the Dragon, failing the deadline once is a critical error just like a game engine being unable to talk to the GPU.
You can learn way too much about it and even operate (a simulation of) one here: http://www.ibiblio.org/apollo/
You just gotta find a balance that works for your requirements and constraints. In my application (it was medical devices), we found that stateful loops fit nicely.
while(true) {
...[50hz tick tasks]
if(tick%5==0) {
...[10hz tick tasks]
}
wait_for_next_tick();
tick++;
}Some systems, like those based on coin cells, will completely shut down the chip, leaving only a single timer or input active, and when triggered will turn back on and initialise the entire CPU again to handle the input/timer. Some microcontrollers give you a small allocation of you can preserve under these conditions. That's how you get year+ runtimes on tiny batteries.
Edit: if you want to nerd out on this, I would start with Jack Ganssle's excellent guide: http://www.ganssle.com/reports/ultra-low-power-design.html
You technically can have a dedicated timer interrupt for every task but there are usually alot more tasks than HW timers, so instead they use a dedicated HW timer for time-keeping and use that as reference for all other tasks.
Also, there sometimes ARE interrupts meshed in with these main loop based systems (for whatever reason). They just aren't predominant. If your willing to through more hardware at things, you can typically always transform anything into a buffer.
This is a great example of a little knowledge being dangerous. There's a reason control systems are not implemented the same way as a browser DOM. There's a reason control systems are designed to avoid subtasks, conditional execution, interrupts (as much as possible) and complex data models. The reasons are pretty damn solid, IMO.
Conceptually, you can talk about the two domains as being somehow equivalent. That says nothing about the appropriate implementations.
If only one sensor input changes then intermediate computations that don't depend on it don't need to be updated.
And you still need a control loop to look for sensor changes? Its just a way of caching most of the computation?
If you mean distributed to mean "over multiple computers", then ya... this is a really specialized field. Your ability to synchronize and maintain hard real-timeness between computers is completely dependent on IO and hardware features, and topology. This makes it much harder to make a generalized framework.
We we ignore the distributed portion, then you're describing a real-time operating system - in which case something like https://www.freertos.org/ might fit the bill?
Perhaps the idea that message passing has to happen realtime is something obvious or elementary to military/defense/aerospace from technical requirements or experiences fighting fog of war etc., but to me it was dumbfounding new and I guess it might also be for the general public?
1: https://en.wikipedia.org/wiki/File:4D-RCS_reference_model_ar...
Erlang came out of Ericcson when they were building giant telephone switches.
Nowadays things like RabbitMQ and WhatsApp run on the Beam VM.
The beam handles both local and remote nodes, mostly transparently.
Then there are things like Nomad or k8s which try to tackle the problem more like *nix(linux/BSD/etc) does, but across multiple nodes. These are not meant for hard real-time systems like launching rockets obviously, not even for soft real-time.
https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
>This sparked an interesting memory for me. I was once working with a customer who was producing on-board software for a missile. In my analysis of the code, I pointed out that they had a number of problems with storage leaks. Imagine my surprise when the customers chief software engineer said "Of course it leaks". He went on to point out that they had calculated the amount of memory the application would leak in the total possible flight time for the missile and then doubled that number. They added this much additional memory to the hardware to "support" the leaks. Since the missile will explode when it hits its target or at the end of its flight, the ultimate in garbage collection is performed without programmer intervention.
Hands down best answer was the engineer from a defense company: "I build flying robots that arrive at their destination in a really bad mood."
Edit: yes I know it’s more complicated than that!
https://www.stroustrup.com/JSF-AV-rules.pdf
which is linked from the Applications page of Bjarne's website.
https://stroustrup.com/autonomics09.pdf
It's basically just a quick list of 10 useful rules to follow for safety-critical code.
(I've seen a different format in the past, that wasn't quite as fancy, but this is the only version I can find at the moment)
[1] There is a busy loop but it's only there to trigger tasks on a timer.
It sounds elegant, but I’m not convinced there’s a benefit to a model of abstractions of abstractions like this unless it’s done at the language level. In practice, I’ve only seen this produce a web of race conditions, performance issues, and system complexity. My experience is anecdotal.
I've seen it work well once, but others struggled to contribute to it because it was so complicated and it added little benefit to the usual busy loop approach. So I'm in the same boat as you.
This makes it hard to reason about the overall capacity of the system (in terms of numbers of sensors and actuators). You will always end up uncertain whether you've ever really evaluated the "full load" condition, even if you formally test it in ways that say you have.
