Not so advanced as very correctly pointed out by https://news.ycombinator.com/item?id=27554949
Motor-attitude control loop cannot, and should not be "fused" with navigation layer, whether INS, or optical flow sensor.
The bog standard deterministic controls algorithm here is "bog standard superior." Computer vision/optical flow sensors have much lower precision than gyroscopes, and accelerometers, let alone aerospace grade ones.
Not my monkey; not my circus. But… is critique of terms “gatekeeping”? I understand this term in the context of ownership (ie. gatekeepers for books are publishers/printers who own the means of production.)
I greatly appreciate some level of debate of terms here on HN.
First time I got my hands on a working mounted arm, I was cautioned again and again the need to run any new program in low-speed mode first. Because the arm had no limiting logic and would cheerfully power-bomb its own base with all the force and torque its motors could muster if I told it to.
Preventing that is still considered an "advanced technique."
Dropped frames are one of the easiest things to handle. Yes, the visual feature tracking depends on frames of video but even cheap phone IMUs these days are good enough to dead-reckon for a second or two, especially when embedded into a sensor fusion framework, so the prediction errors resulting from a single lost frame should be very minimal and not enough to throw off the tracking.
That's why I find it hard to believe that the VIN in use by the Mars Helicopter (part of a multi billion dollar program) wouldn't be able to deal with a dropped frame. It just doesn't add up. I suspect that the situation is much more complex than what the article suggests and that more things went wrong than just a dropped camera frame.
> This paper provides an overview of the Mars Helicopter navigation system, architecture, sensors, vision processing and state estimation algorithms.
The first failure, which delayed the initial spin test, was described as a "watchdog timeout", which for anyone not familiar with embedded development basically means the code crashed. We all write code that crashes, but I am having trouble thinking of an excuse to justify the fact that their code crashed before takeoff, on Mars, and they didn't see it coming. There is nothing about sitting on the ground on Mars that shouldn't have been tested repeatedly on earth, and testing in production is _really_ not the right way to do Aerospace development (although Boeing Starliner would beg to differ)
Similarly, there are a huge number of things that can and will result in dropped frames when running Linux on a Qualcomm mobile chip, and having a software stack that infers frame timing purely from the sequence number is brittle, and would definitely not have passed code review and testing where I work (I actually checked, we do have a robust solution). If I had to guess, I suspect the root cause of the dropped frame wasn't actually anything exciting like a cosmic ray, but instead was some run-of-the-mill event that would have been caught by a couple hours of flight testing on Earth. Either way, it shouldn't have made it to Mars.
I'm sure that there are a lot of great engineers working on the Ingenuity project that _don't_ write these sorts of bugs, and am glad that theae amateur fuckups (barely) haven't crashed the drone before it has been able to do some incredible technology demonstration work.
Presumably they use some kind of Kalman Filter, but those are easy to program to account for missing frames, or frames at non-discrete timepoints, perhaps even for screwy camera images if the programmer had a reasonable prior for the likelihood of it happening. Kalman Filters by design account for measurement error.
I didn’t read into it too much so I may not have all the details right, but I think this is the gist of it.
That would make me curious how the timestamp error occurred: software, hardware? Camera or Navigation code? I assume they have very high standards, what was the process failure point?
Usually with a Kalman filter, you’re taking into account the spatial measurement error (gyro-measured roll rate error, accelerometer-measured acceleration error, etc) but I don’t think I’ve ever encountered a system that explicitly modelled sensor latency variation relative to timestamps. Based on the description of the problem they encountered here, I suspect what happened is that it lost a frame but didn’t adjust the “photo timestamps” appropriately; every frame that came along afterwards would have had an incorrect timestamp? Even if the Kalman filter was set up to handle “this photo was taken 20ms ago” when doing its forward integration, if they didn’t model “this photo was taken 50ms ago but is reporting that it was taken 20ms ago” then you’d pretty readily get the kinds of oscillation they were getting.
Edit: yeah, just like the sibling comment said :)
This bathtub style curve for perception of NASA design by HN commenters makes me question if the perception correlates with reality.
I understand the IMU is not an ideal input and integration over time leads to positional errors. But gyros are much better and the orientation of the drone in flight is paramount. What I wonder is what ‚advanced‘ control law allowed the drone to become unstable wrt. orientation when there was noisy positional input.
It's quite hard to detect when position and velocities are "obviously incorrect", especially when they come from VIO, where the optimization result can jump around in non ideal conditions, so I'm not surprised there was not a more graceful anomaly detection.
When it's on land, they can make the gyro reliably point 'down'. Then at least during flight they know which way 'down' is.
Would this be too fragile for Mars?
MEMS Gyros measure angular rate, not absolute angle; to compute an actual angle, you’re taking the integral of the rate from t=0 to now. Any small errors in the measurements add up quickly to give you completely nonsensical results. For drones-on-Earth, we use a variation of the Kalman filter to combine short-term and long-term measurements. As an example, an accelerometer requires a double-integral to turn into position, so errors accumulate very quickly, but we can correct those errors using GPS. The accelerometer and its integrals give us really quick acceleration, velocity, and position updates (at, say 500hz), and then the GPS is used to correct the long-term position and velocity (at, say 5hz).
I am sure the one on the Mars helicopter had more precision though :)
There are certainly people involved in the project who could have explained this to them. I hope they are learning fast.
This is how nearly all modern high-end quadcopter drones fly - navigation using optical flow camera sensors short circuited directly into attitude/motor control loops.
I guess there is not a small chance they entrusted helicopter autonomous operations programming to people with quadcopter background.
I would not be at all surprised to learn that commercial quads share the design mistake. I was surprised to learn that NASA professionals copied it into a Mars probe. But, notably, not into the vehicle that delivered the lander.
It turns out sideways drift accumulates very quickly - and so quickly that unless you are a very practiced drone operator its very hard to compensate for by hand.
GPS compensates somewhat, but obviously that isn't available on mars (or indoors).
iPhone was not a first.
"Agent V" (2006) — a mobile AR game preinstalled on Nokia 3230 already has such technology.[0]
2) Was this something you even really needed to "Well, ackshully..." anyway?
My interpretion is that the copter, as an inessential system component, was seen as an opportunity for junior people to get some end-to-end experience. I hope they are learning the right things.
So maybe the issue is not as simple as a news article portrays it
or are you relying on a third party article's interpretation and then extrapolating that even further?
In my opinion assuming they were "testing in production" (production being mars!), or writing code that "definitely not have passed code review", or that they did not do couple of hours of flight testing on earth, is an unnecessarily unkind assessment of this project.
[1] Helicopter Models and Test Facilities: https://rotorcraft.arc.nasa.gov/Publications/files/Balaram_A...
I am very skeptical of the claim that this code was not tested before launch.
As an amateur RC pilot familiar with some of the excellent RC flight control systems, it would have been a huge missed opportunity if JPL didn't invite some experienced engineers from the commercial and consumer drone community to provide input (QA folks too!). It's hard to imagine they wouldn't have gotten ample volunteers to spend a few days helping out.
I understand JPL is already designing a larger and more capable iteration. It would be cool if experienced drone flight control devs such as yourself dropped the team a note.
The GPS constellation has 32 satellites. A GPS satellite weighs ~2000kg. Starship should get 100-150 tons to Mars - the math checks out! (Disclosure: I got all my rocket science from sending Kerbals to their doom.)
So, MPS for Mars, LPS for Luna? Just imagine the amount and quality of rover footage we could get if each of them had (constant!) 1Gbps uplink to Earth...
atomic clocks
That would be really cool, I wonder if it would be faster and more precise because there would be negligible human-produced radio noise to interfere and the Martian ionosphere is much thinner, or if it would be worse because the thinner and weaker atmosphere and magnetic field don't protect the receivers from solar noise.
I wonder if a rover could drop a few beacons for time-of-flight 2D triangulation, it's not like they're moving hundreds of miles away over the horizon.
I imagine it’s possible to set up a ground-based network, but you would need a high density to cover large surfaces (you want to see at least four stations from every position). I also imagine that it would be difficult to get accurate vertical positions if the stations are all in the same horizontal plane.
[1]: https://en.wikipedia.org/wiki/Real-time_kinematic_positionin...
I'm not argue. I'm just saying.
On the other hand, scientists are often not as good at design as engineers. They are not routinely trained in it, and are expected to deduce it from first principles. (If you have seen code written by scientists, you will know what I mean.) When they err, it is typically by assuming that theory and practice are the same.
The exact nature of the design flaw is subject to further investigation.
Of course they use rad-proofed chips in the rover. But using a newer Qualcomm processor instead of an ancient chip was a big part of the idea.
Before GPS LORAN [1] and it's versions was used relying on ground stations
Are we reading the same page? I see datasheets linked in the navbar, an article about an EtherCan box, and white papers about encoders. If any site is then certainly this one has a very specialist audience in mind.
Besides i believe what constitutes "advanced" gets wider rather than narrower when you move towards a less specialised audience. To the generic population even two stacked PID loops would qualify as "advanced" while practitioners would call that "conservative" or "old-fashioned".
Not an expert, but the whole network is on communication (with perfect understanding of the satellites locations no less), shouldn't they be able to run an ntpd like protocol to stay in sync, and just sync ground clocks to "starling consensus time" instead of "real time".
I'm also not an expert so I don't know if you can solve it with syncing. Maybe you can. But that doesn't sound like a "just" concept to me - any solution to this is going to be complicated.
There are smart people on HN, but unless they have actual professional experience designing similar vehicles, I'm inclined to give the JPL folks the benefit of the doubt. It's also worth considering that they are the only team that has ever flown such a vehicle on another planet, which imposes constraints that even other experts may not be familiar with; that they were working with a limited budget, and making heavy use of existing open-source code; and that, generally speaking, we weren't in the room.
On the other hand, maybe they really are incompetents whose avionics system is "designed wrong", and who do need these things explained to them so that they can learn faster, as the commenter you linked so delightfully put it. In that view, I guess their groundbreaking achievement is probably more dumb luck than anything else--maybe JPL should do some firing and hiring so that things don't turn out worse next time!
^[1] https://spectrum.ieee.org/automaton/aerospace/robotic-explor...
The commentator is correct. And saying something is done wrong is not the same as saying someone is incompetent.
While I understand you're trying to point out hubris, you're swaying the other way and appealing to authority.
I'm appealing to the fact that they sent the thing to Mars and flew it. If you think "any hobbyist" could have trivially done a better job, which seems to be essentially what you're saying, there isn't much for me to say to you other than that I strongly disagree. And if I'm supposed to be the one who's appealing to authority here, I'm not sure what everybody else thinks they're doing.
I expect JPL tends toward having generalists working on projects like this, and I expect they do have things to learn. Nothing in my comment contradicts any of that. But the level of armchair quarterbacking in here is crazy and IMO largely unjustified.
Sure, our ancestors (who could make their way and thrive in the wild) would probably consider me (who gets lost the moment I make a wrong right turn and depends on the local supermarket) a complete idiot, especially since I couldn't recreate any of the technology my daily life so depends on... But I've got central heating and clean tap-water, so I'm happy with the exchange.
The higher we climb the ladder of technological dependency, the more calamitous it becomes should we ever fall off. Best not to look down. (I think I'm supposed to bang rocks together to make fire, right?)
Won't work very well unless you have flint specifically :-) Even if you know the techniques, starting fire without any manufactured tools or manufactured tools to make the devices is really really hard.
No kingdoms offered huge prizes for the first electric light, or the first antibiotic, the first radio, or the first computer. But geolocation has always been considered a big deal, and it arguably wasn't well and truly solved until the advent of satellite navigation. It'd be surprising if it weren't being planned for the next generation of Mars orbiters.
I wonder what the “useful minimum” for satellites would be if you wanted, say, relatively accurate positioning once every orbit or each day.
Almost all of the technologies that we would need for survival and productivity on Mars are recently-invented.
IIRC, it seemed more like a missing test case or scenario in the software validation suite might have been the ultimate root cause.
[1] https://en.m.wikipedia.org/wiki/Triple_modular_redundancy#Ch...
That is what's being discussed here, this is the failure mode in question. So yes, those experienced in optical flow and embedded control systems _would_ have handled this specific failure better.
But none of us are saying any hobbyist could have trivially sent it to Mars.
It should be obvious what we are specifically addressing: that this kind of error is not the sort that should happen in a properly thought out, bog standard real time system.
> JPL tends toward having generalists
And that's fine too, plus dumb mistakes always happen.
So it shouldn't be much surprise when the "armchair quarterback" responses show up, and rightly point out the issues. It would be more crazy not to be incredulous.
It isn't, actually. The comment I responded to, and the one it linked to, are broadly criticizing the sensor fusion/control loop implementation--the thing that saved Ingenuity after the optical flow subsystem started returning bad data. That's what's being discussed here, in the context of this subthread where I entered it.
The dropped frame thing is much more cut and dried, and it's obviously a bonehead mistake. It's also the kind of thing that's always going to happen, as you say. Many of the commenters here seem to be generally at odds with that reality--I don't believe the state of the art in software development has yet achieved guaranteed bug-free implementation of diverse real world applications, but you wouldn't think that from reading some of these comments (e.g. the one you linked above).
You're right that it's possible, but how long it takes, and whether or not you can do it, are dependent on skill and environment. Wrong kind of wood? Screwed. Been raining for a day? Screwed. Sometimes this can be overcome, but you have to know what you're doing, and it can still take hours to generate enough heat to dry the wood until it's possible to friction combust it. Try it sometime, just bring a lighter with you.
But friction methods are tough, even if you have a well-made spindle and fireboard--which of course require tools. You're not easily creating those without tools to make those tools. And, even then, it's not super easy especially in a non-optimum environment.