Making a low level Linux debugger, part 3: our first program

Making a low level Linux debugger, part 3: our first program(blog.asrpo.com)

171 points by asrp 7 years ago | 27 comments

pmarreck 7 years ago |

I now feel like debuggers are a design smell. I've used them less and less (to the point of not having to use them at all) as I've coded more test-first and designed things to be smaller (lower logic depth, etc.) I mean, all a debugger is, really, is a window into dynamic state, so if you control the, let's just call it what it is, the "rampant dispersion of state" at all times, you end up not needing a debugger (and producing fewer bugs) IMHO.

Now I'm sure that kernel-level stuff wasn't designed like this, and probably won't be for some time (I'm hoping Rust changes that story, but not a low-level coder), so debuggers may still be necessary- just making a point that it seems possible to design in such a way that the need for one is drastically reduced, if not eliminated. Or at least, that's how it seems to me in high-level-language-land (things may differ at the assembly and C levels).

EDIT: I apologize for OT, probably not the ideal place to start a "are debuggers really necessary if you design code correctly?" discussion, after all this is part 3 of a series, my bad

_wmd 7 years ago | |

It seems you've mastered some dark art unknown to an entire industry, you should totally write a book

Meanwhile I'll rely on the smelly old debugger to dig me out of threading weirdness, memory corruption, resource leakages, other people's bugs, and to provide a window into a combinatorially massive input-dependent state my feeble mind simply can't fathom while writing tests for a tiny corner of a typical system that in aggregate amounts to millions of LOC.

jjoonathan 7 years ago | | |

Nah, gradparent is not a wizard, just someone who has spent a lot of time in a single relatively simple environment that is mostly under their control. It's a beautiful thing when it happens. Most aren't so lucky.

pmarreck 7 years ago | | |

> threading weirdness, memory corruption, resource leakages, other people's bugs

Just as an FYI, most of these are theoretically eliminated or at least drastically mitigated by a switch from mutable state to immutable, purely-functional data structures. The cost of course being somewhat increased memory consumption and a bit more slowness (or a lot, depending on the algorithm).

If you ever get a chance to work with the BEAM VM, it's pretty tight.

strong_silent_t 7 years ago | |

I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly.

For example, if I'm working on a Python script under 1000 lines I would never use a debugger. I might use wireshark or linux system tools if something was really wrong, but usually the trackback or program output is sufficient.

I've also tried to debug an Arduino project where I had a lot less knowledge of how the system works, and if I wasn't able to attach a debugger I don't want to know how long it would have taken me to find the problem (calling a null function pointer causes an interrupt that didn't have any handler).

I've also tried to debug other people's projects in high level languages, without a debugger where after taking about an hour to figure it out based on pure reason, realizing it would have been very quick and easy to find with a debugger.

I think if you look at what a program is doing, and at that point you can't form a testable hypothesis as to what is happening, at that point a debugger will certainly be faster if you know how to use it.

pmarreck 7 years ago | | |

> I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly.

Well, I mean... Ever wonder why that is? Your understanding of the various states the program can get into is fresh in your mind. Come back to it in a year after not looking at it all that time and suddenly you're relying hardcore on your unit test suite to contain that same knowledge in "automated proof" form... OR you start spending a lot of time in the debugger to "re-understand" the system, slowly and manually... Or both.

If you need to use a debugger, it's because you are in some unexpected state. With full understanding comes full control and in those cases, no, you need neither a debugger NOR a test suite, but since a test suite on well-written code already identifies many possible state failures, I've found that leaning on that instead of a debugger works out better in the end.

brokenmachine 7 years ago | | |

How did you use a debugger on Arduino? I've been using Serial.print("WTF") more than I'd admit.

yitchelle 7 years ago | |

I have been an embedded developer working quite close to the metal. Over the years, I have noticed that the further you get away from the metal, the less reliant you are on the debugger. In projects past, I have worked with MathLab developers who are deploying their code on a multicore CPU and they have no need to have a debugger (in the traditional sense) in their tool set.

thedirt0115 7 years ago | | |

This is exactly what I was thinking, but couldn't phrase it quite as nicely. I never feel the need to reach for pdb when writing Python, but C on microcontroller? I need my gdb.

pmarreck 7 years ago | | |

Why do you think debuggers are so necessary at the metal when they aren't further above the metal?

asrp 7 years ago | |

I don't have that much to add to the other replies here. I used to never use pdb in Python for many years (not knowing or forgotten it exists) and doing perfectly well. Then I started using post-mortem debugging and finding/fixing bugs got so much faster. Basically, I'm using the computer to do the simulations I previously ran in my head (by evaluating things here and there). But that's just one point of data.

Another debugger I now recommend people (in general) to look at is Squeak Smalltalk's debugger which allows a form of "top down" programming where undefined functions are called and filled in at runtime. The debugger/editor in this post borrows a little bit from that idea in spirit. Part 3 in particular shows how it can be used even when they are no bugs.

About being off-topic, well, no-one has made another top level comment and by the time of your first post it was already 5 hours in. Some discussion is definitely better than none. So thank you for inciting that. If it burried some more on-topic discussion, that would be unfortunate though.

pmarreck 7 years ago | | |

Ha, thanks for that!

remoteorbust 7 years ago | |

I find that I don't need a debugger until I do. People are always asking me what debugger I use for go and I tell them I've gone months without feeling the need for them.

However every once in a while I need to use gdb + remote debugging features to figure out a particularly weird runtime issue and I'm glad it's there.

Then I can often take the odd values I see in the debugger and start putting them into a test or a minimal repro.

pmarreck 7 years ago | | |

This is the ideal use case IMHO

busterarm 7 years ago | |

Having access to a debugger when you need it is a necessity. Rampant use of it can, debatably, be seen as a crutch...but if you need it and you don't have it, you're going to have an extremely bad week.

Procedure for running w/ debugger attached. 1. disconnect debugger's usb 2. connect device via usb 3. arduino IDE program device 4. identify elf, set up gdb with elf. 5. plug in debugger 6. run jlink gdb server 7. attach gdb to jlink server "target remote:2331" 8. "monitor reset", "continue" to re-run target from beginning. 9. observe device is recognized by Arduino IDE 10. open Arduino IDE serial monitor to interact with device. Now, Ctrl+C in GDB halts device, and stack etc. can be inspected, and then it can continue running with "cont". To rerun the device with the same program without redoing everything, go to step 8 and reset the device. and continue from there. To find .elf on linux: ================================================== find /tmp -name sketch_name.ino.elf 2>/dev/null gdb location: ================================================== /home/user/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gdb JLink setup command: ================================================== JLinkGDBServer -device ATSAMD21G18A -if SWD -speed 4000 gdb attachment: ================================================== (gdb) target remote:2331