Herbgrind analyzes binaries to find inaccurate floating point expressions

Herbgrind analyzes binaries to find inaccurate floating point expressions(herbgrind.ucsd.edu)

70 points by bshanks 2 years ago | 13 comments

Wow, didn't expect this tool to be on Hacker News six years after publication! Herbgrind author here, ask me any questions you like! Also, if you're interested in this stuff, the Herbie project (which I also worked on) for numerical program synthesis is also really helpful for writing numerical code, and has had a lot of development over the years.

zokier 2 years ago | |

What do you think programming languages/libraries/tools could do to make floats less scary for the common people? I feel there is lot of superstition around floats, some of it more well deserved than others, but it leads people to approach fp with great distrust.

Of course herbie is already good step here, but it is still somewhat niche.

a1369209993 2 years ago | | |

> to make floats less scary for the common people?

The obvious thing would be making sure that it's easy to get a unambiguous, natural representation of what a float value actually is. (Similar to what C printf %a specifiers produce, rather than something like Ryu float-to-decimal, which is useful for what a float value means for human-readable purposes.)

   0X1.999999999999AP-4
  +0X1.999999999999AP-3
  =0X1.3333333333334P-2
  ≠0X1.3333333333333P-2

is ugly (and could be improved[0]), but far more understandable (and less superstition-inspiring) than:

  0.1 + 0.2 = 0.30000000000000004 ≠ 0.3

0: At the very least, please make it easy to get "0xA.BCDEFp+1", with capital "ABCDEF" and lowercase "xp", so the digits look like digits and the punctuation looks like punctuation.

aSanchezStern 2 years ago | | |

Huh, good question. I think having a herbie-style error visualization graph built into an IDE would go a long way. In practice I think dynamic sampling is pretty good at capturing error behavior, and immediate visual feedback on the floating point code you're writing would make it a lot easier to trust the working code you have, and understand the problems with the error-prone computations. Something like https://github.com/herbie-fp/odyssey, but analyzing the fp expressions in the code files you're editing.

phkahler 2 years ago | |

Is it feasible to run this on something large like Solvespace[1] (CAD) which is ~5MB executable? Or would we just get an insanely long list of issues?

[1] https://solvespace.com/index.pl

There are hundreds of numerical algorithms in there, and we have some bugs that might be related to this kind of implementation error.

aSanchezStern 2 years ago | | |

It should be possible. Herbgrind is designed under the philosophy that it's not bad numerics themselves that are significant, it's how they affect the outputs of your program. So the reports are organized around program outputs (either through print calls, or you can annotate values you care about with macros), and errors are reported in terms of how they are affecting your outputs. That should allow you to skip any numerical errors that are insignificant in the final output, and prioritize the ones that are affecting the outputs you care about most greatly.

But Herbgrind is a heavyweight dynamic binary analysis using the Valgrind framework, so the slowdown it imposes on the runtime might be prohibitive for debugging some programs which take a while to compute their results.

99112000 2 years ago | |

I've tried out HerbGrind but it often gets stuck and reports an error E_STONED.

aSanchezStern 2 years ago | | |

I believe that's a PEBCAK error

touisteur 2 years ago |

Thinking of all the people (some even actual friends) worked so long on numerical stability on scalar floating point operations, nearing a time when model-checking and proof tool are actually usable by the really-motivated developer ; thinking of them witnessing the arrival of GPUs and Tensor Cores, mixed-precision-everywhere and the tools now needing to scale so far it seems like starting from scratch...

1over137 2 years ago |

Is it just me, or does the website not say which OSes it supports?

sph 2 years ago | |

"Herbgrind currently primarily supports 64-bit Linux, but 32-bit might also work, and theres some work in progress to also support OSX."

https://herbgrind.ucsd.edu/getting-herbgrind.html