Go Fuzz Testing

114 points by andrei 4 years ago | 22 comments

That first example is deceptive. In this line:

    for i := 0; i <= n; i++ {

The "<=" character just looks like a "<" character due to the typeface that the author chose. I'm going to assume that wasn't an accident because it perfectly masks the obvious bug in the for loop.

When I did a quick scan of that code it looked good to me. Then he claimed it should panic and I felt crazy. So I copy/pasted the code into an editor to see wtf was going on and sure enough it was "<=".

Andoryuuta 4 years ago | |

This is my main issue with fonts made for programming that use special ligature marks. They _look_ cool, but they are very hard to tell apart from other characters.

It's like someone decided that the I, l, 1 confusion wasn't enough and decided to add more. :P

lalaithion 4 years ago |

    go test fuzz v1
    string("000000000000000000000000000000Ö00000000000000000000000000000")
    rune('\u0083')
    int(60)

Interesting that the minimization engine wasn't able to shrink this further.

evmunro 4 years ago | |

I noticed that as well - most fuzzers will have a maximum duration or number of iterations they're allowed to attempt when minimizing so as not to starve out actual inputs. It could be that the fuzzer hit that limit, or potentially prioritizes readable inputs over small inputs.

lalaithion 4 years ago | | |

For string inputs, some form of binary search ("Check if the bug exists in the first half or second half of the string") would be able to reduce this example to "Ö" in only a few iterations. Not sure if this just isn't implemented, or whether there's something more complex going on.

There's also the fact that I'd expect a fuzzer that knows about Unicode and UTF-8 strings to have a known list of weird behavior hardcoded as seed values, and certainly two-byte runes would be on that list.

Of course, this is only the first release with the fuzzer, and it already looks really amazing - all I'm really saying here is that I can't wait for these to be features of the fuzzer in the future!

jiehong 4 years ago |

It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing.

I guess we still need to rely on something like gopter.

masklinn 4 years ago | |

> It’s a bit surprising to have fuzzing as part of the standard tool chain, but not property based testing.

Property testing doesn’t really benefit from (let alone need) toolchain integration, so while having better support for proptesting would be nice and give them more visibility, it’s far from critical.

akshayshah 4 years ago | |

testing/quick is in the standard lib, though it's a very far cry from Hypothesis or QuickCheck.

kardianos 4 years ago | |

Use table driven tests.

func TestFoo(t *testing.T) { list := []struct {

     want string
     input string
     expectedErr error
  }{
  {item1},
  {item2},
  }

  runtest

troutwine 4 years ago | | |

That’s not property testing. Yes there is a property being checked but you the user are selecting the inputs by hand which lumps in whatever biases you might bring to the test. Property testing is a property to validate, randomized search over the input space and, ideally, shrinking of inputs that fail the property. The benefit is you get to search over more of the input space that a human would normally do with an explicit table of inputs and with hopefully less bias toward inputs that don’t fail the property.

spencerwgreene 4 years ago |

I'm pretty sure fmt.Println("HELLO", str, len(str), len(result), n) isn't supposed to be in the final solution? :)

andrei 4 years ago | |

Good catch :) fixed!

rendall 4 years ago |

Neat, Andrei. Good luck with this venture!

Thaxll 4 years ago |

Curious if fuzzing is a common thing in network programming, does fuzzing plays nicely with binary inputs?

andrei 4 years ago | |

It does! Fuzzing actually started off as a tool built by security researchers to find vulnerabilities in parsers, and other complex codebases, usually written in C/C++ (looking for memory bugs). So anything that deals with untrusted binary data is a prime candidate for fuzz testing.

Go’s fuzzing framework supports `[]byte` arguments as well as all of the standard Go primitives, so you should be able to test netcode this way.

If you're looking for a C/C++ solution, my recommendation is libfuzzer [0]. We've also built our own C/C++ fuzzing engine at Fuzzbuzz [1].

[0] https://llvm.org/docs/LibFuzzer.html

[1] https://docs.fuzzbuzz.io/docs/getting-started-in-c-or-c++

masklinn 4 years ago | |

Along side file parsers it’s a pretty major fuzzing target, as it tends to be exposed to malicious inputs.

Fuzzing works primarily on binary data, “structured” fuzzing is somewhat rarer.

monkpit 4 years ago |

What’s the rationale behind bundling a fuzzing library in the language instead of as a separate library?

masklinn 4 years ago | |

To do guided fuzzing (graybox, like afl or libfuzzer) you need to instrument the binary. To instrument the binary, you need to be part of the building process.

Since Go has a bespoke compilation toolchains and AFAIK doesn’t have compiler plugins, external fuzzing tools had to either fork the toolchain or perform extensive pre and post processing (couldn’t tell you what go-fuzz did but many article about go-fuzz note that the building process can take a while).

As such, building fuzzing into the standard toolchain and maintaining it as part of the project makes a lot of sense. It also gives fuzzing a much higher level of visibility (because sadly there will always be a population for whom an external / third-party tool will be suspicious).

LukeShu 4 years ago | |

It does instrumented fuzzing. The older https://github.com/dvyukov/go-fuzz would rewrite your sources to inject the instrumentation and pass the rewritten sources to the compiler, but it didn't really work with Go modules. This is something that probably makes sense to integrate with the compiler toolchain, same as `go test`'s coverage testing.

spicybright 4 years ago | |

Probably to encourage a standard fuzzing library for the language. Like the same way go fmt is part of the language.