Show HN: OpenAPI fuzzer – fuzzing APIs based on OpenAPI specification

Show HN: OpenAPI fuzzer – fuzzing APIs based on OpenAPI specification(github.com)

76 points by matusf 4 years ago | 22 comments

matusf 4 years ago |

Hi HN, I'd like to share with you a fuzzer I've been working on. It is a black-box, smart, generation-based fuzzer, that fuzzes APIs based on OpenAPI specification. It all started as a bachelor thesis[0], when I wanted to do something security-related and learn Rust along the way. My colleague @viralpoetry tutored me and so far, we've been able to find bugs in software such as k8s, gitea, and vault[1].

As for the choice of language, Rust proved to be a good decision, even though, one would think that dynamic languages are better suited for fuzzing (at least that was the choice for API fuzzers that I looked into). Thanks to Rust's type system, I was able to deserialize the OpenAPI specification to structs and traverse them when creating a fuzzing payload in a type-safe way. Other fuzzers load the specification to a dictionary/hashmap and then fail during the traversal because of some missing key they expected.

0: https://github.com/matusf/bachelor-thesis/releases/download/...

1: https://github.com/matusf/openapi-fuzzer#findings

mch82 4 years ago | |

Fuzzing: “Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program”

[Wikipedia]: https://en.m.wikipedia.org/wiki/Fuzzing

(Just in case others are looking for a definition, though I guess it must be relatively well known)

anitil 4 years ago | |

Very impressive, particularly for an undergrad thesis! It's almost magical how quickly fuzzing can find bugs, even in mature software.

jonahbenton 4 years ago | |

Fantastic, great to see. I was looking into this problem several years back and recall wanting more semantics than openapi provided to be able to drive fuzzing. Maybe you have a clever solution. Will check out the thesis. Cheers!

djakaitis 4 years ago | |

Love this. Sad it’s GPL though.

CuriousCosmic 4 years ago | | |

Fun fact but there are very little hazards for GPL with testing software. Since it doesn't ship with the actual software it can be restricted to the source being distributed to the "users" of the tests which are the developers. If source must be provided to the contractor (in the case of say the US gov or a corporation), the tests only need be run in the view of a restricted set of individuals which allows them to receive the source but none of the standard users of the main, non-testing software.

phantomathkg 4 years ago | | |

Why so?

spuz 4 years ago |

I've just managed to get this set up with an API but it appears to get stuck testing only one of the endpoints. For example, if I set it up to test an endpoint with a query param like so:

> /search?q=?

Then it seems to try every single value for q it can think of (there are quite a few different possible strings) without ever moving onto the next endpoint. Is there anyway to configure its behaviour or provide hints as to what kind of fuzzing to do?

matusf 4 years ago | |

Thanks for the report. However, without more information I'm not able to help you. What is your setup? How did you run it? could you please share the specification file? Also, let's probably move this discussion to GitHub issues: https://github.com/matusf/openapi-fuzzer/issues

yellow_lead 4 years ago |

There's another one here by Microsoft - this is cool though! great to see more Rust tools.

https://github.com/microsoft/restler-fuzzer

matusf 4 years ago | |

Yes, we looked into it. There is a chapter in my thesis[0] about RESTler and comparison with OpenAPI fuzzer. The main difference between those two fuzzer is that RESTler is a statefull fuzzer and OpenAPI fuzzer is a stateless fuzzer.

Thanks to being statefull, RESTler is able to analyze a dependencies between a requests. For example, it will not call and endpoint to get user details before calling endpoint to create a user. This should make it more efficient because it does not waste requests to calling endpoint that will return 404.

On the other hand, fuzzing is all about trying to supply unexpected input to the software. Therefore, OpenAPI fuzzer makes those requests, since it may cause some undefined behavior when we try to get user before creating one.

So while RESTler tries to check some invariant (for example if deleted user cannot be accessed) OpenAPI fuzzer tries to cause the service to crash by invalid input. RESTler is usually not able to crash the service by providing invalid input, because it needs to keep the fuzzing dictionary small (bigger dictionary would make the dependency finding slow). So I think, they complements each other nicely.

Another difference it in reporting error. RESTler considers only status code 500 as an error. However, OpenAPI specification states all possible status codes that we can get from an endpoint. OpenAPI fuzzer utilizes this and reports every time it receives and unexpected status code.

0: https://github.com/matusf/bachelor-thesis/releases/download/...

yellow_lead 4 years ago | | |

Thanks for the reply! Nice to know those differences, I will definitely try your project on a spec in the future.

ludovicianul 4 years ago |

Looks like a good start. Things can get complicated when going into request bodies, especially oneOf, anyOf relations. I went through this journey while writing a similar tool: https://github.com/Endava/cats. Currently it has 72 Fuzzers registered which cover quite a lot of scenarios.

solarmist 4 years ago |

This is fantastic!

I was a just looking for a fuzzer for OpenAPI stuff. I'm building a project with FastAPI that I'd love to try this on.

pgjones 4 years ago | |

You can already do this using Hypothesis with the Pydantic plugin, https://pydantic-docs.helpmanual.io/hypothesis_plugin/. Here is an example for Quart-Schema (similar setup to FastAPI) https://pgjones.dev/blog/automatic-api-testing-2021.

solarmist 4 years ago | | |

Excellent! It's always nice to find useful tools you didn't know about.

kinow 4 years ago |

Great job! I wonder if that could be used with other fuzzers in OSS fuzz (giving tools more options to use for fuzzers and APIs)

matusf 4 years ago | |

Right now, it's probably not well suited for use in CI, since it runs until it's terminated. But I definitely plan to add a support for better usage in CI along with some examples. Thanks for the suggestion. I'll look into it!

nerdponx 4 years ago |

Does it support OAS 3.1?

matusf 4 years ago | |

No, it does not. As for the parsing of the OAS, it relies on openapiv3[0], which does not yet support it.

0: https://github.com/glademiller/openapiv3

nerdponx 4 years ago | | |

That's too bad. It's starting to feel like 3.1 is never going to gain adoption unless we get entirely new tools built for it.