AST-Driven Python Testing

10 points by MarcoDewey 332 days ago | 2 comments

Jtsummers 332 days ago |

What's the argument for having the LLM generate 20 fuzz tests instead of using a fuzzer that will likely be more comprehensive? Especially paired with grey-box and white-box techniques. It'll be cheaper and faster to execute.

MarcoDewey 332 days ago | |

you are right that a traditional fuzzer (especially a grey-box one like AFL or a white-box one) is superior in speed, cost per execution, and comprehensiveness.

The argument for using an LLM to generate a curated set of fuzz inputs isn't to replace traditional fuzzers, but to complement them by targeting a different class of bugs that traditional fuzzers are often poor at finding.

The goal of this tool is two fold.

1. give LLMs the ability to make use of traditional software testing tools

2. enhance some of the shortcomings in traditional software testing tools by selectively using LLMs (specifically their ability to understand the larger context the code is written in)