I recently dipped my toes into bug bounty hunting and finding security flaws in web applications. As a friend of UNIX shells I was building a repertoire of command line tools to make and analyze HTTP requests. Fortunately there are already many suitable tools like curl, jq, different fuzzers and some really nice tools for specific tasks by Tom Hudson [1].

However, I disliked that the existing fuzzers were monoliths where I had no easy way of creating custom behavior or analyses. They commonly do a multitude of things: Create multiple requests using one or more wordlist, sending the request, possibly with rate limiting, displaying progress, applying filters to the received responses and storing the output. If you want something different from the offered features, for example custom delays between requests or a new filter for the responses, your only option is to dig into a moderately large code base and try to adapt it to your needs.

I am a fan of the UNIX philosophy and felt like it could help out here. If there was a common format for communicating HTTP requests and responses, an ecosystem of small, specialized tools could use it to work together and fulfill tasks like fuzzing, while allowing the user to easily create custom behavior by combining the existing tools in different ways or adding small, quick to write tools to the ecosystem.

This is what I've attempted with the httpipe format [2]. It is a line based JSON format for exchanging HTTP requests and responses. I have also built some first tools using this format, namely pfuzz [3] for creating HTTP requests from wordlists, preq [4] for sending HTTP requests and receiving their responses and hpstat [5] for filtering the responses by their HTTP status codes. Since it's a line based format, many UNIX tools can be used with it as well and since each line is JSON, jq can also be used for manipulation, filtering and displaying.

[1] https://github.com/tomnomnom

[2] https://github.com/codesoap/httpipe

[3] https://github.com/codesoap/pfuzz

[4] https://github.com/codesoap/preq

[5] https://github.com/codesoap/hpstat

user:	hugepuppy
created:	September 8, 2022
karma:	0