It's quick and easy to start fuzzing your code on your local machine, but as soon as you try to scale this beyond running one test at a time, it becomes unmanageable. Truly effective fuzzing is achieved with scale, and reduces the amount of developer time needed to triage and fix bugs. Here's what to keep in mind as you consider building a fuzzing solution:
The core of any scalable fuzzing setup requires a solid virtual machine orchestration system. Since the effectiveness of fuzzing scales directly with available compute power, the closer your virtual machines are to bare metal, the better.
Efficiently scheduling fuzzing jobs and distributing workloads across a fleet of machines can be a difficult task, especially when the goal of the workload is to find critical, sometimes disastrous bugs in your code. Fuzzbuzz's auto-healing infrastructure ensures that any fault found in your code is captured, recorded, and your fuzzing job continues running to detect every possible bug.
Once you've scheduled your fuzzing job, it comes down to the fuzzing engine itself to actually test your code and monitor it for bugs. We use AFL's algorithm to generate test cases, but Fuzzbuzz's fuzzing engine does a whole lot more, including: checking for race conditions, automatically syncing code coverage data, generating statistics, ensuring the reproducibility of bugs, and checking to see if old bugs have been fixed in new versions of your code.
When distributing a fuzzing workload over tens or even hundreds of machines, it's important to ensure that work doesn't get duplicated and every machine can benefit off the effort of the rest of the cluster. This involves synchronizing all interesting test cases found to every machine in the fleet, and can be a significant network and time hog if implemented inefficiently.
A side effect of distributing the fuzzing workload across many machines is that there's a good chance duplicate bugs will be discovered. Deduplicating these bugs can be troublesome, and can end up wasting days of developer time. Fuzzbuzz uses a combination of your program's live state, as well as artifacts like stacktraces, to categorize, compare, and deduplicate errors so that you're presented with a minimal, easy-to-prioritize list of unique bugs.
Continuous, automated fuzzing is the only way to be confident that your code is extensively fuzzed - and the only way to ensure you're fuzzing the latest version of your software. Fuzzbuzz provides integrations with source control tools like GitHub, as well as an open API and CLI to allow for integrations with any Continuous Integration platform.
Just like your CI tools, your fuzzing infrastructure needs to fit into your developer workflow. This means integrating with your bug tracking system, and automatically closing reports when the platform detects that bugs have been fixed. Fuzzbuzz integrates with bug tracking tools like GitHub issues and Jira, and messaging systems like Slack, so you're always up to date.
The fuzzing workflow doesn't end as soon as a bug is found - our experience with fuzzing infrastructure has shown us the importance of providing your developers with tools that make it simple to debug and fix problems on using their preferred tools. Fuzzbuzz's CLI allows developers to interact with the platform, run regression testing, and test bug fixes, directly from their machine.
Statistics are important for ensuring the effectiveness of your team and your fuzzing setup. Fuzzbuzz's business tier provides auto-generated reports that give you insights into how efficiently new interesting tests are generated, how code coverage improves over time, and how effective developers are at remediating bugs and ensuring they aren't reintroduced.
Anyone who has built complex software systems understands that bugs, database upgrades, and required
updates come up. Additionally, sometimes you just need some help fuzzing your code in the most
effective way possible.
We've built Fuzzbuzz to be as simple as possible to integrate with and use, to reduce the amount of time your dev team needs to spend dealing with fuzzing tools. If you're still unsure about your fuzzing strategy, we'd love to help - email us at [email protected] and we'll help you work out the best solution for your use case.