Introduction to Fuzzing

The missing link in the modern testing toolchain

What is fuzzing?

Fuzzing is a software testing methodology that excels at finding obscure bugs that developers can't. Instead of testing with a small, pre-defined set of cases, fuzzing tests code 24 hours a day, using the feedback and results it gathers to generate new cases (called fuzz), in an effort to exercise all aspects of the software in question. You can think of fuzzing as a feedback loop:

Why should I fuzz?

Unit testing - the traditional software testing methodology - is great for sanity checks, and ensuring that your code handles all of the cases your developers have thought about. It allows for some level of certainty that the method in question still processes the cases you've chosen to test in the same manner. In other words, unit testing checks that foreseen problems are handled properly. Where unit tests fall short is in the unknown; they look for exactly what you ask them to, and nothing more.

Fuzzing is so effective because it takes a completely different approach to software verification than human-driven test methodologies such as unit testing, code review, manual debugging and even static analysis. By continuously fuzzing your code, you immediately increase your bug-finding potential to include the unforeseen problems that aren't covered by static analysis rules or your developer's imaginations. Fuzzing covers an area of your software test pipeline that no other type of testing methodology touches.

Kostya Serebryany, head of the libFuzzer development team at Google, also thinks fuzzing should be a part of every software test pipeline:

"...testing is not enough. In our experience testing finds roughly 10% of the bugs, 80 more percent of the bugs are found by fuzzing, and then the remaining 10% are found in production by users. I want to proclaim fuzz-driven development." - Kostya Serebryany, CppCon 2017

That 80% figure can vary depending on the complexity, function and user base of your code, but ignoring fuzz testing adds that percentage to the number of bugs that are found in production, potentially long after your code has been deployed. To use a classic example, Heartbleed lay undiscovered in OpenSSL for over two years before it was uncovered in 2014. libFuzzer finds Heartbleed in about 5 seconds.

Do I really need fuzzing?

Chances are, you do. Unless your code is so simple that you're 100% sure you've covered every eventuality, fuzzing can provide value. If you're completely certain that your system can handle every potential permutation of user-submitted data, you may not need fuzzing. If you're confident that there are no potential timing attacks, denial-of-service bugs, leaks, uncaught exceptions or exploitable crashes lurking in your code, you may not need fuzzing. If, however, there's even the faintest possibility that you haven't covered everything, we encourage you to give fuzzing a go. It gives you some assurance that, even when you aren't actively hunting for bugs, your software is being tested millions of times per day, by cases specifically generated to cover as many bases as possible.

Where does Fuzzbuzz come in?

It's relatively simple to decide what part of your code you'd like to fuzz. It's less straightforward to integrate continuous fuzzing into a CI pipeline, intelligently generate fuzz, track code coverage, capture and categorize bugs, and ensure that bugs aren't re-introduced after they are fixed.

Fuzzbuzz takes all of the complexity out of running a fuzzing campaign, and makes fuzz testing as simple as unit testing by providing continuous fuzzing as a service. Instead of having to spin up an entire team to manage infrastructure and frameworks, using Fuzzbuzz lets your developers focus on writing code while reaping the benefits of a more complete software test pipeline, and leaves the busywork of managing fuzzing infrastructure to us.

Interested? Contact us to get started for free and see for yourself!