Fuzzing Tests with Golang
Fuzzing is a type of automated test that continuously manipulates inputs into the test program to find problems such as panics, bugs, or data races to which the code may be susceptible. These semi-random data mutations can discover new code coverage that existing unit tests might miss and edge-case errors that might go undetected.
Fuzzers can often find bugs that unit tests miss because unit tests only contain values entered by the developer, so estimating these values can create errors. The program has been plagued by panics, failed assertions, infinite loops, etc. Instead of using a small, pre-defined set of manually-created inputs (like unit-testing), fuzzing continuously tests code with new cases in an effort to exercise all aspects of the software in question.
// Test function for Unit testing
TestCalculateHighest(t *testing.T){}// Fuzz function for fuzzing testing
FuzzTestHTTPHandler(f *testing.F) {}
The focus of Fuzzing is not to replace traditional tests, but to complement them by randomly iterating over the input values to the code under test. Fuzzing is especially valuable for finding security exploits and vulnerabilities, as it can reach extremes that people often overlook.
Fuzzing Materials
func FuzzIsPalindrome(f *testing.F) {
f.Add("kayak") // seed corpus
f.Fuzz(func(t *testing.T, str string) { // fuzz target
t1 := IsPalindrome(str) // func that be tested
t2 := reverse(str) == str // check to tested func
if t1 != t2 {
t.Fail()
}
})
}Seed Corpus, which you should consider as sample data. This is the data that the fuzzer will use and change to new inputs that are attempted. The seed should reflect as much as possible what the input to the function should look like in order to get the best results from the fuzzing test.
Adding seeds is done by f.Add() which accepts the following data types.
- string, []byte, rune
- int, int8, int16, int32, int64
- uint, uint8, unit16, uint32, uint64
- float32, float64
- bool
You can add many sample data entries, just make sure the sample seeds match the same order as your function input parameters.
It’s time to present Fuzz Target. The fuzz target is the function that is executed for every seed or generated corpus entry. When starting Fuzzing, the f.Fuzz() function is given as input. This function is fuzz target and it should check for errors and trigger the function we fuzzing to prepare the data. The input function to Fuzz has to accept testing.T as the first parameter, followed by the input data types added to the corpus, in the same order.
Fuzzing is a little different from regular tests. The default behavior is to run forever until errors occur and therefore you should either cancel the fuzzer or wait until an error occurs. There is a third option, which is canceled after the set time by adding the -fuzztime flag. So to run for 10 seconds you would run;
go test --fuzz=Fuzz -fuzztime=10sWhen an error occurs while fuzzing, it will cancel the failed input parameters and write them to a file. e.g: testdata\fuzz\NameOfYourFuzz\inputID.
This file tells which input string is causing the error(string(“Ó”)):
$ cat testdata/fuzz/FuzzIsPalindrome/b102348c25c69890607f026bc3186f5faf9de089188791a75c97daf5fdd10caa
go test fuzz v1
string("Ó")
$By using t.Skip(“skip reason”) you can skip the incorrect situations and this can be useful when fuzzing.
func FuzzIsPalindrome(f *testing.F) {
f.Add("kayak")
f.Fuzz(func(t *testing.T, str string) {
t1, err := IsPalindrome(str)
if errors.Is(err, ErrNotAnythingImpossible) {
t.Skip("Only correct requests are intresting")
} t2 := reverse(str) == str
if t1 != t2 {
t.Fail()
}
})
}
This may not make sense, it was just written to show the t.skip() it is written on our example code.
A new unit test can be written based on the results of the failed fuzzing test, this new unit test can be used to debug the problems found by the fuzzing test and harden them for Continuous integration(CI).
Fuzzing is effective. go-fuzz has found 200+ bugs in Go stdlib when it was already mature, written by very experienced developers, and used in production for years. Fuzzing has found 15000+ bugs in Chrome; 1500+ bugs in FFMpeg library; and thousands more. Generally fuzzing finds bugs in any code it is applied for the first time.
step-by-step Fuzzing
- Define fuzz arguments: You need to give at least one
fuzzing argument, otherwisego fuzzingcan’t generate test code, so even if we don’t have good input, we need to define afuzzing argumentthat will have an impact on the result. - How to write fuzzing target: This step focuses on writing a verifiable
fuzzing target, writing test code based on the givenfuzzing arguments, and generating data to verify the correctness of the results. - How to print the input for failed cases: if we can print out the input and form a simple test case, then we can debug it directly.
- Write a new test case: Based on the output of the failure case we should fix the code and test again with that failure cases.
Conclusion
- Fuzzing works on Go basic data types.
- Fuzzing is useful for detecting bugs that we can’t see or predict, even if your regular tests have excellent coverage.
- Failures can be detected based on: errors, panics, a side function to check, a property of the function return value, etc.
- Fuzzing produces test data files that are picked up by tests, and that shall become part of your source code to prevent from future regressions.
- Fuzzing is not deterministic. But the beauty is that it helps you enrich your deterministic tests.
Author
Okan Özşahin
LinkedIn — https://www.linkedin.com/in/okan%C3%B6z%C5%9Fahin/
Email — okan.ozsahin@innology.com
